2024-02-11T16:45:59.315 INFO:root:teuthology version: 0.0.1.dev256+gd9fdb22 2024-02-11T16:45:59.315 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2024-02-11_15:26:36-rados-wip-yuri8-testing-2024-02-07-0820-pacific-distro-default-smithi/7556240 --owner scheduled_yuriw@teuthology --description rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} --name yuriw-2024-02-11_15:26:36-rados-wip-yuri8-testing-2024-02-07-0820-pacific-distro-default-smithi --verbose -- /home/teuthworker/archive/yuriw-2024-02-11_15:26:36-rados-wip-yuri8-testing-2024-02-07-0820-pacific-distro-default-smithi/7556240/orig.config.yaml 2024-02-11T16:45:59.365 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-11T16:45:59.471 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-02-11_15:26:36-rados-wip-yuri8-testing-2024-02-07-0820-pacific-distro-default-smithi/7556240 branch: wip-yuri8-testing-2024-02-07-0820-pacific description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} email: null first_in_suite: false job_id: '7556240' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/pacific v16.2.4 ' name: yuriw-2024-02-11_15:26:36-rados-wip-yuri8-testing-2024-02-07-0820-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri8-testing-2024-02-07-0820-pacific ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 5db25abe00e218b6920215c0bef70af7e47c7152 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 5db25abe00e218b6920215c0bef70af7e47c7152 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri8-testing-2024-02-07-0820-pacific sha1: 5db25abe00e218b6920215c0bef70af7e47c7152 owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 1730 sha1: 5db25abe00e218b6920215c0bef70af7e47c7152 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri8-testing-2024-02-07-0820-pacific suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_5db25abe00e218b6920215c0bef70af7e47c7152/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 5db25abe00e218b6920215c0bef70af7e47c7152 targets: smithi037.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKvJIQHP1Zs60NA9NBl5unD1csME7/q5BnQh7W9nrh82Zli5jZdvbRPSf6Z5S1PSAzAueP0oWi0nyOKITw1FfHE= smithi178.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJdUbnVQDJMpnVjj+CNMfLqoADVfhGpoCxaA8bq8ZQ9GwbRU37n0/uWKqIGL5JrZMUIZjOgDJy3XUtm92aFr/v8= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - pexec: clients: - sudo modprobe -r ceph - sudo modprobe ceph disable_send_metrics=on - install: exclude_packages: - ceph-volume tag: v16.2.4 - print: '**** done install task...' - cephadm: avoid_pacific_features: true cephadm_branch: v16.2.4 cephadm_git_url: https://github.com/ceph/ceph image: docker.io/ceph/ceph:v16.2.4 roleless: true - print: '**** done starting v16.2.4' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 2 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay false - cephadm.shell: host.a: - ceph fs set cephfs inline_data false - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: d9fdb2209e15b39d9f061fd85399f352ce0f0894 timestamp: 2024-02-11_15:26:36 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-02-11T16:45:59.471 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_5db25abe00e218b6920215c0bef70af7e47c7152/qa; will attempt to use it 2024-02-11T16:45:59.472 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_5db25abe00e218b6920215c0bef70af7e47c7152/qa/tasks 2024-02-11T16:45:59.473 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-02-11T16:45:59.475 INFO:teuthology.task.internal:Checking packages... 2024-02-11T16:45:59.510 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5db25abe00e218b6920215c0bef70af7e47c7152' 2024-02-11T16:45:59.510 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-11T16:45:59.510 INFO:teuthology.packaging:ref: None 2024-02-11T16:45:59.511 INFO:teuthology.packaging:tag: None 2024-02-11T16:45:59.511 INFO:teuthology.packaging:branch: wip-yuri8-testing-2024-02-07-0820-pacific 2024-02-11T16:45:59.511 INFO:teuthology.packaging:sha1: 5db25abe00e218b6920215c0bef70af7e47c7152 2024-02-11T16:45:59.511 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri8-testing-2024-02-07-0820-pacific 2024-02-11T16:45:59.744 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-530.g5db25abe 2024-02-11T16:45:59.745 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-02-11T16:45:59.752 INFO:teuthology.task.internal:no buildpackages task found 2024-02-11T16:45:59.752 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-11T16:45:59.761 INFO:teuthology.task.internal:Saving configuration 2024-02-11T16:45:59.780 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-11T16:45:59.787 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-11T16:45:59.813 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi037.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-11_15:26:36-rados-wip-yuri8-testing-2024-02-07-0820-pacific-distro-default-smithi/7556240', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-11 16:40:31.448701', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKvJIQHP1Zs60NA9NBl5unD1csME7/q5BnQh7W9nrh82Zli5jZdvbRPSf6Z5S1PSAzAueP0oWi0nyOKITw1FfHE='} 2024-02-11T16:45:59.833 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi178.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-11_15:26:36-rados-wip-yuri8-testing-2024-02-07-0820-pacific-distro-default-smithi/7556240', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-11 16:40:31.450230', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJdUbnVQDJMpnVjj+CNMfLqoADVfhGpoCxaA8bq8ZQ9GwbRU37n0/uWKqIGL5JrZMUIZjOgDJy3XUtm92aFr/v8='} 2024-02-11T16:45:59.833 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-11T16:45:59.839 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-02-11T16:45:59.839 INFO:teuthology.task.internal:roles: ubuntu@smithi178.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-02-11T16:45:59.839 INFO:teuthology.run_tasks:Running task console_log... 2024-02-11T16:45:59.905 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7faf8bdf1160>, signals=[15]) 2024-02-11T16:45:59.906 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-11T16:45:59.911 INFO:teuthology.task.internal:Opening connections... 2024-02-11T16:45:59.912 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi037.front.sepia.ceph.com 2024-02-11T16:45:59.913 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-11T16:45:59.994 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi178.front.sepia.ceph.com 2024-02-11T16:45:59.995 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-11T16:46:00.068 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-11T16:46:00.075 DEBUG:teuthology.orchestra.run.smithi037:> uname -m 2024-02-11T16:46:00.097 INFO:teuthology.orchestra.run.smithi037.stdout:x86_64 2024-02-11T16:46:00.098 DEBUG:teuthology.orchestra.run.smithi037:> cat /etc/os-release 2024-02-11T16:46:00.158 INFO:teuthology.orchestra.run.smithi037.stdout:NAME="CentOS Stream" 2024-02-11T16:46:00.159 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION="8" 2024-02-11T16:46:00.159 INFO:teuthology.orchestra.run.smithi037.stdout:ID="centos" 2024-02-11T16:46:00.159 INFO:teuthology.orchestra.run.smithi037.stdout:ID_LIKE="rhel fedora" 2024-02-11T16:46:00.159 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION_ID="8" 2024-02-11T16:46:00.159 INFO:teuthology.orchestra.run.smithi037.stdout:PLATFORM_ID="platform:el8" 2024-02-11T16:46:00.159 INFO:teuthology.orchestra.run.smithi037.stdout:PRETTY_NAME="CentOS Stream 8" 2024-02-11T16:46:00.159 INFO:teuthology.orchestra.run.smithi037.stdout:ANSI_COLOR="0;31" 2024-02-11T16:46:00.159 INFO:teuthology.orchestra.run.smithi037.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-02-11T16:46:00.159 INFO:teuthology.orchestra.run.smithi037.stdout:HOME_URL="https://centos.org/" 2024-02-11T16:46:00.159 INFO:teuthology.orchestra.run.smithi037.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-11T16:46:00.159 INFO:teuthology.orchestra.run.smithi037.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-02-11T16:46:00.159 INFO:teuthology.orchestra.run.smithi037.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-11T16:46:00.160 INFO:teuthology.lock.ops:Updating smithi037.front.sepia.ceph.com on lock server 2024-02-11T16:46:00.190 DEBUG:teuthology.orchestra.run.smithi178:> uname -m 2024-02-11T16:46:00.212 INFO:teuthology.orchestra.run.smithi178.stdout:x86_64 2024-02-11T16:46:00.212 DEBUG:teuthology.orchestra.run.smithi178:> cat /etc/os-release 2024-02-11T16:46:00.272 INFO:teuthology.orchestra.run.smithi178.stdout:NAME="CentOS Stream" 2024-02-11T16:46:00.272 INFO:teuthology.orchestra.run.smithi178.stdout:VERSION="8" 2024-02-11T16:46:00.272 INFO:teuthology.orchestra.run.smithi178.stdout:ID="centos" 2024-02-11T16:46:00.272 INFO:teuthology.orchestra.run.smithi178.stdout:ID_LIKE="rhel fedora" 2024-02-11T16:46:00.272 INFO:teuthology.orchestra.run.smithi178.stdout:VERSION_ID="8" 2024-02-11T16:46:00.272 INFO:teuthology.orchestra.run.smithi178.stdout:PLATFORM_ID="platform:el8" 2024-02-11T16:46:00.273 INFO:teuthology.orchestra.run.smithi178.stdout:PRETTY_NAME="CentOS Stream 8" 2024-02-11T16:46:00.273 INFO:teuthology.orchestra.run.smithi178.stdout:ANSI_COLOR="0;31" 2024-02-11T16:46:00.273 INFO:teuthology.orchestra.run.smithi178.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-02-11T16:46:00.273 INFO:teuthology.orchestra.run.smithi178.stdout:HOME_URL="https://centos.org/" 2024-02-11T16:46:00.273 INFO:teuthology.orchestra.run.smithi178.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-11T16:46:00.273 INFO:teuthology.orchestra.run.smithi178.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-02-11T16:46:00.273 INFO:teuthology.orchestra.run.smithi178.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-11T16:46:00.273 INFO:teuthology.lock.ops:Updating smithi178.front.sepia.ceph.com on lock server 2024-02-11T16:46:00.298 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-11T16:46:00.308 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-11T16:46:00.313 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-11T16:46:00.314 DEBUG:teuthology.orchestra.run.smithi037:> test '!' -e /home/ubuntu/cephtest 2024-02-11T16:46:00.316 DEBUG:teuthology.orchestra.run.smithi178:> test '!' -e /home/ubuntu/cephtest 2024-02-11T16:46:00.335 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-11T16:46:00.342 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-11T16:46:00.342 DEBUG:teuthology.orchestra.run.smithi037:> test -z $(ls -A /var/lib/ceph) 2024-02-11T16:46:00.378 DEBUG:teuthology.orchestra.run.smithi178:> test -z $(ls -A /var/lib/ceph) 2024-02-11T16:46:00.407 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-11T16:46:00.454 INFO:teuthology.run_tasks:Running task kernel... 2024-02-11T16:46:00.467 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-02-11T16:46:00.468 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-02-11T16:46:00.468 DEBUG:teuthology.orchestra.run.smithi037:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-11T16:46:00.468 DEBUG:teuthology.orchestra.run.smithi178:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-11T16:46:00.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-11T16:46:00.487 DEBUG:teuthology.orchestra.run.smithi178:> uname -r 2024-02-11T16:46:00.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-11T16:46:00.490 DEBUG:teuthology.orchestra.run.smithi037:> uname -r 2024-02-11T16:46:00.546 INFO:teuthology.orchestra.run.smithi178.stdout:4.18.0-489.el8.x86_64 2024-02-11T16:46:00.546 INFO:teuthology.task.kernel:Running kernel on smithi178: 4.18.0-489.el8.x86_64 2024-02-11T16:46:00.546 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum install -y kernel 2024-02-11T16:46:00.551 INFO:teuthology.orchestra.run.smithi037.stdout:4.18.0-489.el8.x86_64 2024-02-11T16:46:00.551 INFO:teuthology.task.kernel:Running kernel on smithi037: 4.18.0-489.el8.x86_64 2024-02-11T16:46:00.551 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum install -y kernel 2024-02-11T16:46:03.706 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 8 - AppStream 35 MB/s | 27 MB 00:00 2024-02-11T16:46:04.324 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS Stream 8 - AppStream 25 MB/s | 27 MB 00:01 2024-02-11T16:46:09.456 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 8 - BaseOS 11 MB/s | 10 MB 00:00 2024-02-11T16:46:10.333 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS Stream 8 - BaseOS 11 MB/s | 10 MB 00:00 2024-02-11T16:46:11.423 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 8 - Extras 48 kB/s | 18 kB 00:00 2024-02-11T16:46:11.732 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 8 - Extras common packages 35 kB/s | 6.9 kB 00:00 2024-02-11T16:46:11.958 INFO:teuthology.orchestra.run.smithi178.stdout:Copr repo for python3-asyncssh owned by ceph 25 kB/s | 3.5 kB 00:00 2024-02-11T16:46:12.409 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS Stream 8 - Extras 46 kB/s | 18 kB 00:00 2024-02-11T16:46:12.892 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS Stream 8 - Extras common packages 19 kB/s | 6.9 kB 00:00 2024-02-11T16:46:13.144 INFO:teuthology.orchestra.run.smithi037.stdout:Copr repo for python3-asyncssh owned by ceph 23 kB/s | 3.5 kB 00:00 2024-02-11T16:46:13.769 INFO:teuthology.orchestra.run.smithi178.stdout:Extra Packages for Enterprise Linux 9.4 MB/s | 16 MB 00:01 2024-02-11T16:46:14.278 INFO:teuthology.orchestra.run.smithi037.stdout:Extra Packages for Enterprise Linux 16 MB/s | 16 MB 00:01 2024-02-11T16:46:17.517 INFO:teuthology.orchestra.run.smithi178.stdout:lab-extras 653 kB/s | 24 kB 00:00 2024-02-11T16:46:18.295 INFO:teuthology.orchestra.run.smithi037.stdout:lab-extras 438 kB/s | 24 kB 00:00 2024-02-11T16:46:20.447 INFO:teuthology.orchestra.run.smithi178.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-02-11T16:46:20.447 INFO:teuthology.orchestra.run.smithi178.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-02-11T16:46:20.557 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-02-11T16:46:20.558 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-02-11T16:46:20.558 INFO:teuthology.orchestra.run.smithi178.stdout: Package Architecture Version Repository Size 2024-02-11T16:46:20.558 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-02-11T16:46:20.558 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-02-11T16:46:20.558 INFO:teuthology.orchestra.run.smithi178.stdout: kernel x86_64 4.18.0-536.el8 baseos 10 M 2024-02-11T16:46:20.558 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2024-02-11T16:46:20.559 INFO:teuthology.orchestra.run.smithi178.stdout: kernel-core x86_64 4.18.0-536.el8 baseos 44 M 2024-02-11T16:46:20.559 INFO:teuthology.orchestra.run.smithi178.stdout: kernel-modules x86_64 4.18.0-536.el8 baseos 36 M 2024-02-11T16:46:20.559 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:46:20.559 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-02-11T16:46:20.559 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-02-11T16:46:20.559 INFO:teuthology.orchestra.run.smithi178.stdout:Install 3 Packages 2024-02-11T16:46:20.559 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:46:20.559 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 90 M 2024-02-11T16:46:20.559 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 96 M 2024-02-11T16:46:20.559 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-02-11T16:46:21.459 INFO:teuthology.orchestra.run.smithi037.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-02-11T16:46:21.459 INFO:teuthology.orchestra.run.smithi037.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-02-11T16:46:21.585 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-02-11T16:46:21.586 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-02-11T16:46:21.586 INFO:teuthology.orchestra.run.smithi037.stdout: Package Architecture Version Repository Size 2024-02-11T16:46:21.586 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-02-11T16:46:21.586 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-02-11T16:46:21.586 INFO:teuthology.orchestra.run.smithi037.stdout: kernel x86_64 4.18.0-536.el8 baseos 10 M 2024-02-11T16:46:21.587 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-02-11T16:46:21.587 INFO:teuthology.orchestra.run.smithi037.stdout: kernel-core x86_64 4.18.0-536.el8 baseos 44 M 2024-02-11T16:46:21.587 INFO:teuthology.orchestra.run.smithi037.stdout: kernel-modules x86_64 4.18.0-536.el8 baseos 36 M 2024-02-11T16:46:21.587 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:46:21.587 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-02-11T16:46:21.587 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-02-11T16:46:21.587 INFO:teuthology.orchestra.run.smithi037.stdout:Install 3 Packages 2024-02-11T16:46:21.587 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:46:21.587 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 90 M 2024-02-11T16:46:21.588 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 96 M 2024-02-11T16:46:21.588 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-02-11T16:46:24.393 INFO:teuthology.orchestra.run.smithi178.stdout:(1/3): kernel-4.18.0-536.el8.x86_64.rpm 2.8 MB/s | 10 MB 00:03 2024-02-11T16:46:26.280 INFO:teuthology.orchestra.run.smithi037.stdout:(1/3): kernel-4.18.0-536.el8.x86_64.rpm 2.3 MB/s | 10 MB 00:04 2024-02-11T16:46:28.826 INFO:teuthology.orchestra.run.smithi178.stdout:(2/3): kernel-modules-4.18.0-536.el8.x86_64.rpm 4.5 MB/s | 36 MB 00:08 2024-02-11T16:46:30.504 INFO:teuthology.orchestra.run.smithi037.stdout:(2/3): kernel-modules-4.18.0-536.el8.x86_64.rpm 4.2 MB/s | 36 MB 00:08 2024-02-11T16:46:30.718 INFO:teuthology.orchestra.run.smithi178.stdout:(3/3): kernel-core-4.18.0-536.el8.x86_64.rpm 4.4 MB/s | 44 MB 00:09 2024-02-11T16:46:30.718 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-02-11T16:46:30.718 INFO:teuthology.orchestra.run.smithi178.stdout:Total 8.9 MB/s | 90 MB 00:10 2024-02-11T16:46:31.402 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-02-11T16:46:31.864 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-02-11T16:46:31.864 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-02-11T16:46:33.102 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-02-11T16:46:33.126 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-02-11T16:46:33.546 INFO:teuthology.orchestra.run.smithi037.stdout:(3/3): kernel-core-4.18.0-536.el8.x86_64.rpm 3.7 MB/s | 44 MB 00:11 2024-02-11T16:46:33.546 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-02-11T16:46:33.546 INFO:teuthology.orchestra.run.smithi037.stdout:Total 7.5 MB/s | 90 MB 00:11 2024-02-11T16:46:34.260 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-02-11T16:46:34.698 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-02-11T16:46:34.698 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-02-11T16:46:36.015 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-02-11T16:46:36.021 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-02-11T16:46:36.048 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-02-11T16:46:36.546 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : kernel-core-4.18.0-536.el8.x86_64 1/3 2024-02-11T16:46:39.071 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-02-11T16:46:39.658 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : kernel-core-4.18.0-536.el8.x86_64 1/3 2024-02-11T16:46:40.239 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: kernel-core-4.18.0-536.el8.x86_64 1/3 2024-02-11T16:46:40.612 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : kernel-modules-4.18.0-536.el8.x86_64 2/3 2024-02-11T16:46:43.440 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: kernel-core-4.18.0-536.el8.x86_64 1/3 2024-02-11T16:46:43.842 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : kernel-modules-4.18.0-536.el8.x86_64 2/3 2024-02-11T16:46:44.040 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: kernel-modules-4.18.0-536.el8.x86_64 2/3 2024-02-11T16:46:44.429 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : kernel-4.18.0-536.el8.x86_64 3/3 2024-02-11T16:46:47.514 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: kernel-modules-4.18.0-536.el8.x86_64 2/3 2024-02-11T16:46:47.818 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : kernel-4.18.0-536.el8.x86_64 3/3 2024-02-11T16:47:06.172 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: kernel-core-4.18.0-536.el8.x86_64 3/3 2024-02-11T16:47:06.195 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: kernel-modules-4.18.0-536.el8.x86_64 3/3 2024-02-11T16:47:06.534 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: kernel-4.18.0-536.el8.x86_64 3/3 2024-02-11T16:47:06.534 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : kernel-4.18.0-536.el8.x86_64 1/3 2024-02-11T16:47:06.534 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : kernel-core-4.18.0-536.el8.x86_64 2/3 2024-02-11T16:47:06.993 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : kernel-modules-4.18.0-536.el8.x86_64 3/3 2024-02-11T16:47:06.993 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:47:06.994 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-02-11T16:47:06.994 INFO:teuthology.orchestra.run.smithi178.stdout: kernel-4.18.0-536.el8.x86_64 kernel-core-4.18.0-536.el8.x86_64 2024-02-11T16:47:06.994 INFO:teuthology.orchestra.run.smithi178.stdout: kernel-modules-4.18.0-536.el8.x86_64 2024-02-11T16:47:06.994 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:47:06.994 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-02-11T16:47:07.128 DEBUG:teuthology.orchestra.run.smithi178:> rpm -q kernel | sort -rV | head -n 1 2024-02-11T16:47:07.346 INFO:teuthology.orchestra.run.smithi178.stdout:kernel-4.18.0-536.el8.x86_64 2024-02-11T16:47:07.346 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-536.el8.x86_64 2024-02-11T16:47:07.347 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-536.el8.x86_64 2024-02-11T16:47:07.347 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-11T16:47:07.347 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-11T16:47:07.347 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-02-11T16:47:07.347 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi178.front.sepia.ceph.com, path=None, version=distro) 2024-02-11T16:47:07.347 DEBUG:teuthology.orchestra.run.smithi178:> rpm -q kernel | sort -rV | head -n 1 2024-02-11T16:47:07.557 INFO:teuthology.orchestra.run.smithi178.stdout:kernel-4.18.0-536.el8.x86_64 2024-02-11T16:47:07.558 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-536.el8.x86_64 2024-02-11T16:47:07.558 DEBUG:teuthology.orchestra.run.smithi178:> sudo rpm -qi grub2-tools 2024-02-11T16:47:07.591 INFO:teuthology.orchestra.run.smithi178.stdout:Name : grub2-tools 2024-02-11T16:47:07.591 INFO:teuthology.orchestra.run.smithi178.stdout:Epoch : 1 2024-02-11T16:47:07.591 INFO:teuthology.orchestra.run.smithi178.stdout:Version : 2.02 2024-02-11T16:47:07.591 INFO:teuthology.orchestra.run.smithi178.stdout:Release : 129.el8 2024-02-11T16:47:07.591 INFO:teuthology.orchestra.run.smithi178.stdout:Architecture: x86_64 2024-02-11T16:47:07.592 INFO:teuthology.orchestra.run.smithi178.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-02-11T16:47:07.592 INFO:teuthology.orchestra.run.smithi178.stdout:Group : System Environment/Base 2024-02-11T16:47:07.592 INFO:teuthology.orchestra.run.smithi178.stdout:Size : 9265586 2024-02-11T16:47:07.592 INFO:teuthology.orchestra.run.smithi178.stdout:License : GPLv3+ 2024-02-11T16:47:07.592 INFO:teuthology.orchestra.run.smithi178.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-02-11T16:47:07.592 INFO:teuthology.orchestra.run.smithi178.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-02-11T16:47:07.592 INFO:teuthology.orchestra.run.smithi178.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-02-11T16:47:07.592 INFO:teuthology.orchestra.run.smithi178.stdout:Build Host : kbuilder.bsys.centos.org 2024-02-11T16:47:07.592 INFO:teuthology.orchestra.run.smithi178.stdout:Relocations : (not relocatable) 2024-02-11T16:47:07.592 INFO:teuthology.orchestra.run.smithi178.stdout:Packager : CentOS BuildSystem 2024-02-11T16:47:07.592 INFO:teuthology.orchestra.run.smithi178.stdout:Vendor : CentOS 2024-02-11T16:47:07.592 INFO:teuthology.orchestra.run.smithi178.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-11T16:47:07.592 INFO:teuthology.orchestra.run.smithi178.stdout:Summary : Support tools for GRUB. 2024-02-11T16:47:07.592 INFO:teuthology.orchestra.run.smithi178.stdout:Description : 2024-02-11T16:47:07.592 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:47:07.592 INFO:teuthology.orchestra.run.smithi178.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-11T16:47:07.593 INFO:teuthology.orchestra.run.smithi178.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-11T16:47:07.593 INFO:teuthology.orchestra.run.smithi178.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-11T16:47:07.593 INFO:teuthology.orchestra.run.smithi178.stdout:hardware devices. 2024-02-11T16:47:07.593 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:47:07.593 INFO:teuthology.orchestra.run.smithi178.stdout:This subpackage provides tools for support of all platforms. 2024-02-11T16:47:07.594 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-11T16:47:07.594 INFO:teuthology.task.kernel:Updating grub on smithi178 to boot 4.18.0-536.el8.x86_64 2024-02-11T16:47:07.594 DEBUG:teuthology.orchestra.run.smithi178:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-11T16:47:09.144 INFO:teuthology.orchestra.run.smithi178.stderr:Generating grub configuration file ... 2024-02-11T16:47:10.921 INFO:teuthology.orchestra.run.smithi178.stderr:done 2024-02-11T16:47:10.923 DEBUG:teuthology.orchestra.run.smithi178:> mktemp 2024-02-11T16:47:10.956 INFO:teuthology.orchestra.run.smithi178.stdout:/tmp/tmp.ij8jlJUZjF 2024-02-11T16:47:10.956 DEBUG:teuthology.orchestra.run.smithi178:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.ij8jlJUZjF 2024-02-11T16:47:11.029 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod 0666 /tmp/tmp.ij8jlJUZjF 2024-02-11T16:47:11.169 DEBUG:teuthology.orchestra.remote:smithi178:/tmp/tmp.ij8jlJUZjF is 6KB 2024-02-11T16:47:11.217 DEBUG:teuthology.orchestra.run.smithi178:> rm -fr /tmp/tmp.ij8jlJUZjF 2024-02-11T16:47:11.235 DEBUG:teuthology.orchestra.run.smithi178:> sudo /bin/ls /boot/loader/entries || true 2024-02-11T16:47:11.304 INFO:teuthology.orchestra.run.smithi178.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-02-11T16:47:11.304 INFO:teuthology.orchestra.run.smithi178.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-02-11T16:47:11.305 INFO:teuthology.orchestra.run.smithi178.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-02-11T16:47:11.305 INFO:teuthology.orchestra.run.smithi178.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-536.el8.x86_64.conf 2024-02-11T16:47:11.306 DEBUG:teuthology.orchestra.run.smithi178:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-536.el8.x86_64 2024-02-11T16:47:11.385 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: kernel-core-4.18.0-536.el8.x86_64 3/3 2024-02-11T16:47:11.408 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: kernel-modules-4.18.0-536.el8.x86_64 3/3 2024-02-11T16:47:11.477 DEBUG:teuthology.orchestra.run.smithi178:> sudo shutdown -r now 2024-02-11T16:47:11.785 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: kernel-4.18.0-536.el8.x86_64 3/3 2024-02-11T16:47:11.786 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : kernel-4.18.0-536.el8.x86_64 1/3 2024-02-11T16:47:11.786 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : kernel-core-4.18.0-536.el8.x86_64 2/3 2024-02-11T16:47:12.344 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : kernel-modules-4.18.0-536.el8.x86_64 3/3 2024-02-11T16:47:12.344 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:47:12.344 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-02-11T16:47:12.344 INFO:teuthology.orchestra.run.smithi037.stdout: kernel-4.18.0-536.el8.x86_64 kernel-core-4.18.0-536.el8.x86_64 2024-02-11T16:47:12.344 INFO:teuthology.orchestra.run.smithi037.stdout: kernel-modules-4.18.0-536.el8.x86_64 2024-02-11T16:47:12.344 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:47:12.344 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-02-11T16:47:12.491 DEBUG:teuthology.orchestra.run.smithi037:> rpm -q kernel | sort -rV | head -n 1 2024-02-11T16:47:12.719 INFO:teuthology.orchestra.run.smithi037.stdout:kernel-4.18.0-536.el8.x86_64 2024-02-11T16:47:12.719 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-536.el8.x86_64 2024-02-11T16:47:12.719 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-536.el8.x86_64 2024-02-11T16:47:12.719 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-11T16:47:12.719 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-11T16:47:12.719 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-02-11T16:47:12.719 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi037.front.sepia.ceph.com, path=None, version=distro) 2024-02-11T16:47:12.720 DEBUG:teuthology.orchestra.run.smithi037:> rpm -q kernel | sort -rV | head -n 1 2024-02-11T16:47:12.940 INFO:teuthology.orchestra.run.smithi037.stdout:kernel-4.18.0-536.el8.x86_64 2024-02-11T16:47:12.940 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-536.el8.x86_64 2024-02-11T16:47:12.940 DEBUG:teuthology.orchestra.run.smithi037:> sudo rpm -qi grub2-tools 2024-02-11T16:47:12.975 INFO:teuthology.orchestra.run.smithi037.stdout:Name : grub2-tools 2024-02-11T16:47:12.975 INFO:teuthology.orchestra.run.smithi037.stdout:Epoch : 1 2024-02-11T16:47:12.975 INFO:teuthology.orchestra.run.smithi037.stdout:Version : 2.02 2024-02-11T16:47:12.975 INFO:teuthology.orchestra.run.smithi037.stdout:Release : 129.el8 2024-02-11T16:47:12.975 INFO:teuthology.orchestra.run.smithi037.stdout:Architecture: x86_64 2024-02-11T16:47:12.975 INFO:teuthology.orchestra.run.smithi037.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-02-11T16:47:12.975 INFO:teuthology.orchestra.run.smithi037.stdout:Group : System Environment/Base 2024-02-11T16:47:12.975 INFO:teuthology.orchestra.run.smithi037.stdout:Size : 9265586 2024-02-11T16:47:12.975 INFO:teuthology.orchestra.run.smithi037.stdout:License : GPLv3+ 2024-02-11T16:47:12.975 INFO:teuthology.orchestra.run.smithi037.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-02-11T16:47:12.975 INFO:teuthology.orchestra.run.smithi037.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-02-11T16:47:12.975 INFO:teuthology.orchestra.run.smithi037.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-02-11T16:47:12.976 INFO:teuthology.orchestra.run.smithi037.stdout:Build Host : kbuilder.bsys.centos.org 2024-02-11T16:47:12.976 INFO:teuthology.orchestra.run.smithi037.stdout:Relocations : (not relocatable) 2024-02-11T16:47:12.976 INFO:teuthology.orchestra.run.smithi037.stdout:Packager : CentOS BuildSystem 2024-02-11T16:47:12.976 INFO:teuthology.orchestra.run.smithi037.stdout:Vendor : CentOS 2024-02-11T16:47:12.976 INFO:teuthology.orchestra.run.smithi037.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-11T16:47:12.976 INFO:teuthology.orchestra.run.smithi037.stdout:Summary : Support tools for GRUB. 2024-02-11T16:47:12.976 INFO:teuthology.orchestra.run.smithi037.stdout:Description : 2024-02-11T16:47:12.976 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:47:12.976 INFO:teuthology.orchestra.run.smithi037.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-11T16:47:12.976 INFO:teuthology.orchestra.run.smithi037.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-11T16:47:12.976 INFO:teuthology.orchestra.run.smithi037.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-11T16:47:12.976 INFO:teuthology.orchestra.run.smithi037.stdout:hardware devices. 2024-02-11T16:47:12.976 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:47:12.976 INFO:teuthology.orchestra.run.smithi037.stdout:This subpackage provides tools for support of all platforms. 2024-02-11T16:47:12.977 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-11T16:47:12.977 INFO:teuthology.task.kernel:Updating grub on smithi037 to boot 4.18.0-536.el8.x86_64 2024-02-11T16:47:12.977 DEBUG:teuthology.orchestra.run.smithi037:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-11T16:47:14.237 INFO:teuthology.orchestra.run.smithi037.stderr:Generating grub configuration file ... 2024-02-11T16:47:16.242 INFO:teuthology.orchestra.run.smithi037.stderr:done 2024-02-11T16:47:16.244 DEBUG:teuthology.orchestra.run.smithi037:> mktemp 2024-02-11T16:47:16.273 INFO:teuthology.orchestra.run.smithi037.stdout:/tmp/tmp.5mDrAGF7dx 2024-02-11T16:47:16.274 DEBUG:teuthology.orchestra.run.smithi037:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.5mDrAGF7dx 2024-02-11T16:47:16.345 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod 0666 /tmp/tmp.5mDrAGF7dx 2024-02-11T16:47:16.487 DEBUG:teuthology.orchestra.remote:smithi037:/tmp/tmp.5mDrAGF7dx is 6KB 2024-02-11T16:47:16.511 DEBUG:teuthology.orchestra.run.smithi037:> rm -fr /tmp/tmp.5mDrAGF7dx 2024-02-11T16:47:16.528 DEBUG:teuthology.orchestra.run.smithi037:> sudo /bin/ls /boot/loader/entries || true 2024-02-11T16:47:16.610 INFO:teuthology.orchestra.run.smithi037.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-02-11T16:47:16.610 INFO:teuthology.orchestra.run.smithi037.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-02-11T16:47:16.610 INFO:teuthology.orchestra.run.smithi037.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-02-11T16:47:16.610 INFO:teuthology.orchestra.run.smithi037.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-536.el8.x86_64.conf 2024-02-11T16:47:16.611 DEBUG:teuthology.orchestra.run.smithi037:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-536.el8.x86_64 2024-02-11T16:47:16.767 DEBUG:teuthology.orchestra.run.smithi037:> sudo shutdown -r now 2024-02-11T16:47:41.500 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-02-11T16:47:41.500 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-02-11T16:47:41.501 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-11T16:47:46.772 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-02-11T16:47:46.772 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-02-11T16:47:46.773 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-11T16:48:00.060 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.178 2024-02-11T16:48:05.184 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-02-11T16:48:09.064 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-02-11T16:48:09.065 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-11T16:48:12.124 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.178 2024-02-11T16:48:14.188 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-02-11T16:48:14.188 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-11T16:48:24.126 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-02-11T16:48:24.127 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-11T16:48:27.196 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.178 2024-02-11T16:48:32.576 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-02-11T16:48:42.207 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-02-11T16:48:42.208 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-11T16:48:44.579 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-02-11T16:48:44.580 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-11T16:48:44.969 DEBUG:teuthology.orchestra.run.smithi037:> true 2024-02-11T16:48:45.200 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-02-11T16:48:45.200 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "4.18.0-536.el8.x86_64"... 2024-02-11T16:48:45.200 DEBUG:teuthology.orchestra.run.smithi037:> uname -r 2024-02-11T16:48:45.259 INFO:teuthology.orchestra.run.smithi037.stdout:4.18.0-536.el8.x86_64 2024-02-11T16:48:45.260 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-536.el8.x86_64 vs 4.18.0-536.el8.x86_64 2024-02-11T16:48:45.260 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-11T16:48:45.260 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-11T16:48:46.261 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-02-11T16:48:46.261 DEBUG:teuthology.orchestra.run.smithi037:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-11T16:48:46.343 INFO:teuthology.orchestra.run.smithi037.stdout:ttyS1 2024-02-11T16:48:46.357 DEBUG:teuthology.parallel:result is None 2024-02-11T16:49:14.071 DEBUG:teuthology.orchestra.run.smithi178:> true 2024-02-11T16:49:14.352 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-02-11T16:49:14.352 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "4.18.0-536.el8.x86_64"... 2024-02-11T16:49:14.353 DEBUG:teuthology.orchestra.run.smithi178:> uname -r 2024-02-11T16:49:14.370 INFO:teuthology.orchestra.run.smithi178.stdout:4.18.0-536.el8.x86_64 2024-02-11T16:49:14.370 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-536.el8.x86_64 vs 4.18.0-536.el8.x86_64 2024-02-11T16:49:14.370 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-11T16:49:14.370 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-11T16:49:15.371 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-02-11T16:49:15.372 DEBUG:teuthology.orchestra.run.smithi178:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-11T16:49:15.442 INFO:teuthology.orchestra.run.smithi178.stdout:ttyS1 2024-02-11T16:49:15.455 DEBUG:teuthology.parallel:result is None 2024-02-11T16:49:15.455 INFO:teuthology.run_tasks:Running task internal.base... 2024-02-11T16:49:15.463 INFO:teuthology.task.internal:Creating test directory... 2024-02-11T16:49:15.463 DEBUG:teuthology.orchestra.run.smithi037:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-11T16:49:15.466 DEBUG:teuthology.orchestra.run.smithi178:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-11T16:49:15.487 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-02-11T16:49:15.494 INFO:teuthology.run_tasks:Running task internal.archive... 2024-02-11T16:49:15.504 INFO:teuthology.task.internal:Creating archive directory... 2024-02-11T16:49:15.505 DEBUG:teuthology.orchestra.run.smithi037:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-11T16:49:15.530 DEBUG:teuthology.orchestra.run.smithi178:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-11T16:49:15.569 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-02-11T16:49:15.576 INFO:teuthology.task.internal:Enabling coredump saving... 2024-02-11T16:49:15.576 DEBUG:teuthology.orchestra.run.smithi037:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-11T16:49:15.601 DEBUG:teuthology.orchestra.run.smithi178:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-11T16:49:15.641 INFO:teuthology.orchestra.run.smithi037.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-11T16:49:15.653 INFO:teuthology.orchestra.run.smithi037.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-11T16:49:15.664 INFO:teuthology.orchestra.run.smithi178.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-11T16:49:15.675 INFO:teuthology.orchestra.run.smithi178.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-11T16:49:15.676 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-02-11T16:49:15.686 INFO:teuthology.task.internal:Configuring sudo... 2024-02-11T16:49:15.686 DEBUG:teuthology.orchestra.run.smithi037:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-11T16:49:15.697 DEBUG:teuthology.orchestra.run.smithi178:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-11T16:49:15.727 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-02-11T16:49:15.737 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-02-11T16:49:15.737 DEBUG:teuthology.orchestra.run.smithi037:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-11T16:49:15.771 DEBUG:teuthology.orchestra.run.smithi178:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-11T16:49:15.789 DEBUG:teuthology.orchestra.run.smithi037:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-11T16:49:15.876 DEBUG:teuthology.orchestra.run.smithi037:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-11T16:49:15.933 DEBUG:teuthology.orchestra.run.smithi037:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-11T16:49:16.020 DEBUG:teuthology.orchestra.run.smithi037:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-11T16:49:16.062 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-02-11T16:49:16.063 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-11T16:49:16.131 DEBUG:teuthology.orchestra.run.smithi178:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-11T16:49:16.176 DEBUG:teuthology.orchestra.run.smithi178:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-11T16:49:16.226 DEBUG:teuthology.orchestra.run.smithi178:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-11T16:49:16.312 DEBUG:teuthology.orchestra.run.smithi178:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-11T16:49:16.352 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-02-11T16:49:16.353 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-11T16:49:16.419 DEBUG:teuthology.orchestra.run.smithi037:> sudo service rsyslog restart 2024-02-11T16:49:16.421 DEBUG:teuthology.orchestra.run.smithi178:> sudo service rsyslog restart 2024-02-11T16:49:16.500 INFO:teuthology.orchestra.run.smithi037.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-11T16:49:16.525 INFO:teuthology.orchestra.run.smithi178.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-11T16:49:16.994 INFO:teuthology.run_tasks:Running task internal.timer... 2024-02-11T16:49:17.004 INFO:teuthology.task.internal:Starting timer... 2024-02-11T16:49:17.004 INFO:teuthology.run_tasks:Running task pcp... 2024-02-11T16:49:17.020 INFO:teuthology.run_tasks:Running task selinux... 2024-02-11T16:49:17.030 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-02-11T16:49:17.031 DEBUG:teuthology.orchestra.run.smithi037:> sudo service auditd rotate 2024-02-11T16:49:17.121 INFO:teuthology.orchestra.run.smithi037.stdout:Rotating logs: 2024-02-11T16:49:17.123 DEBUG:teuthology.orchestra.run.smithi178:> sudo service auditd rotate 2024-02-11T16:49:17.199 INFO:teuthology.orchestra.run.smithi178.stdout:Rotating logs: 2024-02-11T16:49:17.201 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-02-11T16:49:17.201 DEBUG:teuthology.orchestra.run.smithi037:> /usr/sbin/getenforce 2024-02-11T16:49:17.252 INFO:teuthology.orchestra.run.smithi037.stdout:Permissive 2024-02-11T16:49:17.252 DEBUG:teuthology.orchestra.run.smithi178:> /usr/sbin/getenforce 2024-02-11T16:49:17.273 INFO:teuthology.orchestra.run.smithi178.stdout:Permissive 2024-02-11T16:49:17.273 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi037.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi178.front.sepia.ceph.com': 'permissive'} 2024-02-11T16:49:17.273 DEBUG:teuthology.orchestra.run.smithi037:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-11T16:49:17.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-11T16:49:17.302 DEBUG:teuthology.orchestra.run.smithi178:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-11T16:49:17.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-11T16:49:17.340 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-02-11T16:49:17.341 DEBUG:teuthology.orchestra.run.smithi037:> sudo /usr/sbin/setenforce permissive 2024-02-11T16:49:17.374 DEBUG:teuthology.orchestra.run.smithi178:> sudo /usr/sbin/setenforce permissive 2024-02-11T16:49:17.416 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-02-11T16:49:17.427 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-02-11T16:49:17.432 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-02-11T16:49:17.537 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-02-11T16:49:17.552 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-02-11T16:49:17.554 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi037.front.sepia.ceph.com,smithi178.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-02-11T16:55:08.390 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi037.front.sepia.ceph.com'), Remote(name='ubuntu@smithi178.front.sepia.ceph.com')] 2024-02-11T16:55:08.391 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-02-11T16:55:08.393 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-11T16:55:08.473 DEBUG:teuthology.orchestra.run.smithi037:> true 2024-02-11T16:55:08.545 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-02-11T16:55:08.545 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-02-11T16:55:08.546 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-11T16:55:08.624 DEBUG:teuthology.orchestra.run.smithi178:> true 2024-02-11T16:55:08.692 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi178.front.sepia.ceph.com' 2024-02-11T16:55:08.693 INFO:teuthology.run_tasks:Running task clock... 2024-02-11T16:55:08.703 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-02-11T16:55:08.704 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-11T16:55:08.704 DEBUG:teuthology.orchestra.run.smithi037:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-11T16:55:08.707 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-11T16:55:08.707 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-11T16:55:08.740 INFO:teuthology.orchestra.run.smithi037.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-11T16:55:08.755 INFO:teuthology.orchestra.run.smithi037.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-11T16:55:08.770 INFO:teuthology.orchestra.run.smithi178.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-11T16:55:08.784 INFO:teuthology.orchestra.run.smithi178.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-11T16:55:08.834 INFO:teuthology.orchestra.run.smithi037.stderr:sudo: ntpd: command not found 2024-02-11T16:55:08.845 INFO:teuthology.orchestra.run.smithi037.stdout:506 Cannot talk to daemon 2024-02-11T16:55:08.855 INFO:teuthology.orchestra.run.smithi178.stderr:sudo: ntpd: command not found 2024-02-11T16:55:08.859 INFO:teuthology.orchestra.run.smithi037.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-11T16:55:08.868 INFO:teuthology.orchestra.run.smithi178.stdout:506 Cannot talk to daemon 2024-02-11T16:55:08.873 INFO:teuthology.orchestra.run.smithi037.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-11T16:55:08.882 INFO:teuthology.orchestra.run.smithi178.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-11T16:55:08.895 INFO:teuthology.orchestra.run.smithi178.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-11T16:55:08.942 INFO:teuthology.orchestra.run.smithi037.stderr:bash: ntpq: command not found 2024-02-11T16:55:08.946 INFO:teuthology.orchestra.run.smithi037.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-11T16:55:08.946 INFO:teuthology.orchestra.run.smithi037.stdout:=============================================================================== 2024-02-11T16:55:08.946 INFO:teuthology.orchestra.run.smithi037.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-11T16:55:08.946 INFO:teuthology.orchestra.run.smithi037.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-11T16:55:08.946 INFO:teuthology.orchestra.run.smithi037.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-11T16:55:08.946 INFO:teuthology.orchestra.run.smithi037.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-11T16:55:08.971 INFO:teuthology.orchestra.run.smithi178.stderr:bash: ntpq: command not found 2024-02-11T16:55:08.975 INFO:teuthology.orchestra.run.smithi178.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-11T16:55:08.975 INFO:teuthology.orchestra.run.smithi178.stdout:=============================================================================== 2024-02-11T16:55:08.975 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-11T16:55:08.975 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-11T16:55:08.975 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-11T16:55:08.975 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-11T16:55:08.975 INFO:teuthology.run_tasks:Running task pexec... 2024-02-11T16:55:08.985 INFO:teuthology.task.pexec:Executing custom commands... 2024-02-11T16:55:08.986 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi037.front.sepia.ceph.com 2024-02-11T16:55:08.986 DEBUG:teuthology.orchestra.run.smithi037:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-11T16:55:08.986 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi178.front.sepia.ceph.com 2024-02-11T16:55:08.986 DEBUG:teuthology.orchestra.run.smithi178:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-11T16:55:09.800 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:25 ago on Sun 11 Feb 2024 04:53:44 PM UTC. 2024-02-11T16:55:09.805 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:19 ago on Sun 11 Feb 2024 04:53:50 PM UTC. 2024-02-11T16:55:11.757 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-02-11T16:55:11.758 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-02-11T16:55:11.758 INFO:teuthology.orchestra.run.smithi178.stdout: Package Architecture Version Repository Size 2024-02-11T16:55:11.758 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-02-11T16:55:11.758 INFO:teuthology.orchestra.run.smithi178.stdout:Resetting modules: 2024-02-11T16:55:11.758 INFO:teuthology.orchestra.run.smithi178.stdout: container-tools 2024-02-11T16:55:11.758 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:55:11.758 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-02-11T16:55:11.758 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-02-11T16:55:11.758 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:55:11.909 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-02-11T16:55:11.910 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-02-11T16:55:11.910 INFO:teuthology.orchestra.run.smithi037.stdout: Package Architecture Version Repository Size 2024-02-11T16:55:11.910 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-02-11T16:55:11.910 INFO:teuthology.orchestra.run.smithi037.stdout:Resetting modules: 2024-02-11T16:55:11.910 INFO:teuthology.orchestra.run.smithi037.stdout: container-tools 2024-02-11T16:55:11.910 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:55:11.910 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-02-11T16:55:11.910 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-02-11T16:55:11.910 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:55:12.011 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-02-11T16:55:12.174 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-02-11T16:55:12.863 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:28 ago on Sun 11 Feb 2024 04:53:44 PM UTC. 2024-02-11T16:55:13.058 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:22 ago on Sun 11 Feb 2024 04:53:50 PM UTC. 2024-02-11T16:55:14.822 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-02-11T16:55:14.857 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-02-11T16:55:14.857 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2024-02-11T16:55:14.857 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-02-11T16:55:14.857 INFO:teuthology.orchestra.run.smithi178.stdout:Installing group/module packages: 2024-02-11T16:55:14.857 INFO:teuthology.orchestra.run.smithi178.stdout: aardvark-dns x86_64 2:1.9.0-1.module_el8+804+f131391c appstream 1.0 M 2024-02-11T16:55:14.857 INFO:teuthology.orchestra.run.smithi178.stdout: buildah x86_64 1:1.34.0-1.module_el8+804+f131391c appstream 9.7 M 2024-02-11T16:55:14.857 INFO:teuthology.orchestra.run.smithi178.stdout: cockpit-podman noarch 82-1.module_el8+804+f131391c appstream 759 k 2024-02-11T16:55:14.857 INFO:teuthology.orchestra.run.smithi178.stdout: crun x86_64 1.12-1.module_el8+804+f131391c appstream 248 k 2024-02-11T16:55:14.857 INFO:teuthology.orchestra.run.smithi178.stdout: netavark x86_64 2:1.9.0-1.module_el8+804+f131391c appstream 4.0 M 2024-02-11T16:55:14.857 INFO:teuthology.orchestra.run.smithi178.stdout: python3-podman noarch 4.8.2-1.module_el8+804+f131391c appstream 179 k 2024-02-11T16:55:14.857 INFO:teuthology.orchestra.run.smithi178.stdout: skopeo x86_64 2:1.14.0-2.module_el8+804+f131391c appstream 8.8 M 2024-02-11T16:55:14.857 INFO:teuthology.orchestra.run.smithi178.stdout: toolbox x86_64 0.0.99.4-8.module_el8+804+f131391c appstream 2.5 M 2024-02-11T16:55:14.857 INFO:teuthology.orchestra.run.smithi178.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-02-11T16:55:14.858 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2024-02-11T16:55:14.858 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-02-11T16:55:14.858 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-02-11T16:55:14.858 INFO:teuthology.orchestra.run.smithi178.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-02-11T16:55:14.858 INFO:teuthology.orchestra.run.smithi178.stdout:Installing weak dependencies: 2024-02-11T16:55:14.858 INFO:teuthology.orchestra.run.smithi178.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-02-11T16:55:14.858 INFO:teuthology.orchestra.run.smithi178.stdout:Installing module profiles: 2024-02-11T16:55:14.858 INFO:teuthology.orchestra.run.smithi178.stdout: container-tools/common 2024-02-11T16:55:14.858 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:55:14.858 INFO:teuthology.orchestra.run.smithi178.stdout:Enabling module streams: 2024-02-11T16:55:14.858 INFO:teuthology.orchestra.run.smithi178.stdout: container-tools rhel8 2024-02-11T16:55:14.858 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:55:14.858 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-02-11T16:55:14.858 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-02-11T16:55:14.858 INFO:teuthology.orchestra.run.smithi178.stdout:Install 13 Packages 2024-02-11T16:55:14.859 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:55:14.859 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 27 M 2024-02-11T16:55:14.859 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 91 M 2024-02-11T16:55:14.859 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-02-11T16:55:15.000 INFO:teuthology.orchestra.run.smithi178.stdout:(1/13): cockpit-podman-82-1.module_el8+804+f131 7.4 MB/s | 759 kB 00:00 2024-02-11T16:55:15.034 INFO:teuthology.orchestra.run.smithi178.stdout:(2/13): aardvark-dns-1.9.0-1.module_el8+804+f13 7.5 MB/s | 1.0 MB 00:00 2024-02-11T16:55:15.067 INFO:teuthology.orchestra.run.smithi178.stdout:(3/13): crun-1.12-1.module_el8+804+f131391c.x86 7.3 MB/s | 248 kB 00:00 2024-02-11T16:55:15.110 INFO:teuthology.orchestra.run.smithi178.stdout:(4/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 348 kB/s | 38 kB 00:00 2024-02-11T16:55:15.143 INFO:teuthology.orchestra.run.smithi178.stdout:(5/13): python3-podman-4.8.2-1.module_el8+804+f 5.2 MB/s | 179 kB 00:00 2024-02-11T16:55:15.168 INFO:teuthology.orchestra.run.smithi178.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 996 kB/s | 25 kB 00:00 2024-02-11T16:55:15.171 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-02-11T16:55:15.210 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-02-11T16:55:15.210 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-02-11T16:55:15.210 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-02-11T16:55:15.210 INFO:teuthology.orchestra.run.smithi037.stdout:Installing group/module packages: 2024-02-11T16:55:15.210 INFO:teuthology.orchestra.run.smithi037.stdout: aardvark-dns x86_64 2:1.9.0-1.module_el8+804+f131391c appstream 1.0 M 2024-02-11T16:55:15.210 INFO:teuthology.orchestra.run.smithi037.stdout: buildah x86_64 1:1.34.0-1.module_el8+804+f131391c appstream 9.7 M 2024-02-11T16:55:15.210 INFO:teuthology.orchestra.run.smithi037.stdout: cockpit-podman noarch 82-1.module_el8+804+f131391c appstream 759 k 2024-02-11T16:55:15.210 INFO:teuthology.orchestra.run.smithi037.stdout: crun x86_64 1.12-1.module_el8+804+f131391c appstream 248 k 2024-02-11T16:55:15.210 INFO:teuthology.orchestra.run.smithi037.stdout: netavark x86_64 2:1.9.0-1.module_el8+804+f131391c appstream 4.0 M 2024-02-11T16:55:15.210 INFO:teuthology.orchestra.run.smithi037.stdout: python3-podman noarch 4.8.2-1.module_el8+804+f131391c appstream 179 k 2024-02-11T16:55:15.210 INFO:teuthology.orchestra.run.smithi037.stdout: skopeo x86_64 2:1.14.0-2.module_el8+804+f131391c appstream 8.8 M 2024-02-11T16:55:15.210 INFO:teuthology.orchestra.run.smithi037.stdout: toolbox x86_64 0.0.99.4-8.module_el8+804+f131391c appstream 2.5 M 2024-02-11T16:55:15.211 INFO:teuthology.orchestra.run.smithi037.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-02-11T16:55:15.211 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-02-11T16:55:15.211 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-02-11T16:55:15.211 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-02-11T16:55:15.211 INFO:teuthology.orchestra.run.smithi037.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-02-11T16:55:15.211 INFO:teuthology.orchestra.run.smithi037.stdout:Installing weak dependencies: 2024-02-11T16:55:15.211 INFO:teuthology.orchestra.run.smithi037.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-02-11T16:55:15.211 INFO:teuthology.orchestra.run.smithi037.stdout:Installing module profiles: 2024-02-11T16:55:15.211 INFO:teuthology.orchestra.run.smithi037.stdout: container-tools/common 2024-02-11T16:55:15.211 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:55:15.211 INFO:teuthology.orchestra.run.smithi037.stdout:Enabling module streams: 2024-02-11T16:55:15.211 INFO:teuthology.orchestra.run.smithi037.stdout: container-tools rhel8 2024-02-11T16:55:15.211 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:55:15.211 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-02-11T16:55:15.211 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-02-11T16:55:15.212 INFO:teuthology.orchestra.run.smithi037.stdout:Install 13 Packages 2024-02-11T16:55:15.212 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:55:15.212 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 27 M 2024-02-11T16:55:15.212 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 91 M 2024-02-11T16:55:15.212 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-02-11T16:55:15.227 INFO:teuthology.orchestra.run.smithi178.stdout:(7/13): netavark-1.9.0-1.module_el8+804+f131391 25 MB/s | 4.0 MB 00:00 2024-02-11T16:55:15.273 INFO:teuthology.orchestra.run.smithi178.stdout:(8/13): python3-pyxdg-0.25-16.el8.noarch.rpm 899 kB/s | 94 kB 00:00 2024-02-11T16:55:15.359 INFO:teuthology.orchestra.run.smithi037.stdout:(1/13): aardvark-dns-1.9.0-1.module_el8+804+f13 9.6 MB/s | 1.0 MB 00:00 2024-02-11T16:55:15.384 INFO:teuthology.orchestra.run.smithi037.stdout:(2/13): cockpit-podman-82-1.module_el8+804+f131 5.7 MB/s | 759 kB 00:00 2024-02-11T16:55:15.401 INFO:teuthology.orchestra.run.smithi037.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 901 kB/s | 38 kB 00:00 2024-02-11T16:55:15.403 INFO:teuthology.orchestra.run.smithi178.stdout:(9/13): buildah-1.34.0-1.module_el8+804+f131391 19 MB/s | 9.7 MB 00:00 2024-02-11T16:55:15.426 INFO:teuthology.orchestra.run.smithi037.stdout:(4/13): crun-1.12-1.module_el8+804+f131391c.x86 5.8 MB/s | 248 kB 00:00 2024-02-11T16:55:15.453 INFO:teuthology.orchestra.run.smithi178.stdout:(10/13): toolbox-0.0.99.4-8.module_el8+804+f131 14 MB/s | 2.5 MB 00:00 2024-02-11T16:55:15.460 INFO:teuthology.orchestra.run.smithi037.stdout:(5/13): python3-podman-4.8.2-1.module_el8+804+f 5.3 MB/s | 179 kB 00:00 2024-02-11T16:55:15.478 INFO:teuthology.orchestra.run.smithi178.stdout:(11/13): udica-0.2.6-20.module_el8+804+f131391c 700 kB/s | 53 kB 00:00 2024-02-11T16:55:15.510 INFO:teuthology.orchestra.run.smithi037.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 494 kB/s | 25 kB 00:00 2024-02-11T16:55:15.512 INFO:teuthology.orchestra.run.smithi178.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 699 kB/s | 41 kB 00:00 2024-02-11T16:55:15.577 INFO:teuthology.orchestra.run.smithi037.stdout:(7/13): netavark-1.9.0-1.module_el8+804+f131391 22 MB/s | 4.0 MB 00:00 2024-02-11T16:55:15.602 INFO:teuthology.orchestra.run.smithi037.stdout:(8/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.0 MB/s | 94 kB 00:00 2024-02-11T16:55:15.654 INFO:teuthology.orchestra.run.smithi178.stdout:(13/13): skopeo-1.14.0-2.module_el8+804+f131391 21 MB/s | 8.8 MB 00:00 2024-02-11T16:55:15.654 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-02-11T16:55:15.655 INFO:teuthology.orchestra.run.smithi178.stdout:Total 34 MB/s | 27 MB 00:00 2024-02-11T16:55:15.711 INFO:teuthology.orchestra.run.smithi037.stdout:(9/13): buildah-1.34.0-1.module_el8+804+f131391 21 MB/s | 9.7 MB 00:00 2024-02-11T16:55:15.795 INFO:teuthology.orchestra.run.smithi037.stdout:(10/13): udica-0.2.6-20.module_el8+804+f131391c 635 kB/s | 53 kB 00:00 2024-02-11T16:55:15.837 INFO:teuthology.orchestra.run.smithi037.stdout:(11/13): toolbox-0.0.99.4-8.module_el8+804+f131 11 MB/s | 2.5 MB 00:00 2024-02-11T16:55:15.854 INFO:teuthology.orchestra.run.smithi037.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 698 kB/s | 41 kB 00:00 2024-02-11T16:55:15.924 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-02-11T16:55:15.945 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-02-11T16:55:15.945 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-02-11T16:55:15.954 INFO:teuthology.orchestra.run.smithi037.stdout:(13/13): skopeo-1.14.0-2.module_el8+804+f131391 23 MB/s | 8.8 MB 00:00 2024-02-11T16:55:15.954 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-02-11T16:55:15.954 INFO:teuthology.orchestra.run.smithi037.stdout:Total 37 MB/s | 27 MB 00:00 2024-02-11T16:55:16.225 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-02-11T16:55:16.229 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-02-11T16:55:16.231 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-02-11T16:55:16.254 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-02-11T16:55:16.254 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-02-11T16:55:16.533 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-02-11T16:55:16.537 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-02-11T16:55:16.938 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-02-11T16:55:17.168 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-02-11T16:55:17.236 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-02-11T16:55:17.331 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-02-11T16:55:17.428 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-02-11T16:55:17.526 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2024-02-11T16:55:17.580 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-02-11T16:55:17.773 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-02-11T16:55:17.774 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2024-02-11T16:55:18.029 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-02-11T16:55:18.312 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : aardvark-dns-2:1.9.0-1.module_el8+804+f131391c.x86 5/13 2024-02-11T16:55:18.586 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : aardvark-dns-2:1.9.0-1.module_el8+804+f131391c.x86 5/13 2024-02-11T16:55:19.280 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : netavark-2:1.9.0-1.module_el8+804+f131391c.x86_64 6/13 2024-02-11T16:55:19.528 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : buildah-1:1.34.0-1.module_el8+804+f131391c.x86_64 7/13 2024-02-11T16:55:19.586 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : netavark-2:1.9.0-1.module_el8+804+f131391c.x86_64 6/13 2024-02-11T16:55:19.765 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : crun-1.12-1.module_el8+804+f131391c.x86_64 8/13 2024-02-11T16:55:19.817 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : buildah-1:1.34.0-1.module_el8+804+f131391c.x86_64 7/13 2024-02-11T16:55:20.001 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-podman-4.8.2-1.module_el8+804+f131391c.noa 9/13 2024-02-11T16:55:20.008 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : crun-1.12-1.module_el8+804+f131391c.x86_64 8/13 2024-02-11T16:55:20.276 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-podman-4.8.2-1.module_el8+804+f131391c.noa 9/13 2024-02-11T16:55:20.391 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 10/13 2024-02-11T16:55:20.671 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 10/13 2024-02-11T16:55:21.330 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : toolbox-0.0.99.4-8.module_el8+804+f131391c.x86_64 11/13 2024-02-11T16:55:21.535 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : skopeo-2:1.14.0-2.module_el8+804+f131391c.x86_64 12/13 2024-02-11T16:55:21.672 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : toolbox-0.0.99.4-8.module_el8+804+f131391c.x86_64 11/13 2024-02-11T16:55:21.721 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : cockpit-podman-82-1.module_el8+804+f131391c.noarch 13/13 2024-02-11T16:55:21.880 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : skopeo-2:1.14.0-2.module_el8+804+f131391c.x86_64 12/13 2024-02-11T16:55:22.096 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : cockpit-podman-82-1.module_el8+804+f131391c.noarch 13/13 2024-02-11T16:55:22.147 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: cockpit-podman-82-1.module_el8+804+f131391c.noarch 13/13 2024-02-11T16:55:22.147 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : aardvark-dns-2:1.9.0-1.module_el8+804+f131391c.x86 1/13 2024-02-11T16:55:22.147 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : buildah-1:1.34.0-1.module_el8+804+f131391c.x86_64 2/13 2024-02-11T16:55:22.148 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : cockpit-podman-82-1.module_el8+804+f131391c.noarch 3/13 2024-02-11T16:55:22.148 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-02-11T16:55:22.148 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : crun-1.12-1.module_el8+804+f131391c.x86_64 5/13 2024-02-11T16:55:22.148 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : netavark-2:1.9.0-1.module_el8+804+f131391c.x86_64 6/13 2024-02-11T16:55:22.148 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-podman-4.8.2-1.module_el8+804+f131391c.noa 7/13 2024-02-11T16:55:22.148 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-02-11T16:55:22.148 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-02-11T16:55:22.148 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : skopeo-2:1.14.0-2.module_el8+804+f131391c.x86_64 10/13 2024-02-11T16:55:22.148 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : toolbox-0.0.99.4-8.module_el8+804+f131391c.x86_64 11/13 2024-02-11T16:55:22.148 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-11T16:55:22.548 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: cockpit-podman-82-1.module_el8+804+f131391c.noarch 13/13 2024-02-11T16:55:22.548 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : aardvark-dns-2:1.9.0-1.module_el8+804+f131391c.x86 1/13 2024-02-11T16:55:22.548 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : buildah-1:1.34.0-1.module_el8+804+f131391c.x86_64 2/13 2024-02-11T16:55:22.548 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : cockpit-podman-82-1.module_el8+804+f131391c.noarch 3/13 2024-02-11T16:55:22.548 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-02-11T16:55:22.548 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : crun-1.12-1.module_el8+804+f131391c.x86_64 5/13 2024-02-11T16:55:22.548 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : netavark-2:1.9.0-1.module_el8+804+f131391c.x86_64 6/13 2024-02-11T16:55:22.548 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-podman-4.8.2-1.module_el8+804+f131391c.noa 7/13 2024-02-11T16:55:22.548 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-02-11T16:55:22.549 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-02-11T16:55:22.549 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : skopeo-2:1.14.0-2.module_el8+804+f131391c.x86_64 10/13 2024-02-11T16:55:22.549 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : toolbox-0.0.99.4-8.module_el8+804+f131391c.x86_64 11/13 2024-02-11T16:55:22.549 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-11T16:55:22.643 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-02-11T16:55:22.643 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:55:22.643 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-02-11T16:55:22.643 INFO:teuthology.orchestra.run.smithi178.stdout: aardvark-dns-2:1.9.0-1.module_el8+804+f131391c.x86_64 2024-02-11T16:55:22.643 INFO:teuthology.orchestra.run.smithi178.stdout: buildah-1:1.34.0-1.module_el8+804+f131391c.x86_64 2024-02-11T16:55:22.643 INFO:teuthology.orchestra.run.smithi178.stdout: cockpit-podman-82-1.module_el8+804+f131391c.noarch 2024-02-11T16:55:22.643 INFO:teuthology.orchestra.run.smithi178.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-02-11T16:55:22.643 INFO:teuthology.orchestra.run.smithi178.stdout: crun-1.12-1.module_el8+804+f131391c.x86_64 2024-02-11T16:55:22.643 INFO:teuthology.orchestra.run.smithi178.stdout: netavark-2:1.9.0-1.module_el8+804+f131391c.x86_64 2024-02-11T16:55:22.644 INFO:teuthology.orchestra.run.smithi178.stdout: python3-podman-4.8.2-1.module_el8+804+f131391c.noarch 2024-02-11T16:55:22.644 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-02-11T16:55:22.644 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-02-11T16:55:22.644 INFO:teuthology.orchestra.run.smithi178.stdout: skopeo-2:1.14.0-2.module_el8+804+f131391c.x86_64 2024-02-11T16:55:22.644 INFO:teuthology.orchestra.run.smithi178.stdout: toolbox-0.0.99.4-8.module_el8+804+f131391c.x86_64 2024-02-11T16:55:22.644 INFO:teuthology.orchestra.run.smithi178.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-02-11T16:55:22.644 INFO:teuthology.orchestra.run.smithi178.stdout: yajl-2.1.0-12.el8.x86_64 2024-02-11T16:55:22.644 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:55:22.644 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-02-11T16:55:22.935 DEBUG:teuthology.parallel:result is None 2024-02-11T16:55:23.052 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-02-11T16:55:23.052 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:55:23.052 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-02-11T16:55:23.052 INFO:teuthology.orchestra.run.smithi037.stdout: aardvark-dns-2:1.9.0-1.module_el8+804+f131391c.x86_64 2024-02-11T16:55:23.052 INFO:teuthology.orchestra.run.smithi037.stdout: buildah-1:1.34.0-1.module_el8+804+f131391c.x86_64 2024-02-11T16:55:23.052 INFO:teuthology.orchestra.run.smithi037.stdout: cockpit-podman-82-1.module_el8+804+f131391c.noarch 2024-02-11T16:55:23.052 INFO:teuthology.orchestra.run.smithi037.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-02-11T16:55:23.052 INFO:teuthology.orchestra.run.smithi037.stdout: crun-1.12-1.module_el8+804+f131391c.x86_64 2024-02-11T16:55:23.052 INFO:teuthology.orchestra.run.smithi037.stdout: netavark-2:1.9.0-1.module_el8+804+f131391c.x86_64 2024-02-11T16:55:23.052 INFO:teuthology.orchestra.run.smithi037.stdout: python3-podman-4.8.2-1.module_el8+804+f131391c.noarch 2024-02-11T16:55:23.052 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-02-11T16:55:23.053 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-02-11T16:55:23.053 INFO:teuthology.orchestra.run.smithi037.stdout: skopeo-2:1.14.0-2.module_el8+804+f131391c.x86_64 2024-02-11T16:55:23.053 INFO:teuthology.orchestra.run.smithi037.stdout: toolbox-0.0.99.4-8.module_el8+804+f131391c.x86_64 2024-02-11T16:55:23.053 INFO:teuthology.orchestra.run.smithi037.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-02-11T16:55:23.053 INFO:teuthology.orchestra.run.smithi037.stdout: yajl-2.1.0-12.el8.x86_64 2024-02-11T16:55:23.053 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:55:23.053 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-02-11T16:55:23.358 DEBUG:teuthology.parallel:result is None 2024-02-11T16:55:23.358 INFO:teuthology.run_tasks:Running task pexec... 2024-02-11T16:55:23.368 INFO:teuthology.task.pexec:Executing custom commands... 2024-02-11T16:55:23.369 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi037.front.sepia.ceph.com 2024-02-11T16:55:23.369 DEBUG:teuthology.orchestra.run.smithi037:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-11T16:55:23.369 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi178.front.sepia.ceph.com 2024-02-11T16:55:23.369 DEBUG:teuthology.orchestra.run.smithi178:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-11T16:55:23.574 DEBUG:teuthology.parallel:result is None 2024-02-11T16:55:23.579 DEBUG:teuthology.parallel:result is None 2024-02-11T16:55:23.579 INFO:teuthology.run_tasks:Running task install... 2024-02-11T16:55:23.588 DEBUG:teuthology.task.install:project ceph 2024-02-11T16:55:23.588 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5db25abe00e218b6920215c0bef70af7e47c7152'}} 2024-02-11T16:55:23.588 DEBUG:teuthology.task.install:config {'exclude_packages': ['ceph-volume'], 'tag': 'v16.2.4', 'flavor': 'default', 'sha1': '5db25abe00e218b6920215c0bef70af7e47c7152'} 2024-02-11T16:55:23.588 INFO:teuthology.task.install:Using flavor: default 2024-02-11T16:55:23.596 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-02-11T16:55:23.596 INFO:teuthology.task.install:extra packages: [] 2024-02-11T16:55:23.596 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5db25abe00e218b6920215c0bef70af7e47c7152', 'tag': 'v16.2.4', 'wait_for_package': False} 2024-02-11T16:55:23.597 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-11T16:55:23.597 INFO:teuthology.packaging:ref: None 2024-02-11T16:55:23.597 INFO:teuthology.packaging:tag: v16.2.4 2024-02-11T16:55:23.597 INFO:teuthology.packaging:branch: None 2024-02-11T16:55:23.597 INFO:teuthology.packaging:sha1: 5db25abe00e218b6920215c0bef70af7e47c7152 2024-02-11T16:55:23.780 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-11T16:55:23.780 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-11T16:55:23.784 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5db25abe00e218b6920215c0bef70af7e47c7152', 'tag': 'v16.2.4', 'wait_for_package': False} 2024-02-11T16:55:23.784 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-11T16:55:23.784 INFO:teuthology.packaging:ref: None 2024-02-11T16:55:23.784 INFO:teuthology.packaging:tag: v16.2.4 2024-02-11T16:55:23.784 INFO:teuthology.packaging:branch: None 2024-02-11T16:55:23.784 INFO:teuthology.packaging:sha1: 5db25abe00e218b6920215c0bef70af7e47c7152 2024-02-11T16:55:23.784 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-11T16:55:24.002 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2024-02-11T16:55:24.002 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2024-02-11T16:55:24.014 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2024-02-11T16:55:24.015 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2024-02-11T16:55:24.043 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-11T16:55:24.043 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-02-11T16:55:24.043 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-11T16:55:24.062 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-11T16:55:24.062 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-02-11T16:55:24.062 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-11T16:55:24.075 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-11T16:55:24.075 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-11T16:55:24.075 INFO:teuthology.packaging:ref: None 2024-02-11T16:55:24.075 INFO:teuthology.packaging:tag: v16.2.4 2024-02-11T16:55:24.076 INFO:teuthology.packaging:branch: None 2024-02-11T16:55:24.076 INFO:teuthology.packaging:sha1: 5db25abe00e218b6920215c0bef70af7e47c7152 2024-02-11T16:55:24.076 DEBUG:teuthology.orchestra.run.smithi037:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-11T16:55:24.091 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-11T16:55:24.091 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-11T16:55:24.091 INFO:teuthology.packaging:ref: None 2024-02-11T16:55:24.091 INFO:teuthology.packaging:tag: v16.2.4 2024-02-11T16:55:24.091 INFO:teuthology.packaging:branch: None 2024-02-11T16:55:24.091 INFO:teuthology.packaging:sha1: 5db25abe00e218b6920215c0bef70af7e47c7152 2024-02-11T16:55:24.092 DEBUG:teuthology.orchestra.run.smithi178:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-11T16:55:24.146 DEBUG:teuthology.orchestra.run.smithi037:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-11T16:55:24.159 DEBUG:teuthology.orchestra.run.smithi178:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-11T16:55:24.227 DEBUG:teuthology.orchestra.run.smithi037:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-11T16:55:24.236 DEBUG:teuthology.orchestra.run.smithi178:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-11T16:55:24.295 INFO:teuthology.orchestra.run.smithi037.stdout:check_obsoletes = 1 2024-02-11T16:55:24.297 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum clean all 2024-02-11T16:55:24.302 INFO:teuthology.orchestra.run.smithi178.stdout:check_obsoletes = 1 2024-02-11T16:55:24.303 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum clean all 2024-02-11T16:55:24.571 INFO:teuthology.orchestra.run.smithi178.stdout:76 files removed 2024-02-11T16:55:24.575 INFO:teuthology.orchestra.run.smithi037.stdout:76 files removed 2024-02-11T16:55:24.597 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-radosgw 2024-02-11T16:55:24.624 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-radosgw 2024-02-11T16:55:25.070 INFO:teuthology.orchestra.run.smithi178.stdout:ceph packages for x86_64 1.1 MB/s | 74 kB 00:00 2024-02-11T16:55:25.131 INFO:teuthology.orchestra.run.smithi037.stdout:ceph packages for x86_64 954 kB/s | 74 kB 00:00 2024-02-11T16:55:25.171 INFO:teuthology.orchestra.run.smithi178.stdout:ceph noarch packages 241 kB/s | 15 kB 00:00 2024-02-11T16:55:25.256 INFO:teuthology.orchestra.run.smithi037.stdout:ceph noarch packages 180 kB/s | 15 kB 00:00 2024-02-11T16:55:25.271 INFO:teuthology.orchestra.run.smithi178.stdout:ceph source packages 24 kB/s | 1.7 kB 00:00 2024-02-11T16:55:25.373 INFO:teuthology.orchestra.run.smithi037.stdout:ceph source packages 20 kB/s | 1.7 kB 00:00 2024-02-11T16:55:26.205 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 8 - AppStream 30 MB/s | 27 MB 00:00 2024-02-11T16:55:26.374 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS Stream 8 - AppStream 28 MB/s | 27 MB 00:00 2024-02-11T16:55:32.614 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 8 - BaseOS 6.3 MB/s | 10 MB 00:01 2024-02-11T16:55:32.875 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS Stream 8 - BaseOS 8.1 MB/s | 10 MB 00:01 2024-02-11T16:55:34.440 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 8 - Extras 75 kB/s | 18 kB 00:00 2024-02-11T16:55:34.907 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 8 - Extras common packages 19 kB/s | 6.9 kB 00:00 2024-02-11T16:55:35.109 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS Stream 8 - Extras 33 kB/s | 18 kB 00:00 2024-02-11T16:55:35.427 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS Stream 8 - Extras common packages 35 kB/s | 6.9 kB 00:00 2024-02-11T16:55:35.817 INFO:teuthology.orchestra.run.smithi178.stdout:CentOS Stream 8 - PowerTools 5.8 MB/s | 4.8 MB 00:00 2024-02-11T16:55:36.377 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS Stream 8 - PowerTools 5.6 MB/s | 4.8 MB 00:00 2024-02-11T16:55:36.843 INFO:teuthology.orchestra.run.smithi178.stdout:Copr repo for python3-asyncssh owned by ceph 27 kB/s | 3.5 kB 00:00 2024-02-11T16:55:37.528 INFO:teuthology.orchestra.run.smithi037.stdout:Copr repo for python3-asyncssh owned by ceph 22 kB/s | 3.5 kB 00:00 2024-02-11T16:55:37.911 INFO:teuthology.orchestra.run.smithi178.stdout:Extra Packages for Enterprise Linux 17 MB/s | 16 MB 00:00 2024-02-11T16:55:39.263 INFO:teuthology.orchestra.run.smithi037.stdout:Extra Packages for Enterprise Linux 9.9 MB/s | 16 MB 00:01 2024-02-11T16:55:41.686 INFO:teuthology.orchestra.run.smithi178.stdout:lab-extras 425 kB/s | 24 kB 00:00 2024-02-11T16:55:43.247 INFO:teuthology.orchestra.run.smithi037.stdout:lab-extras 478 kB/s | 24 kB 00:00 2024-02-11T16:55:45.182 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-02-11T16:55:45.185 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-02-11T16:55:45.185 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2024-02-11T16:55:45.185 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-02-11T16:55:45.185 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-02-11T16:55:45.185 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2024-02-11T16:55:45.185 INFO:teuthology.orchestra.run.smithi178.stdout:Upgrading: 2024-02-11T16:55:45.185 INFO:teuthology.orchestra.run.smithi178.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-02-11T16:55:45.186 INFO:teuthology.orchestra.run.smithi178.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2024-02-11T16:55:45.186 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2024-02-11T16:55:45.186 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2024-02-11T16:55:45.186 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2024-02-11T16:55:45.186 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2024-02-11T16:55:45.186 INFO:teuthology.orchestra.run.smithi178.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2024-02-11T16:55:45.186 INFO:teuthology.orchestra.run.smithi178.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2024-02-11T16:55:45.186 INFO:teuthology.orchestra.run.smithi178.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-02-11T16:55:45.186 INFO:teuthology.orchestra.run.smithi178.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-02-11T16:55:45.186 INFO:teuthology.orchestra.run.smithi178.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2024-02-11T16:55:45.187 INFO:teuthology.orchestra.run.smithi178.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-02-11T16:55:45.187 INFO:teuthology.orchestra.run.smithi178.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2024-02-11T16:55:45.187 INFO:teuthology.orchestra.run.smithi178.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-02-11T16:55:45.187 INFO:teuthology.orchestra.run.smithi178.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2024-02-11T16:55:45.187 INFO:teuthology.orchestra.run.smithi178.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-02-11T16:55:45.187 INFO:teuthology.orchestra.run.smithi178.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2024-02-11T16:55:45.187 INFO:teuthology.orchestra.run.smithi178.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-02-11T16:55:45.187 INFO:teuthology.orchestra.run.smithi178.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-02-11T16:55:45.187 INFO:teuthology.orchestra.run.smithi178.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2024-02-11T16:55:45.187 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2024-02-11T16:55:45.187 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2024-02-11T16:55:45.187 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2024-02-11T16:55:45.187 INFO:teuthology.orchestra.run.smithi178.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-02-11T16:55:45.187 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2024-02-11T16:55:45.187 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2024-02-11T16:55:45.188 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2024-02-11T16:55:45.188 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:55:45.188 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-02-11T16:55:45.188 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-02-11T16:55:45.188 INFO:teuthology.orchestra.run.smithi178.stdout:Install 25 Packages 2024-02-11T16:55:45.188 INFO:teuthology.orchestra.run.smithi178.stdout:Upgrade 2 Packages 2024-02-11T16:55:45.188 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:55:45.189 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 59 M 2024-02-11T16:55:45.189 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-02-11T16:55:47.038 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-02-11T16:55:47.041 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-02-11T16:55:47.041 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-02-11T16:55:47.041 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-02-11T16:55:47.041 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-02-11T16:55:47.041 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2024-02-11T16:55:47.041 INFO:teuthology.orchestra.run.smithi037.stdout:Upgrading: 2024-02-11T16:55:47.041 INFO:teuthology.orchestra.run.smithi037.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-02-11T16:55:47.041 INFO:teuthology.orchestra.run.smithi037.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2024-02-11T16:55:47.041 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-02-11T16:55:47.041 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2024-02-11T16:55:47.042 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2024-02-11T16:55:47.042 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2024-02-11T16:55:47.042 INFO:teuthology.orchestra.run.smithi037.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2024-02-11T16:55:47.042 INFO:teuthology.orchestra.run.smithi037.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2024-02-11T16:55:47.042 INFO:teuthology.orchestra.run.smithi037.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-02-11T16:55:47.042 INFO:teuthology.orchestra.run.smithi037.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-02-11T16:55:47.042 INFO:teuthology.orchestra.run.smithi037.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2024-02-11T16:55:47.042 INFO:teuthology.orchestra.run.smithi037.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-02-11T16:55:47.042 INFO:teuthology.orchestra.run.smithi037.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2024-02-11T16:55:47.042 INFO:teuthology.orchestra.run.smithi037.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-02-11T16:55:47.042 INFO:teuthology.orchestra.run.smithi037.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2024-02-11T16:55:47.042 INFO:teuthology.orchestra.run.smithi037.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-02-11T16:55:47.042 INFO:teuthology.orchestra.run.smithi037.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2024-02-11T16:55:47.042 INFO:teuthology.orchestra.run.smithi037.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-02-11T16:55:47.042 INFO:teuthology.orchestra.run.smithi037.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-02-11T16:55:47.043 INFO:teuthology.orchestra.run.smithi037.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2024-02-11T16:55:47.043 INFO:teuthology.orchestra.run.smithi037.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2024-02-11T16:55:47.043 INFO:teuthology.orchestra.run.smithi037.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2024-02-11T16:55:47.043 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2024-02-11T16:55:47.043 INFO:teuthology.orchestra.run.smithi037.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-02-11T16:55:47.043 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2024-02-11T16:55:47.043 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2024-02-11T16:55:47.043 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2024-02-11T16:55:47.043 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:55:47.043 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-02-11T16:55:47.043 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-02-11T16:55:47.043 INFO:teuthology.orchestra.run.smithi037.stdout:Install 25 Packages 2024-02-11T16:55:47.043 INFO:teuthology.orchestra.run.smithi037.stdout:Upgrade 2 Packages 2024-02-11T16:55:47.043 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:55:47.044 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 59 M 2024-02-11T16:55:47.044 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-02-11T16:55:47.328 INFO:teuthology.orchestra.run.smithi178.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 38 MB/s | 6.0 MB 00:00 2024-02-11T16:55:47.353 INFO:teuthology.orchestra.run.smithi178.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 1.0 MB/s | 25 kB 00:00 2024-02-11T16:55:47.395 INFO:teuthology.orchestra.run.smithi178.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 19 MB/s | 801 kB 00:00 2024-02-11T16:55:47.429 INFO:teuthology.orchestra.run.smithi178.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 14 MB/s | 486 kB 00:00 2024-02-11T16:55:47.490 INFO:teuthology.orchestra.run.smithi037.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 37 MB/s | 6.0 MB 00:00 2024-02-11T16:55:47.515 INFO:teuthology.orchestra.run.smithi037.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 1.0 MB/s | 25 kB 00:00 2024-02-11T16:55:47.529 INFO:teuthology.orchestra.run.smithi178.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 41 MB/s | 4.1 MB 00:00 2024-02-11T16:55:47.554 INFO:teuthology.orchestra.run.smithi178.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 1.8 MB/s | 44 kB 00:00 2024-02-11T16:55:47.557 INFO:teuthology.orchestra.run.smithi037.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 19 MB/s | 801 kB 00:00 2024-02-11T16:55:47.571 INFO:teuthology.orchestra.run.smithi178.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 4.7 MB/s | 80 kB 00:00 2024-02-11T16:55:47.590 INFO:teuthology.orchestra.run.smithi037.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 14 MB/s | 486 kB 00:00 2024-02-11T16:55:47.597 INFO:teuthology.orchestra.run.smithi178.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 8.3 MB/s | 213 kB 00:00 2024-02-11T16:55:47.691 INFO:teuthology.orchestra.run.smithi037.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 41 MB/s | 4.1 MB 00:00 2024-02-11T16:55:47.758 INFO:teuthology.orchestra.run.smithi037.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 669 kB/s | 44 kB 00:00 2024-02-11T16:55:47.764 INFO:teuthology.orchestra.run.smithi178.stdout:(9/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 21 MB/s | 13 MB 00:00 2024-02-11T16:55:47.783 INFO:teuthology.orchestra.run.smithi037.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 3.1 MB/s | 80 kB 00:00 2024-02-11T16:55:47.797 INFO:teuthology.orchestra.run.smithi178.stdout:(10/27): python3-rados-16.2.4-0.el8.x86_64.rpm 1.9 MB/s | 387 kB 00:00 2024-02-11T16:55:47.808 INFO:teuthology.orchestra.run.smithi037.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 8.3 MB/s | 213 kB 00:00 2024-02-11T16:55:47.823 INFO:teuthology.orchestra.run.smithi178.stdout:(11/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 6.1 MB/s | 368 kB 00:00 2024-02-11T16:55:47.842 INFO:teuthology.orchestra.run.smithi037.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 11 MB/s | 387 kB 00:00 2024-02-11T16:55:47.848 INFO:teuthology.orchestra.run.smithi178.stdout:(12/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 2.2 MB/s | 114 kB 00:00 2024-02-11T16:55:47.892 INFO:teuthology.orchestra.run.smithi037.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 7.2 MB/s | 368 kB 00:00 2024-02-11T16:55:47.907 INFO:teuthology.orchestra.run.smithi178.stdout:(13/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 1.7 MB/s | 145 kB 00:00 2024-02-11T16:55:47.917 INFO:teuthology.orchestra.run.smithi037.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 4.5 MB/s | 114 kB 00:00 2024-02-11T16:55:48.068 INFO:teuthology.orchestra.run.smithi037.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 17 MB/s | 13 MB 00:00 2024-02-11T16:55:48.093 INFO:teuthology.orchestra.run.smithi037.stdout:(13/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 829 kB/s | 145 kB 00:00 2024-02-11T16:55:48.135 INFO:teuthology.orchestra.run.smithi037.stdout:(14/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 6.1 MB/s | 259 kB 00:00 2024-02-11T16:55:48.149 INFO:teuthology.orchestra.run.smithi178.stdout:(14/27): ceph-common-16.2.4-0.el8.x86_64.rpm 25 MB/s | 24 MB 00:00 2024-02-11T16:55:48.165 INFO:teuthology.orchestra.run.smithi178.stdout:(15/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.0 MB/s | 259 kB 00:00 2024-02-11T16:55:48.168 INFO:teuthology.orchestra.run.smithi037.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 1.3 MB/s | 44 kB 00:00 2024-02-11T16:55:48.191 INFO:teuthology.orchestra.run.smithi178.stdout:(16/27): librdkafka-1.6.1-1.el8.x86_64.rpm 1.9 MB/s | 677 kB 00:00 2024-02-11T16:55:48.219 INFO:teuthology.orchestra.run.smithi037.stdout:(16/27): librdkafka-1.6.1-1.el8.x86_64.rpm 4.4 MB/s | 677 kB 00:00 2024-02-11T16:55:48.247 INFO:teuthology.orchestra.run.smithi178.stdout:(17/27): python3-prettytable-0.7.2-14.el8.noarc 450 kB/s | 44 kB 00:00 2024-02-11T16:55:48.325 INFO:teuthology.orchestra.run.smithi178.stdout:(18/27): librabbitmq-0.9.0-5.el8.x86_64.rpm 352 kB/s | 47 kB 00:00 2024-02-11T16:55:48.351 INFO:teuthology.orchestra.run.smithi178.stdout:(19/27): daxctl-libs-71.1-7.el8.x86_64.rpm 229 kB/s | 42 kB 00:00 2024-02-11T16:55:48.375 INFO:teuthology.orchestra.run.smithi178.stdout:(20/27): ndctl-libs-71.1-7.el8.x86_64.rpm 626 kB/s | 80 kB 00:00 2024-02-11T16:55:48.463 INFO:teuthology.orchestra.run.smithi037.stdout:(17/27): ceph-common-16.2.4-0.el8.x86_64.rpm 21 MB/s | 24 MB 00:01 2024-02-11T16:55:48.495 INFO:teuthology.orchestra.run.smithi037.stdout:(18/27): daxctl-libs-71.1-7.el8.x86_64.rpm 129 kB/s | 42 kB 00:00 2024-02-11T16:55:48.545 INFO:teuthology.orchestra.run.smithi037.stdout:(19/27): librabbitmq-0.9.0-5.el8.x86_64.rpm 144 kB/s | 47 kB 00:00 2024-02-11T16:55:48.570 INFO:teuthology.orchestra.run.smithi037.stdout:(20/27): ndctl-libs-71.1-7.el8.x86_64.rpm 751 kB/s | 80 kB 00:00 2024-02-11T16:55:48.637 INFO:teuthology.orchestra.run.smithi037.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 818 kB/s | 116 kB 00:00 2024-02-11T16:55:48.679 INFO:teuthology.orchestra.run.smithi037.stdout:(22/27): liboath-2.6.2-3.el8.x86_64.rpm 1.4 MB/s | 59 kB 00:00 2024-02-11T16:55:48.704 INFO:teuthology.orchestra.run.smithi037.stdout:(23/27): gperftools-libs-2.7-9.el8.x86_64.rpm 1.9 MB/s | 306 kB 00:00 2024-02-11T16:55:48.730 INFO:teuthology.orchestra.run.smithi037.stdout:(24/27): leveldb-1.22-1.el8.x86_64.rpm 1.1 MB/s | 181 kB 00:00 2024-02-11T16:55:48.746 INFO:teuthology.orchestra.run.smithi037.stdout:(25/27): libunwind-1.3.1-3.el8.x86_64.rpm 1.1 MB/s | 75 kB 00:00 2024-02-11T16:55:48.750 INFO:teuthology.orchestra.run.smithi178.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 273 kB/s | 116 kB 00:00 2024-02-11T16:55:48.776 INFO:teuthology.orchestra.run.smithi178.stdout:(22/27): leveldb-1.22-1.el8.x86_64.rpm 452 kB/s | 181 kB 00:00 2024-02-11T16:55:48.826 INFO:teuthology.orchestra.run.smithi178.stdout:(23/27): gperftools-libs-2.7-9.el8.x86_64.rpm 644 kB/s | 306 kB 00:00 2024-02-11T16:55:48.830 INFO:teuthology.orchestra.run.smithi037.stdout:(26/27): librados2-16.2.4-0.el8.x86_64.rpm 30 MB/s | 3.7 MB 00:00 2024-02-11T16:55:48.860 INFO:teuthology.orchestra.run.smithi178.stdout:(24/27): liboath-2.6.2-3.el8.x86_64.rpm 540 kB/s | 59 kB 00:00 2024-02-11T16:55:48.885 INFO:teuthology.orchestra.run.smithi178.stdout:(25/27): libunwind-1.3.1-3.el8.x86_64.rpm 688 kB/s | 75 kB 00:00 2024-02-11T16:55:48.888 INFO:teuthology.orchestra.run.smithi037.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 26 MB/s | 4.0 MB 00:00 2024-02-11T16:55:48.890 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-02-11T16:55:48.890 INFO:teuthology.orchestra.run.smithi037.stdout:Total 32 MB/s | 59 MB 00:01 2024-02-11T16:55:48.959 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-02-11T16:55:48.960 INFO:teuthology.orchestra.run.smithi178.stdout:(26/27): librados2-16.2.4-0.el8.x86_64.rpm 28 MB/s | 3.7 MB 00:00 2024-02-11T16:55:49.010 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-02-11T16:55:49.010 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-02-11T16:55:49.211 INFO:teuthology.orchestra.run.smithi178.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 12 MB/s | 4.0 MB 00:00 2024-02-11T16:55:49.213 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-02-11T16:55:49.213 INFO:teuthology.orchestra.run.smithi178.stdout:Total 15 MB/s | 59 MB 00:04 2024-02-11T16:55:49.284 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-02-11T16:55:49.332 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-02-11T16:55:49.332 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-02-11T16:55:49.360 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-02-11T16:55:49.363 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-02-11T16:55:49.691 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-02-11T16:55:49.694 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-02-11T16:55:50.531 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-02-11T16:55:50.808 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-02-11T16:55:50.848 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2024-02-11T16:55:51.022 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2024-02-11T16:55:51.045 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2024-02-11T16:55:51.193 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-02-11T16:55:51.243 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2024-02-11T16:55:51.383 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-02-11T16:55:51.613 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-02-11T16:55:51.776 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-02-11T16:55:51.784 INFO:teuthology.orchestra.run.smithi037.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2024-02-11T16:55:51.866 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2024-02-11T16:55:51.958 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2024-02-11T16:55:52.037 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2024-02-11T16:55:52.120 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2024-02-11T16:55:52.251 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-02-11T16:55:52.282 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2024-02-11T16:55:52.301 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-02-11T16:55:52.483 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-02-11T16:55:52.515 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/29 2024-02-11T16:55:52.548 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-02-11T16:55:52.775 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/29 2024-02-11T16:55:53.068 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/29 2024-02-11T16:55:53.200 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-02-11T16:55:53.284 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-02-11T16:55:53.366 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/29 2024-02-11T16:55:53.456 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2024-02-11T16:55:53.533 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-02-11T16:55:53.600 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-02-11T16:55:53.609 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2024-02-11T16:55:53.699 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2024-02-11T16:55:53.779 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2024-02-11T16:55:53.875 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2024-02-11T16:55:53.932 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2024-02-11T16:55:53.987 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2024-02-11T16:55:54.052 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2024-02-11T16:55:54.171 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2024-02-11T16:55:54.212 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2024-02-11T16:55:54.276 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2024-02-11T16:55:54.332 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2024-02-11T16:55:54.427 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-02-11T16:55:54.482 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2024-02-11T16:55:54.555 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2024-02-11T16:55:54.872 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-02-11T16:55:55.067 INFO:teuthology.orchestra.run.smithi037.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-02-11T16:55:55.150 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-02-11T16:55:55.175 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-02-11T16:55:55.313 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2024-02-11T16:55:55.530 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2024-02-11T16:55:55.608 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-02-11T16:55:55.650 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-02-11T16:55:55.709 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-02-11T16:55:55.832 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-02-11T16:55:55.929 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-02-11T16:55:56.111 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2024-02-11T16:55:56.309 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2024-02-11T16:55:56.356 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2024-02-11T16:55:56.465 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-02-11T16:55:56.523 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-02-11T16:55:56.526 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2024-02-11T16:55:56.724 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2024-02-11T16:55:56.739 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2024-02-11T16:55:56.940 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2024-02-11T16:55:56.942 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2024-02-11T16:55:57.089 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2024-02-11T16:55:57.316 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2024-02-11T16:55:59.659 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-11T16:55:59.833 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-11T16:55:59.901 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-11T16:56:00.065 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-11T16:56:00.643 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-11T16:56:00.788 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-11T16:56:00.899 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-02-11T16:56:00.996 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-02-11T16:56:01.100 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-02-11T16:56:01.150 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-02-11T16:56:01.200 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-02-11T16:56:01.364 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-02-11T16:56:20.506 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-02-11T16:56:20.506 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /sys 2024-02-11T16:56:20.506 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /proc 2024-02-11T16:56:20.506 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /dev 2024-02-11T16:56:20.506 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /run 2024-02-11T16:56:20.507 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /mnt 2024-02-11T16:56:20.507 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /var/tmp 2024-02-11T16:56:20.507 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /home 2024-02-11T16:56:20.507 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /tmp 2024-02-11T16:56:20.507 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /dev 2024-02-11T16:56:20.507 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:56:21.220 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-02-11T16:56:21.220 INFO:teuthology.orchestra.run.smithi037.stdout:skipping the directory /sys 2024-02-11T16:56:21.220 INFO:teuthology.orchestra.run.smithi037.stdout:skipping the directory /proc 2024-02-11T16:56:21.221 INFO:teuthology.orchestra.run.smithi037.stdout:skipping the directory /dev 2024-02-11T16:56:21.221 INFO:teuthology.orchestra.run.smithi037.stdout:skipping the directory /run 2024-02-11T16:56:21.221 INFO:teuthology.orchestra.run.smithi037.stdout:skipping the directory /mnt 2024-02-11T16:56:21.221 INFO:teuthology.orchestra.run.smithi037.stdout:skipping the directory /var/tmp 2024-02-11T16:56:21.221 INFO:teuthology.orchestra.run.smithi037.stdout:skipping the directory /home 2024-02-11T16:56:21.221 INFO:teuthology.orchestra.run.smithi037.stdout:skipping the directory /tmp 2024-02-11T16:56:21.221 INFO:teuthology.orchestra.run.smithi037.stdout:skipping the directory /dev 2024-02-11T16:56:21.221 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:56:21.846 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-02-11T16:56:21.920 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-02-11T16:56:21.920 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-02-11T16:56:22.163 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-02-11T16:56:22.163 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2024-02-11T16:56:22.605 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-02-11T16:56:22.665 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-02-11T16:56:22.666 INFO:teuthology.orchestra.run.smithi037.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-02-11T16:56:22.854 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2024-02-11T16:56:22.855 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2024-02-11T16:56:22.855 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2024-02-11T16:56:22.855 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2024-02-11T16:56:22.855 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2024-02-11T16:56:22.855 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2024-02-11T16:56:22.855 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2024-02-11T16:56:22.855 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2024-02-11T16:56:22.856 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2024-02-11T16:56:22.856 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2024-02-11T16:56:22.856 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2024-02-11T16:56:22.856 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2024-02-11T16:56:22.856 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2024-02-11T16:56:22.856 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2024-02-11T16:56:22.857 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2024-02-11T16:56:22.857 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 15/29 2024-02-11T16:56:22.857 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2024-02-11T16:56:22.857 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2024-02-11T16:56:22.857 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2024-02-11T16:56:22.858 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 19/29 2024-02-11T16:56:22.858 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2024-02-11T16:56:22.858 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2024-02-11T16:56:22.858 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2024-02-11T16:56:22.858 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2024-02-11T16:56:22.858 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2024-02-11T16:56:22.858 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2024-02-11T16:56:22.859 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2024-02-11T16:56:22.859 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2024-02-11T16:56:22.859 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2024-02-11T16:56:22.891 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-02-11T16:56:22.891 INFO:teuthology.orchestra.run.smithi037.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2024-02-11T16:56:23.563 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2024-02-11T16:56:23.563 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2024-02-11T16:56:23.563 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2024-02-11T16:56:23.563 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2024-02-11T16:56:23.563 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2024-02-11T16:56:23.563 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2024-02-11T16:56:23.563 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2024-02-11T16:56:23.563 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2024-02-11T16:56:23.563 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2024-02-11T16:56:23.564 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2024-02-11T16:56:23.564 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2024-02-11T16:56:23.564 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2024-02-11T16:56:23.564 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2024-02-11T16:56:23.564 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2024-02-11T16:56:23.564 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2024-02-11T16:56:23.564 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 15/29 2024-02-11T16:56:23.564 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2024-02-11T16:56:23.564 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2024-02-11T16:56:23.564 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2024-02-11T16:56:23.564 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 19/29 2024-02-11T16:56:23.564 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2024-02-11T16:56:23.564 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2024-02-11T16:56:23.565 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2024-02-11T16:56:23.565 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2024-02-11T16:56:23.565 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2024-02-11T16:56:23.565 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2024-02-11T16:56:23.565 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2024-02-11T16:56:23.565 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2024-02-11T16:56:23.565 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2024-02-11T16:56:23.845 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2024-02-11T16:56:23.845 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:56:23.845 INFO:teuthology.orchestra.run.smithi178.stdout:Upgraded: 2024-02-11T16:56:23.846 INFO:teuthology.orchestra.run.smithi178.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2024-02-11T16:56:23.846 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-02-11T16:56:23.846 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2024-02-11T16:56:23.846 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2024-02-11T16:56:23.846 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2024-02-11T16:56:23.846 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2024-02-11T16:56:23.846 INFO:teuthology.orchestra.run.smithi178.stdout: daxctl-libs-71.1-7.el8.x86_64 2024-02-11T16:56:23.846 INFO:teuthology.orchestra.run.smithi178.stdout: fmt-6.2.1-1.el8.x86_64 2024-02-11T16:56:23.846 INFO:teuthology.orchestra.run.smithi178.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-02-11T16:56:23.846 INFO:teuthology.orchestra.run.smithi178.stdout: leveldb-1.22-1.el8.x86_64 2024-02-11T16:56:23.846 INFO:teuthology.orchestra.run.smithi178.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2024-02-11T16:56:23.846 INFO:teuthology.orchestra.run.smithi178.stdout: liboath-2.6.2-3.el8.x86_64 2024-02-11T16:56:23.846 INFO:teuthology.orchestra.run.smithi178.stdout: libpmemobj-1.6.1-1.el8.x86_64 2024-02-11T16:56:23.846 INFO:teuthology.orchestra.run.smithi178.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-02-11T16:56:23.847 INFO:teuthology.orchestra.run.smithi178.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2024-02-11T16:56:23.847 INFO:teuthology.orchestra.run.smithi178.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-02-11T16:56:23.847 INFO:teuthology.orchestra.run.smithi178.stdout: librgw2-2:16.2.4-0.el8.x86_64 2024-02-11T16:56:23.847 INFO:teuthology.orchestra.run.smithi178.stdout: libunwind-1.3.1-3.el8.x86_64 2024-02-11T16:56:23.847 INFO:teuthology.orchestra.run.smithi178.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-02-11T16:56:23.847 INFO:teuthology.orchestra.run.smithi178.stdout: ndctl-libs-71.1-7.el8.x86_64 2024-02-11T16:56:23.847 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2024-02-11T16:56:23.847 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2024-02-11T16:56:23.847 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2024-02-11T16:56:23.847 INFO:teuthology.orchestra.run.smithi178.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-02-11T16:56:23.847 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2024-02-11T16:56:23.847 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2024-02-11T16:56:23.847 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2024-02-11T16:56:23.848 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:56:23.848 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-02-11T16:56:24.158 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-test 2024-02-11T16:56:24.432 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2024-02-11T16:56:24.432 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:56:24.432 INFO:teuthology.orchestra.run.smithi037.stdout:Upgraded: 2024-02-11T16:56:24.433 INFO:teuthology.orchestra.run.smithi037.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2024-02-11T16:56:24.433 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-02-11T16:56:24.433 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2024-02-11T16:56:24.433 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2024-02-11T16:56:24.433 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2024-02-11T16:56:24.433 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2024-02-11T16:56:24.433 INFO:teuthology.orchestra.run.smithi037.stdout: daxctl-libs-71.1-7.el8.x86_64 2024-02-11T16:56:24.433 INFO:teuthology.orchestra.run.smithi037.stdout: fmt-6.2.1-1.el8.x86_64 2024-02-11T16:56:24.433 INFO:teuthology.orchestra.run.smithi037.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-02-11T16:56:24.433 INFO:teuthology.orchestra.run.smithi037.stdout: leveldb-1.22-1.el8.x86_64 2024-02-11T16:56:24.433 INFO:teuthology.orchestra.run.smithi037.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2024-02-11T16:56:24.433 INFO:teuthology.orchestra.run.smithi037.stdout: liboath-2.6.2-3.el8.x86_64 2024-02-11T16:56:24.433 INFO:teuthology.orchestra.run.smithi037.stdout: libpmemobj-1.6.1-1.el8.x86_64 2024-02-11T16:56:24.434 INFO:teuthology.orchestra.run.smithi037.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-02-11T16:56:24.434 INFO:teuthology.orchestra.run.smithi037.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2024-02-11T16:56:24.434 INFO:teuthology.orchestra.run.smithi037.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-02-11T16:56:24.434 INFO:teuthology.orchestra.run.smithi037.stdout: librgw2-2:16.2.4-0.el8.x86_64 2024-02-11T16:56:24.434 INFO:teuthology.orchestra.run.smithi037.stdout: libunwind-1.3.1-3.el8.x86_64 2024-02-11T16:56:24.434 INFO:teuthology.orchestra.run.smithi037.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-02-11T16:56:24.434 INFO:teuthology.orchestra.run.smithi037.stdout: ndctl-libs-71.1-7.el8.x86_64 2024-02-11T16:56:24.434 INFO:teuthology.orchestra.run.smithi037.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2024-02-11T16:56:24.434 INFO:teuthology.orchestra.run.smithi037.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2024-02-11T16:56:24.434 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2024-02-11T16:56:24.434 INFO:teuthology.orchestra.run.smithi037.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-02-11T16:56:24.434 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2024-02-11T16:56:24.434 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2024-02-11T16:56:24.435 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2024-02-11T16:56:24.435 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:56:24.435 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-02-11T16:56:24.720 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-test 2024-02-11T16:56:24.916 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:00:43 ago on Sun 11 Feb 2024 04:55:41 PM UTC. 2024-02-11T16:56:25.511 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:00:42 ago on Sun 11 Feb 2024 04:55:43 PM UTC. 2024-02-11T16:56:26.422 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-02-11T16:56:26.423 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-02-11T16:56:26.423 INFO:teuthology.orchestra.run.smithi178.stdout: Package Architecture Version Repository Size 2024-02-11T16:56:26.423 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-02-11T16:56:26.423 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-02-11T16:56:26.423 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2024-02-11T16:56:26.423 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2024-02-11T16:56:26.423 INFO:teuthology.orchestra.run.smithi178.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-02-11T16:56:26.423 INFO:teuthology.orchestra.run.smithi178.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2024-02-11T16:56:26.423 INFO:teuthology.orchestra.run.smithi178.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-02-11T16:56:26.423 INFO:teuthology.orchestra.run.smithi178.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-02-11T16:56:26.423 INFO:teuthology.orchestra.run.smithi178.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-02-11T16:56:26.423 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:56:26.424 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-02-11T16:56:26.424 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-02-11T16:56:26.424 INFO:teuthology.orchestra.run.smithi178.stdout:Install 6 Packages 2024-02-11T16:56:26.424 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:56:26.424 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 53 M 2024-02-11T16:56:26.424 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 229 M 2024-02-11T16:56:26.425 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-02-11T16:56:26.678 INFO:teuthology.orchestra.run.smithi178.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 2.4 MB/s | 169 kB 00:00 2024-02-11T16:56:26.703 INFO:teuthology.orchestra.run.smithi178.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 2.1 MB/s | 203 kB 00:00 2024-02-11T16:56:26.754 INFO:teuthology.orchestra.run.smithi178.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 6.3 MB/s | 323 kB 00:00 2024-02-11T16:56:26.779 INFO:teuthology.orchestra.run.smithi178.stdout:(4/6): oniguruma-6.8.2-3.el8.x86_64.rpm 1.8 MB/s | 188 kB 00:00 2024-02-11T16:56:26.804 INFO:teuthology.orchestra.run.smithi178.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.4 MB/s | 71 kB 00:00 2024-02-11T16:56:27.130 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-02-11T16:56:27.131 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-02-11T16:56:27.131 INFO:teuthology.orchestra.run.smithi037.stdout: Package Architecture Version Repository Size 2024-02-11T16:56:27.131 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-02-11T16:56:27.131 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-02-11T16:56:27.131 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2024-02-11T16:56:27.131 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-02-11T16:56:27.132 INFO:teuthology.orchestra.run.smithi037.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-02-11T16:56:27.132 INFO:teuthology.orchestra.run.smithi037.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2024-02-11T16:56:27.132 INFO:teuthology.orchestra.run.smithi037.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-02-11T16:56:27.132 INFO:teuthology.orchestra.run.smithi037.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-02-11T16:56:27.132 INFO:teuthology.orchestra.run.smithi037.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-02-11T16:56:27.132 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:56:27.132 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-02-11T16:56:27.132 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-02-11T16:56:27.132 INFO:teuthology.orchestra.run.smithi037.stdout:Install 6 Packages 2024-02-11T16:56:27.132 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:56:27.133 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 53 M 2024-02-11T16:56:27.133 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 229 M 2024-02-11T16:56:27.133 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-02-11T16:56:27.223 INFO:teuthology.orchestra.run.smithi037.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 3.4 MB/s | 169 kB 00:00 2024-02-11T16:56:27.280 INFO:teuthology.orchestra.run.smithi037.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 2.7 MB/s | 203 kB 00:00 2024-02-11T16:56:27.324 INFO:teuthology.orchestra.run.smithi037.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 4.2 MB/s | 323 kB 00:00 2024-02-11T16:56:27.357 INFO:teuthology.orchestra.run.smithi037.stdout:(4/6): oniguruma-6.8.2-3.el8.x86_64.rpm 1.4 MB/s | 188 kB 00:00 2024-02-11T16:56:27.413 INFO:teuthology.orchestra.run.smithi178.stdout:(6/6): ceph-test-16.2.4-0.el8.x86_64.rpm 65 MB/s | 52 MB 00:00 2024-02-11T16:56:27.415 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-02-11T16:56:27.415 INFO:teuthology.orchestra.run.smithi178.stdout:Total 53 MB/s | 53 MB 00:00 2024-02-11T16:56:27.432 INFO:teuthology.orchestra.run.smithi037.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 652 kB/s | 71 kB 00:00 2024-02-11T16:56:27.457 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-02-11T16:56:27.479 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-02-11T16:56:27.479 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-02-11T16:56:27.789 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-02-11T16:56:27.792 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-02-11T16:56:28.024 INFO:teuthology.orchestra.run.smithi037.stdout:(6/6): ceph-test-16.2.4-0.el8.x86_64.rpm 61 MB/s | 52 MB 00:00 2024-02-11T16:56:28.027 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-02-11T16:56:28.027 INFO:teuthology.orchestra.run.smithi037.stdout:Total 59 MB/s | 53 MB 00:00 2024-02-11T16:56:28.070 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-02-11T16:56:28.093 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-02-11T16:56:28.093 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-02-11T16:56:28.292 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-02-11T16:56:28.429 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-02-11T16:56:28.432 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-02-11T16:56:28.512 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-02-11T16:56:28.699 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-02-11T16:56:28.851 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-11T16:56:28.957 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-11T16:56:28.968 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-02-11T16:56:29.125 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-02-11T16:56:29.168 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-02-11T16:56:29.267 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-02-11T16:56:29.358 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-02-11T16:56:29.488 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-11T16:56:29.571 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-11T16:56:29.748 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-02-11T16:56:29.896 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-02-11T16:56:34.075 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-02-11T16:56:34.340 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-02-11T16:56:34.634 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-02-11T16:56:34.634 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2024-02-11T16:56:34.634 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2024-02-11T16:56:34.634 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-02-11T16:56:34.634 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-02-11T16:56:34.634 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-02-11T16:56:35.060 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-02-11T16:56:35.075 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-02-11T16:56:35.075 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:56:35.075 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-02-11T16:56:35.075 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-02-11T16:56:35.075 INFO:teuthology.orchestra.run.smithi178.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-3.el8.x86_64 2024-02-11T16:56:35.075 INFO:teuthology.orchestra.run.smithi178.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2024-02-11T16:56:35.075 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:56:35.075 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-02-11T16:56:35.269 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-02-11T16:56:35.305 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph 2024-02-11T16:56:35.568 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-02-11T16:56:35.568 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2024-02-11T16:56:35.568 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2024-02-11T16:56:35.568 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-02-11T16:56:35.568 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-02-11T16:56:35.568 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-02-11T16:56:35.995 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-02-11T16:56:35.995 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:56:35.995 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-02-11T16:56:35.996 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-02-11T16:56:35.996 INFO:teuthology.orchestra.run.smithi037.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-3.el8.x86_64 2024-02-11T16:56:35.996 INFO:teuthology.orchestra.run.smithi037.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2024-02-11T16:56:35.996 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:56:35.996 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-02-11T16:56:36.067 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:00:55 ago on Sun 11 Feb 2024 04:55:41 PM UTC. 2024-02-11T16:56:36.300 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph 2024-02-11T16:56:37.082 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:00:54 ago on Sun 11 Feb 2024 04:55:43 PM UTC. 2024-02-11T16:56:37.580 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-02-11T16:56:37.583 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-02-11T16:56:37.583 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2024-02-11T16:56:37.584 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-02-11T16:56:37.584 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-02-11T16:56:37.584 INFO:teuthology.orchestra.run.smithi178.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2024-02-11T16:56:37.584 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2024-02-11T16:56:37.584 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2024-02-11T16:56:37.584 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2024-02-11T16:56:37.584 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2024-02-11T16:56:37.584 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2024-02-11T16:56:37.584 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2024-02-11T16:56:37.584 INFO:teuthology.orchestra.run.smithi178.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-02-11T16:56:37.584 INFO:teuthology.orchestra.run.smithi178.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-02-11T16:56:37.584 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-02-11T16:56:37.585 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-02-11T16:56:37.585 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-02-11T16:56:37.585 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-02-11T16:56:37.585 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-02-11T16:56:37.585 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-02-11T16:56:37.585 INFO:teuthology.orchestra.run.smithi178.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-02-11T16:56:37.585 INFO:teuthology.orchestra.run.smithi178.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-02-11T16:56:37.585 INFO:teuthology.orchestra.run.smithi178.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-02-11T16:56:37.585 INFO:teuthology.orchestra.run.smithi178.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-02-11T16:56:37.585 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-02-11T16:56:37.585 INFO:teuthology.orchestra.run.smithi178.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-02-11T16:56:37.585 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-02-11T16:56:37.586 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-02-11T16:56:37.586 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-02-11T16:56:37.586 INFO:teuthology.orchestra.run.smithi178.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-02-11T16:56:37.586 INFO:teuthology.orchestra.run.smithi178.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-02-11T16:56:37.586 INFO:teuthology.orchestra.run.smithi178.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-02-11T16:56:37.586 INFO:teuthology.orchestra.run.smithi178.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-02-11T16:56:37.586 INFO:teuthology.orchestra.run.smithi178.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-02-11T16:56:37.586 INFO:teuthology.orchestra.run.smithi178.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-02-11T16:56:37.586 INFO:teuthology.orchestra.run.smithi178.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-02-11T16:56:37.586 INFO:teuthology.orchestra.run.smithi178.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-02-11T16:56:37.586 INFO:teuthology.orchestra.run.smithi178.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-02-11T16:56:37.586 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:56:37.586 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-02-11T16:56:37.586 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-02-11T16:56:37.587 INFO:teuthology.orchestra.run.smithi178.stdout:Install 32 Packages 2024-02-11T16:56:37.587 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:56:37.587 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 31 M 2024-02-11T16:56:37.587 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 111 M 2024-02-11T16:56:37.587 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-02-11T16:56:38.214 INFO:teuthology.orchestra.run.smithi178.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 123 kB/s | 6.3 kB 00:00 2024-02-11T16:56:38.264 INFO:teuthology.orchestra.run.smithi178.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-02-11T16:56:38.331 INFO:teuthology.orchestra.run.smithi178.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 14 MB/s | 2.4 MB 00:00 2024-02-11T16:56:38.365 INFO:teuthology.orchestra.run.smithi178.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 6.5 MB/s | 220 kB 00:00 2024-02-11T16:56:38.440 INFO:teuthology.orchestra.run.smithi178.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 22 MB/s | 5.0 MB 00:00 2024-02-11T16:56:38.465 INFO:teuthology.orchestra.run.smithi178.stdout:(6/32): python3-mako-1.0.6-14.el8.noarch.rpm 1.5 MB/s | 157 kB 00:00 2024-02-11T16:56:38.507 INFO:teuthology.orchestra.run.smithi178.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 2.4 MB/s | 103 kB 00:00 2024-02-11T16:56:38.525 INFO:teuthology.orchestra.run.smithi178.stdout:(8/32): python3-markupsafe-0.23-19.el8.x86_64.r 461 kB/s | 39 kB 00:00 2024-02-11T16:56:38.541 INFO:teuthology.orchestra.run.smithi178.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 1.7 MB/s | 57 kB 00:00 2024-02-11T16:56:38.583 INFO:teuthology.orchestra.run.smithi178.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 7.7 MB/s | 457 kB 00:00 2024-02-11T16:56:38.709 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-02-11T16:56:38.713 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-02-11T16:56:38.713 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-02-11T16:56:38.713 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-02-11T16:56:38.713 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-02-11T16:56:38.713 INFO:teuthology.orchestra.run.smithi037.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2024-02-11T16:56:38.713 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-02-11T16:56:38.714 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2024-02-11T16:56:38.714 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2024-02-11T16:56:38.714 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2024-02-11T16:56:38.714 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2024-02-11T16:56:38.714 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2024-02-11T16:56:38.714 INFO:teuthology.orchestra.run.smithi037.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-02-11T16:56:38.714 INFO:teuthology.orchestra.run.smithi037.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-02-11T16:56:38.714 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-02-11T16:56:38.714 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-02-11T16:56:38.714 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-02-11T16:56:38.714 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-02-11T16:56:38.714 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-02-11T16:56:38.714 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-02-11T16:56:38.714 INFO:teuthology.orchestra.run.smithi037.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-02-11T16:56:38.714 INFO:teuthology.orchestra.run.smithi037.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-02-11T16:56:38.714 INFO:teuthology.orchestra.run.smithi037.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-02-11T16:56:38.715 INFO:teuthology.orchestra.run.smithi037.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-02-11T16:56:38.715 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-02-11T16:56:38.715 INFO:teuthology.orchestra.run.smithi037.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-02-11T16:56:38.715 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-02-11T16:56:38.715 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-02-11T16:56:38.715 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-02-11T16:56:38.715 INFO:teuthology.orchestra.run.smithi037.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-02-11T16:56:38.715 INFO:teuthology.orchestra.run.smithi037.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-02-11T16:56:38.715 INFO:teuthology.orchestra.run.smithi037.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-02-11T16:56:38.715 INFO:teuthology.orchestra.run.smithi037.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-02-11T16:56:38.715 INFO:teuthology.orchestra.run.smithi037.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-02-11T16:56:38.715 INFO:teuthology.orchestra.run.smithi037.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-02-11T16:56:38.715 INFO:teuthology.orchestra.run.smithi037.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-02-11T16:56:38.715 INFO:teuthology.orchestra.run.smithi037.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-02-11T16:56:38.715 INFO:teuthology.orchestra.run.smithi037.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-02-11T16:56:38.716 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:56:38.716 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-02-11T16:56:38.716 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-02-11T16:56:38.716 INFO:teuthology.orchestra.run.smithi037.stdout:Install 32 Packages 2024-02-11T16:56:38.716 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:56:38.716 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 31 M 2024-02-11T16:56:38.717 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 111 M 2024-02-11T16:56:38.717 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-02-11T16:56:38.809 INFO:teuthology.orchestra.run.smithi178.stdout:(11/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 33 MB/s | 18 MB 00:00 2024-02-11T16:56:38.893 INFO:teuthology.orchestra.run.smithi178.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 709 kB/s | 249 kB 00:00 2024-02-11T16:56:38.951 INFO:teuthology.orchestra.run.smithi178.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 770 kB/s | 109 kB 00:00 2024-02-11T16:56:39.010 INFO:teuthology.orchestra.run.smithi178.stdout:(14/32): python3-cryptography-3.2.1-7.el8.x86_6 1.5 MB/s | 641 kB 00:00 2024-02-11T16:56:39.210 INFO:teuthology.orchestra.run.smithi178.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 137 kB/s | 44 kB 00:00 2024-02-11T16:56:39.334 INFO:teuthology.orchestra.run.smithi037.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 125 kB/s | 6.3 kB 00:00 2024-02-11T16:56:39.393 INFO:teuthology.orchestra.run.smithi037.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-02-11T16:56:39.427 INFO:teuthology.orchestra.run.smithi178.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 389 kB/s | 185 kB 00:00 2024-02-11T16:56:39.451 INFO:teuthology.orchestra.run.smithi037.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 14 MB/s | 2.4 MB 00:00 2024-02-11T16:56:39.477 INFO:teuthology.orchestra.run.smithi037.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 8.6 MB/s | 220 kB 00:00 2024-02-11T16:56:39.477 INFO:teuthology.orchestra.run.smithi178.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 369 kB/s | 173 kB 00:00 2024-02-11T16:56:39.527 INFO:teuthology.orchestra.run.smithi178.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 111 kB/s | 11 kB 00:00 2024-02-11T16:56:39.552 INFO:teuthology.orchestra.run.smithi178.stdout:(19/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.1 MB/s | 384 kB 00:00 2024-02-11T16:56:39.578 INFO:teuthology.orchestra.run.smithi178.stdout:(20/32): python3-jaraco-functools-2.0-4.el8.noa 175 kB/s | 18 kB 00:00 2024-02-11T16:56:39.588 INFO:teuthology.orchestra.run.smithi037.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 20 MB/s | 5.0 MB 00:00 2024-02-11T16:56:39.619 INFO:teuthology.orchestra.run.smithi037.stdout:(6/32): python3-mako-1.0.6-14.el8.noarch.rpm 1.1 MB/s | 157 kB 00:00 2024-02-11T16:56:39.636 INFO:teuthology.orchestra.run.smithi178.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 448 kB/s | 49 kB 00:00 2024-02-11T16:56:39.653 INFO:teuthology.orchestra.run.smithi037.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 3.0 MB/s | 103 kB 00:00 2024-02-11T16:56:39.661 INFO:teuthology.orchestra.run.smithi178.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 539 kB/s | 59 kB 00:00 2024-02-11T16:56:39.670 INFO:teuthology.orchestra.run.smithi037.stdout:(8/32): python3-markupsafe-0.23-19.el8.x86_64.r 473 kB/s | 39 kB 00:00 2024-02-11T16:56:39.687 INFO:teuthology.orchestra.run.smithi037.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 1.7 MB/s | 57 kB 00:00 2024-02-11T16:56:39.737 INFO:teuthology.orchestra.run.smithi178.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 156 kB/s | 16 kB 00:00 2024-02-11T16:56:39.745 INFO:teuthology.orchestra.run.smithi037.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 5.9 MB/s | 457 kB 00:00 2024-02-11T16:56:39.762 INFO:teuthology.orchestra.run.smithi178.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 187 kB/s | 19 kB 00:00 2024-02-11T16:56:39.787 INFO:teuthology.orchestra.run.smithi178.stdout:(25/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.3 MB/s | 283 kB 00:00 2024-02-11T16:56:39.838 INFO:teuthology.orchestra.run.smithi178.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 238 kB/s | 24 kB 00:00 2024-02-11T16:56:39.863 INFO:teuthology.orchestra.run.smithi178.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 291 kB/s | 29 kB 00:00 2024-02-11T16:56:39.888 INFO:teuthology.orchestra.run.smithi178.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 274 kB/s | 27 kB 00:00 2024-02-11T16:56:39.963 INFO:teuthology.orchestra.run.smithi037.stdout:(11/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 32 MB/s | 18 MB 00:00 2024-02-11T16:56:39.988 INFO:teuthology.orchestra.run.smithi037.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 827 kB/s | 249 kB 00:00 2024-02-11T16:56:39.997 INFO:teuthology.orchestra.run.smithi178.stdout:(29/32): python3-webtest-2.0.33-1.el8.noarch.rp 787 kB/s | 85 kB 00:00 2024-02-11T16:56:40.022 INFO:teuthology.orchestra.run.smithi178.stdout:(30/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.5 MB/s | 251 kB 00:00 2024-02-11T16:56:40.047 INFO:teuthology.orchestra.run.smithi178.stdout:(31/32): python3-waitress-1.4.3-1.el8.noarch.rp 1.2 MB/s | 255 kB 00:00 2024-02-11T16:56:40.097 INFO:teuthology.orchestra.run.smithi178.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 229 kB/s | 23 kB 00:00 2024-02-11T16:56:40.098 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-02-11T16:56:40.099 INFO:teuthology.orchestra.run.smithi178.stdout:Total 12 MB/s | 31 MB 00:02 2024-02-11T16:56:40.113 INFO:teuthology.orchestra.run.smithi037.stdout:(13/32): python3-cryptography-3.2.1-7.el8.x86_6 1.7 MB/s | 641 kB 00:00 2024-02-11T16:56:40.138 INFO:teuthology.orchestra.run.smithi037.stdout:(14/32): python3-pycparser-2.14-14.el8.noarch.r 622 kB/s | 109 kB 00:00 2024-02-11T16:56:40.180 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-02-11T16:56:40.219 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-02-11T16:56:40.219 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-02-11T16:56:40.505 INFO:teuthology.orchestra.run.smithi037.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 84 kB/s | 44 kB 00:00 2024-02-11T16:56:40.547 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-02-11T16:56:40.550 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-02-11T16:56:40.745 INFO:teuthology.orchestra.run.smithi037.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 293 kB/s | 185 kB 00:00 2024-02-11T16:56:40.772 INFO:teuthology.orchestra.run.smithi037.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 272 kB/s | 173 kB 00:00 2024-02-11T16:56:40.839 INFO:teuthology.orchestra.run.smithi037.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 118 kB/s | 11 kB 00:00 2024-02-11T16:56:40.881 INFO:teuthology.orchestra.run.smithi037.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 162 kB/s | 18 kB 00:00 2024-02-11T16:56:40.940 INFO:teuthology.orchestra.run.smithi037.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 486 kB/s | 49 kB 00:00 2024-02-11T16:56:40.982 INFO:teuthology.orchestra.run.smithi037.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 586 kB/s | 59 kB 00:00 2024-02-11T16:56:41.082 INFO:teuthology.orchestra.run.smithi037.stdout:(22/32): python3-portend-2.6-1.el8.noarch.rpm 156 kB/s | 16 kB 00:00 2024-02-11T16:56:41.107 INFO:teuthology.orchestra.run.smithi037.stdout:(23/32): python3-cherrypy-18.4.0-1.el8.noarch.r 639 kB/s | 384 kB 00:00 2024-02-11T16:56:41.183 INFO:teuthology.orchestra.run.smithi037.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 187 kB/s | 19 kB 00:00 2024-02-11T16:56:41.208 INFO:teuthology.orchestra.run.smithi037.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 239 kB/s | 24 kB 00:00 2024-02-11T16:56:41.283 INFO:teuthology.orchestra.run.smithi037.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 292 kB/s | 29 kB 00:00 2024-02-11T16:56:41.333 INFO:teuthology.orchestra.run.smithi037.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 220 kB/s | 27 kB 00:00 2024-02-11T16:56:41.375 INFO:teuthology.orchestra.run.smithi037.stdout:(28/32): python3-pecan-1.3.2-9.el8.noarch.rpm 650 kB/s | 283 kB 00:00 2024-02-11T16:56:41.567 INFO:teuthology.orchestra.run.smithi037.stdout:(29/32): python3-webtest-2.0.33-1.el8.noarch.rp 446 kB/s | 85 kB 00:00 2024-02-11T16:56:41.626 INFO:teuthology.orchestra.run.smithi037.stdout:(30/32): python3-waitress-1.4.3-1.el8.noarch.rp 745 kB/s | 255 kB 00:00 2024-02-11T16:56:41.669 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-02-11T16:56:41.676 INFO:teuthology.orchestra.run.smithi037.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 212 kB/s | 23 kB 00:00 2024-02-11T16:56:41.701 INFO:teuthology.orchestra.run.smithi037.stdout:(32/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 684 kB/s | 251 kB 00:00 2024-02-11T16:56:41.705 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-02-11T16:56:41.705 INFO:teuthology.orchestra.run.smithi037.stdout:Total 10 MB/s | 31 MB 00:02 2024-02-11T16:56:41.789 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-02-11T16:56:41.831 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-02-11T16:56:41.831 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-02-11T16:56:41.884 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-02-11T16:56:42.090 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-02-11T16:56:42.157 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-02-11T16:56:42.160 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-02-11T16:56:42.363 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-02-11T16:56:42.572 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-02-11T16:56:42.737 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-02-11T16:56:42.951 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-02-11T16:56:43.118 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-02-11T16:56:43.276 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-02-11T16:56:43.279 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-02-11T16:56:43.471 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-02-11T16:56:43.497 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-02-11T16:56:43.678 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-02-11T16:56:43.692 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-02-11T16:56:43.872 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-02-11T16:56:43.926 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-02-11T16:56:44.101 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-02-11T16:56:44.130 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-02-11T16:56:44.299 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-02-11T16:56:44.465 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-02-11T16:56:44.506 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-02-11T16:56:44.631 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-02-11T16:56:44.778 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-02-11T16:56:44.834 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-02-11T16:56:44.956 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-02-11T16:56:45.020 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-02-11T16:56:45.202 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-02-11T16:56:45.224 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-02-11T16:56:45.383 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-11T16:56:45.437 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-02-11T16:56:45.678 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-02-11T16:56:45.680 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-02-11T16:56:45.888 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-02-11T16:56:46.038 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-02-11T16:56:46.051 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-02-11T16:56:46.243 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-02-11T16:56:46.282 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-02-11T16:56:46.444 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-02-11T16:56:46.520 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-02-11T16:56:46.714 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-02-11T16:56:46.752 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-02-11T16:56:46.871 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-11T16:56:46.980 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-02-11T16:56:47.187 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-02-11T16:56:47.328 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-02-11T16:56:47.493 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-02-11T16:56:47.655 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-02-11T16:56:47.690 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-02-11T16:56:47.931 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-02-11T16:56:48.004 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2024-02-11T16:56:48.170 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-02-11T16:56:48.241 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-02-11T16:56:48.432 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-02-11T16:56:48.632 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-02-11T16:56:48.977 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-02-11T16:56:49.250 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-02-11T16:56:49.682 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2024-02-11T16:56:49.774 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-02-11T16:56:49.868 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-02-11T16:56:50.016 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-02-11T16:56:50.449 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-02-11T16:56:50.707 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-02-11T16:56:50.930 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-02-11T16:56:51.131 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-02-11T16:56:51.148 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-02-11T16:56:51.263 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2024-02-11T16:56:51.500 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-02-11T16:56:51.665 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2024-02-11T16:56:51.665 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2024-02-11T16:56:51.665 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2024-02-11T16:56:51.665 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2024-02-11T16:56:51.665 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2024-02-11T16:56:51.665 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2024-02-11T16:56:51.665 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2024-02-11T16:56:51.665 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-02-11T16:56:51.665 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-02-11T16:56:51.665 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-02-11T16:56:51.665 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-02-11T16:56:51.665 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-02-11T16:56:51.666 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-02-11T16:56:51.666 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-02-11T16:56:51.666 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-02-11T16:56:51.666 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-02-11T16:56:51.666 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-02-11T16:56:51.666 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-11T16:56:51.666 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-02-11T16:56:51.666 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-02-11T16:56:51.666 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-02-11T16:56:51.666 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-02-11T16:56:51.666 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-02-11T16:56:51.666 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-02-11T16:56:51.666 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-02-11T16:56:51.666 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-02-11T16:56:51.667 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-02-11T16:56:51.667 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-02-11T16:56:51.667 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-02-11T16:56:51.667 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-02-11T16:56:51.667 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-02-11T16:56:51.667 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-02-11T16:56:51.726 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-02-11T16:56:52.187 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-02-11T16:56:52.426 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-02-11T16:56:52.430 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-02-11T16:56:52.430 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:56:52.430 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-02-11T16:56:52.430 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-2:16.2.4-0.el8.x86_64 2024-02-11T16:56:52.431 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2024-02-11T16:56:52.431 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2024-02-11T16:56:52.431 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2024-02-11T16:56:52.431 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2024-02-11T16:56:52.431 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2024-02-11T16:56:52.431 INFO:teuthology.orchestra.run.smithi178.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-02-11T16:56:52.432 INFO:teuthology.orchestra.run.smithi178.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-02-11T16:56:52.432 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-02-11T16:56:52.432 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-02-11T16:56:52.432 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-02-11T16:56:52.432 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-02-11T16:56:52.432 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-6.2-6.el8.noarch 2024-02-11T16:56:52.432 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-02-11T16:56:52.433 INFO:teuthology.orchestra.run.smithi178.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-02-11T16:56:52.433 INFO:teuthology.orchestra.run.smithi178.stdout: python3-mako-1.0.6-14.el8.noarch 2024-02-11T16:56:52.433 INFO:teuthology.orchestra.run.smithi178.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-02-11T16:56:52.433 INFO:teuthology.orchestra.run.smithi178.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-02-11T16:56:52.433 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-02-11T16:56:52.433 INFO:teuthology.orchestra.run.smithi178.stdout: python3-portend-2.6-1.el8.noarch 2024-02-11T16:56:52.433 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-02-11T16:56:52.433 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pycparser-2.14-14.el8.noarch 2024-02-11T16:56:52.434 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pytz-2017.2-11.el8.noarch 2024-02-11T16:56:52.434 INFO:teuthology.orchestra.run.smithi178.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-02-11T16:56:52.434 INFO:teuthology.orchestra.run.smithi178.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-02-11T16:56:52.434 INFO:teuthology.orchestra.run.smithi178.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-02-11T16:56:52.434 INFO:teuthology.orchestra.run.smithi178.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-02-11T16:56:52.434 INFO:teuthology.orchestra.run.smithi178.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-02-11T16:56:52.434 INFO:teuthology.orchestra.run.smithi178.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-02-11T16:56:52.435 INFO:teuthology.orchestra.run.smithi178.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-02-11T16:56:52.435 INFO:teuthology.orchestra.run.smithi178.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-02-11T16:56:52.435 INFO:teuthology.orchestra.run.smithi178.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-02-11T16:56:52.435 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:56:52.435 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-02-11T16:56:52.661 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-02-11T16:56:52.687 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-base 2024-02-11T16:56:52.877 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-02-11T16:56:52.896 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-02-11T16:56:53.027 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2024-02-11T16:56:53.440 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:12 ago on Sun 11 Feb 2024 04:55:41 PM UTC. 2024-02-11T16:56:53.475 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2024-02-11T16:56:53.475 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2024-02-11T16:56:53.475 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2024-02-11T16:56:53.475 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2024-02-11T16:56:53.475 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2024-02-11T16:56:53.475 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2024-02-11T16:56:53.475 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2024-02-11T16:56:53.475 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-02-11T16:56:53.475 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-02-11T16:56:53.475 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-02-11T16:56:53.476 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-02-11T16:56:53.476 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-02-11T16:56:53.476 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-02-11T16:56:53.476 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-02-11T16:56:53.476 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-02-11T16:56:53.476 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-02-11T16:56:53.476 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-02-11T16:56:53.476 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-11T16:56:53.476 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-02-11T16:56:53.476 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-02-11T16:56:53.476 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-02-11T16:56:53.476 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-02-11T16:56:53.476 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-02-11T16:56:53.476 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-02-11T16:56:53.477 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-02-11T16:56:53.477 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-02-11T16:56:53.477 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-02-11T16:56:53.477 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-02-11T16:56:53.477 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-02-11T16:56:53.477 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-02-11T16:56:53.477 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-02-11T16:56:53.477 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-02-11T16:56:54.381 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-02-11T16:56:54.381 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:56:54.381 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-02-11T16:56:54.381 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-2:16.2.4-0.el8.x86_64 2024-02-11T16:56:54.381 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2024-02-11T16:56:54.381 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2024-02-11T16:56:54.381 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2024-02-11T16:56:54.381 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2024-02-11T16:56:54.382 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2024-02-11T16:56:54.382 INFO:teuthology.orchestra.run.smithi037.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-02-11T16:56:54.382 INFO:teuthology.orchestra.run.smithi037.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-02-11T16:56:54.382 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-02-11T16:56:54.382 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-02-11T16:56:54.382 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-02-11T16:56:54.382 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-02-11T16:56:54.382 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco-6.2-6.el8.noarch 2024-02-11T16:56:54.382 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-02-11T16:56:54.382 INFO:teuthology.orchestra.run.smithi037.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-02-11T16:56:54.382 INFO:teuthology.orchestra.run.smithi037.stdout: python3-mako-1.0.6-14.el8.noarch 2024-02-11T16:56:54.382 INFO:teuthology.orchestra.run.smithi037.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-02-11T16:56:54.382 INFO:teuthology.orchestra.run.smithi037.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-02-11T16:56:54.382 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-02-11T16:56:54.383 INFO:teuthology.orchestra.run.smithi037.stdout: python3-portend-2.6-1.el8.noarch 2024-02-11T16:56:54.383 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-02-11T16:56:54.383 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pycparser-2.14-14.el8.noarch 2024-02-11T16:56:54.383 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pytz-2017.2-11.el8.noarch 2024-02-11T16:56:54.383 INFO:teuthology.orchestra.run.smithi037.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-02-11T16:56:54.383 INFO:teuthology.orchestra.run.smithi037.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-02-11T16:56:54.383 INFO:teuthology.orchestra.run.smithi037.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-02-11T16:56:54.383 INFO:teuthology.orchestra.run.smithi037.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-02-11T16:56:54.383 INFO:teuthology.orchestra.run.smithi037.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-02-11T16:56:54.383 INFO:teuthology.orchestra.run.smithi037.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-02-11T16:56:54.383 INFO:teuthology.orchestra.run.smithi037.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-02-11T16:56:54.383 INFO:teuthology.orchestra.run.smithi037.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-02-11T16:56:54.383 INFO:teuthology.orchestra.run.smithi037.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-02-11T16:56:54.383 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:56:54.383 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-02-11T16:56:54.645 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-base 2024-02-11T16:56:54.893 INFO:teuthology.orchestra.run.smithi178.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-11T16:56:54.939 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-02-11T16:56:54.940 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2024-02-11T16:56:54.941 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-02-11T16:56:55.054 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install cephadm 2024-02-11T16:56:55.431 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:12 ago on Sun 11 Feb 2024 04:55:43 PM UTC. 2024-02-11T16:56:55.806 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:14 ago on Sun 11 Feb 2024 04:55:41 PM UTC. 2024-02-11T16:56:57.004 INFO:teuthology.orchestra.run.smithi037.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-11T16:56:57.055 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-02-11T16:56:57.056 INFO:teuthology.orchestra.run.smithi037.stdout:Nothing to do. 2024-02-11T16:56:57.056 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-02-11T16:56:57.177 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install cephadm 2024-02-11T16:56:57.314 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-02-11T16:56:57.315 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-02-11T16:56:57.315 INFO:teuthology.orchestra.run.smithi178.stdout: Package Architecture Version Repository Size 2024-02-11T16:56:57.315 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-02-11T16:56:57.315 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-02-11T16:56:57.315 INFO:teuthology.orchestra.run.smithi178.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2024-02-11T16:56:57.316 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:56:57.316 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-02-11T16:56:57.316 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-02-11T16:56:57.316 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-02-11T16:56:57.316 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:56:57.316 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 71 k 2024-02-11T16:56:57.316 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 290 k 2024-02-11T16:56:57.317 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-02-11T16:56:57.360 INFO:teuthology.orchestra.run.smithi178.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.6 MB/s | 71 kB 00:00 2024-02-11T16:56:57.361 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-02-11T16:56:57.361 INFO:teuthology.orchestra.run.smithi178.stdout:Total 1.5 MB/s | 71 kB 00:00 2024-02-11T16:56:57.361 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-02-11T16:56:57.368 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-02-11T16:56:57.368 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-02-11T16:56:57.370 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-02-11T16:56:57.373 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-02-11T16:56:57.415 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-02-11T16:56:57.877 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-11T16:56:57.961 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:14 ago on Sun 11 Feb 2024 04:55:43 PM UTC. 2024-02-11T16:56:58.036 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-11T16:56:58.266 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-11T16:56:58.619 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-11T16:56:58.620 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:56:58.620 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-02-11T16:56:58.620 INFO:teuthology.orchestra.run.smithi178.stdout: cephadm-2:16.2.4-0.el8.noarch 2024-02-11T16:56:58.620 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:56:58.620 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-02-11T16:56:58.863 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-immutable-object-cache 2024-02-11T16:56:59.596 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-02-11T16:56:59.598 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-02-11T16:56:59.598 INFO:teuthology.orchestra.run.smithi037.stdout: Package Architecture Version Repository Size 2024-02-11T16:56:59.598 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-02-11T16:56:59.598 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-02-11T16:56:59.598 INFO:teuthology.orchestra.run.smithi037.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2024-02-11T16:56:59.598 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:56:59.598 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-02-11T16:56:59.598 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-02-11T16:56:59.598 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-02-11T16:56:59.598 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:56:59.598 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 71 k 2024-02-11T16:56:59.598 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 290 k 2024-02-11T16:56:59.598 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-02-11T16:56:59.624 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:18 ago on Sun 11 Feb 2024 04:55:41 PM UTC. 2024-02-11T16:56:59.639 INFO:teuthology.orchestra.run.smithi037.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.7 MB/s | 71 kB 00:00 2024-02-11T16:56:59.639 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-02-11T16:56:59.639 INFO:teuthology.orchestra.run.smithi037.stdout:Total 1.7 MB/s | 71 kB 00:00 2024-02-11T16:56:59.640 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-02-11T16:56:59.647 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-02-11T16:56:59.647 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-02-11T16:56:59.649 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-02-11T16:56:59.652 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-02-11T16:56:59.699 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-02-11T16:57:00.226 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-11T16:57:00.430 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-11T16:57:00.664 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-11T16:57:01.085 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-11T16:57:01.085 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:57:01.085 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-02-11T16:57:01.085 INFO:teuthology.orchestra.run.smithi037.stdout: cephadm-2:16.2.4-0.el8.noarch 2024-02-11T16:57:01.085 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:57:01.085 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-02-11T16:57:01.135 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-02-11T16:57:01.135 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-02-11T16:57:01.135 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repo Size 2024-02-11T16:57:01.136 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-02-11T16:57:01.136 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-02-11T16:57:01.136 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2024-02-11T16:57:01.136 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:57:01.136 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-02-11T16:57:01.136 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-02-11T16:57:01.136 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-02-11T16:57:01.136 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:57:01.137 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 163 k 2024-02-11T16:57:01.137 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 587 k 2024-02-11T16:57:01.137 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-02-11T16:57:01.321 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-immutable-object-cache 2024-02-11T16:57:01.548 INFO:teuthology.orchestra.run.smithi178.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 396 kB/s | 163 kB 00:00 2024-02-11T16:57:01.549 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-02-11T16:57:01.550 INFO:teuthology.orchestra.run.smithi178.stdout:Total 394 kB/s | 163 kB 00:00 2024-02-11T16:57:01.550 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-02-11T16:57:01.561 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-02-11T16:57:01.561 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-02-11T16:57:01.610 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-02-11T16:57:01.613 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-02-11T16:57:01.730 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-02-11T16:57:01.903 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-11T16:57:02.111 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:19 ago on Sun 11 Feb 2024 04:55:43 PM UTC. 2024-02-11T16:57:02.318 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-11T16:57:02.675 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-11T16:57:02.675 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:57:02.675 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-02-11T16:57:02.675 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2024-02-11T16:57:02.675 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:57:02.675 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-02-11T16:57:02.889 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-mgr 2024-02-11T16:57:03.652 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:22 ago on Sun 11 Feb 2024 04:55:41 PM UTC. 2024-02-11T16:57:03.719 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-02-11T16:57:03.720 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-02-11T16:57:03.720 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repo Size 2024-02-11T16:57:03.720 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-02-11T16:57:03.720 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-02-11T16:57:03.720 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2024-02-11T16:57:03.720 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:57:03.720 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-02-11T16:57:03.720 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-02-11T16:57:03.720 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-02-11T16:57:03.720 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:57:03.721 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 163 k 2024-02-11T16:57:03.721 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 587 k 2024-02-11T16:57:03.721 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-02-11T16:57:03.761 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 4.0 MB/s | 163 kB 00:00 2024-02-11T16:57:03.761 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-02-11T16:57:03.761 INFO:teuthology.orchestra.run.smithi037.stdout:Total 3.9 MB/s | 163 kB 00:00 2024-02-11T16:57:03.762 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-02-11T16:57:03.774 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-02-11T16:57:03.774 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-02-11T16:57:03.825 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-02-11T16:57:03.828 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-02-11T16:57:03.942 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-02-11T16:57:04.144 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-11T16:57:04.577 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-11T16:57:05.020 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-11T16:57:05.020 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:57:05.020 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-02-11T16:57:05.020 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2024-02-11T16:57:05.020 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:57:05.020 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-02-11T16:57:05.117 INFO:teuthology.orchestra.run.smithi178.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-11T16:57:05.164 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-02-11T16:57:05.165 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2024-02-11T16:57:05.165 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-02-11T16:57:05.283 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-mgr 2024-02-11T16:57:05.288 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-mgr-dashboard 2024-02-11T16:57:06.044 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:25 ago on Sun 11 Feb 2024 04:55:41 PM UTC. 2024-02-11T16:57:06.070 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:23 ago on Sun 11 Feb 2024 04:55:43 PM UTC. 2024-02-11T16:57:07.532 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-02-11T16:57:07.533 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-02-11T16:57:07.533 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2024-02-11T16:57:07.533 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-02-11T16:57:07.533 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-02-11T16:57:07.533 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2024-02-11T16:57:07.534 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2024-02-11T16:57:07.534 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2024-02-11T16:57:07.534 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2024-02-11T16:57:07.534 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-02-11T16:57:07.534 INFO:teuthology.orchestra.run.smithi178.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-02-11T16:57:07.534 INFO:teuthology.orchestra.run.smithi178.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-02-11T16:57:07.534 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:57:07.534 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-02-11T16:57:07.534 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-02-11T16:57:07.534 INFO:teuthology.orchestra.run.smithi178.stdout:Install 6 Packages 2024-02-11T16:57:07.534 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:57:07.535 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 3.0 M 2024-02-11T16:57:07.535 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 15 M 2024-02-11T16:57:07.535 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-02-11T16:57:07.633 INFO:teuthology.orchestra.run.smithi037.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-11T16:57:07.684 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-02-11T16:57:07.685 INFO:teuthology.orchestra.run.smithi037.stdout:Nothing to do. 2024-02-11T16:57:07.685 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-02-11T16:57:07.943 INFO:teuthology.orchestra.run.smithi178.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 477 kB/s | 24 kB 00:00 2024-02-11T16:57:07.967 INFO:teuthology.orchestra.run.smithi178.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 118 kB/s | 8.9 kB 00:00 2024-02-11T16:57:08.027 INFO:teuthology.orchestra.run.smithi178.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 20 MB/s | 2.7 MB 00:00 2024-02-11T16:57:08.108 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-mgr-dashboard 2024-02-11T16:57:08.126 INFO:teuthology.orchestra.run.smithi178.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 232 kB/s | 43 kB 00:00 2024-02-11T16:57:08.435 INFO:teuthology.orchestra.run.smithi178.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 72 kB/s | 34 kB 00:00 2024-02-11T16:57:08.602 INFO:teuthology.orchestra.run.smithi178.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 341 kB/s | 196 kB 00:00 2024-02-11T16:57:08.605 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-02-11T16:57:08.606 INFO:teuthology.orchestra.run.smithi178.stdout:Total 2.8 MB/s | 3.0 MB 00:01 2024-02-11T16:57:08.617 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-02-11T16:57:08.627 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-02-11T16:57:08.627 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-02-11T16:57:08.822 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-02-11T16:57:08.825 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-02-11T16:57:08.899 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:25 ago on Sun 11 Feb 2024 04:55:43 PM UTC. 2024-02-11T16:57:09.100 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-02-11T16:57:09.324 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-02-11T16:57:09.569 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-02-11T16:57:09.770 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-02-11T16:57:09.891 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2024-02-11T16:57:10.523 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-02-11T16:57:10.524 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-02-11T16:57:10.524 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-02-11T16:57:10.524 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-02-11T16:57:10.524 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-02-11T16:57:10.524 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2024-02-11T16:57:10.525 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-02-11T16:57:10.525 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2024-02-11T16:57:10.525 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2024-02-11T16:57:10.525 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-02-11T16:57:10.525 INFO:teuthology.orchestra.run.smithi037.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-02-11T16:57:10.525 INFO:teuthology.orchestra.run.smithi037.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-02-11T16:57:10.525 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:57:10.525 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-02-11T16:57:10.525 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-02-11T16:57:10.525 INFO:teuthology.orchestra.run.smithi037.stdout:Install 6 Packages 2024-02-11T16:57:10.525 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:57:10.526 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 3.0 M 2024-02-11T16:57:10.526 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 15 M 2024-02-11T16:57:10.526 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-02-11T16:57:10.865 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2024-02-11T16:57:11.145 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-02-11T16:57:11.230 INFO:teuthology.orchestra.run.smithi037.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 515 kB/s | 24 kB 00:00 2024-02-11T16:57:11.255 INFO:teuthology.orchestra.run.smithi037.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 124 kB/s | 8.9 kB 00:00 2024-02-11T16:57:11.314 INFO:teuthology.orchestra.run.smithi037.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 20 MB/s | 2.7 MB 00:00 2024-02-11T16:57:11.398 INFO:teuthology.orchestra.run.smithi037.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 255 kB/s | 43 kB 00:00 2024-02-11T16:57:11.438 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-02-11T16:57:11.438 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2024-02-11T16:57:11.438 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2024-02-11T16:57:11.438 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2024-02-11T16:57:11.438 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-02-11T16:57:11.439 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-02-11T16:57:11.456 INFO:teuthology.orchestra.run.smithi037.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 168 kB/s | 34 kB 00:00 2024-02-11T16:57:11.565 INFO:teuthology.orchestra.run.smithi037.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 785 kB/s | 196 kB 00:00 2024-02-11T16:57:11.568 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-02-11T16:57:11.568 INFO:teuthology.orchestra.run.smithi037.stdout:Total 2.8 MB/s | 3.0 MB 00:01 2024-02-11T16:57:11.580 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-02-11T16:57:11.592 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-02-11T16:57:11.592 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-02-11T16:57:11.770 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-02-11T16:57:11.773 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-02-11T16:57:11.825 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-02-11T16:57:11.825 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:57:11.825 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-02-11T16:57:11.825 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2024-02-11T16:57:11.825 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2024-02-11T16:57:11.825 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2024-02-11T16:57:11.825 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-02-11T16:57:11.825 INFO:teuthology.orchestra.run.smithi178.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-02-11T16:57:11.825 INFO:teuthology.orchestra.run.smithi178.stdout: python3-routes-2.4.1-12.el8.noarch 2024-02-11T16:57:11.825 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:57:11.826 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-02-11T16:57:11.961 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-11T16:57:12.067 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-02-11T16:57:12.300 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-02-11T16:57:12.544 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-02-11T16:57:12.711 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:31 ago on Sun 11 Feb 2024 04:55:41 PM UTC. 2024-02-11T16:57:12.742 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-02-11T16:57:12.898 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2024-02-11T16:57:13.941 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2024-02-11T16:57:14.206 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-02-11T16:57:14.228 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-02-11T16:57:14.229 INFO:teuthology.orchestra.run.smithi178.stdout:============================================================================================ 2024-02-11T16:57:14.229 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2024-02-11T16:57:14.229 INFO:teuthology.orchestra.run.smithi178.stdout:============================================================================================ 2024-02-11T16:57:14.230 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-02-11T16:57:14.230 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2024-02-11T16:57:14.230 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2024-02-11T16:57:14.230 INFO:teuthology.orchestra.run.smithi178.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-02-11T16:57:14.230 INFO:teuthology.orchestra.run.smithi178.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-02-11T16:57:14.230 INFO:teuthology.orchestra.run.smithi178.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-02-11T16:57:14.230 INFO:teuthology.orchestra.run.smithi178.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-02-11T16:57:14.230 INFO:teuthology.orchestra.run.smithi178.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-02-11T16:57:14.230 INFO:teuthology.orchestra.run.smithi178.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-02-11T16:57:14.230 INFO:teuthology.orchestra.run.smithi178.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-02-11T16:57:14.230 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:57:14.230 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-02-11T16:57:14.230 INFO:teuthology.orchestra.run.smithi178.stdout:============================================================================================ 2024-02-11T16:57:14.231 INFO:teuthology.orchestra.run.smithi178.stdout:Install 8 Packages 2024-02-11T16:57:14.231 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:57:14.231 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 37 M 2024-02-11T16:57:14.231 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 210 M 2024-02-11T16:57:14.231 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-02-11T16:57:14.490 INFO:teuthology.orchestra.run.smithi178.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 48 MB/s | 8.6 MB 00:00 2024-02-11T16:57:14.512 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-02-11T16:57:14.512 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2024-02-11T16:57:14.512 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2024-02-11T16:57:14.512 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2024-02-11T16:57:14.512 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-02-11T16:57:14.512 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-02-11T16:57:14.657 INFO:teuthology.orchestra.run.smithi178.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 14 MB/s | 4.9 MB 00:00 2024-02-11T16:57:14.724 INFO:teuthology.orchestra.run.smithi178.stdout:(3/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 16 MB/s | 3.7 MB 00:00 2024-02-11T16:57:14.749 INFO:teuthology.orchestra.run.smithi178.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.4 MB/s | 225 kB 00:00 2024-02-11T16:57:14.841 INFO:teuthology.orchestra.run.smithi178.stdout:(5/8): openblas-0.3.15-4.el8.x86_64.rpm 8.9 MB/s | 4.7 MB 00:00 2024-02-11T16:57:14.927 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-02-11T16:57:14.927 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:57:14.927 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-02-11T16:57:14.927 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2024-02-11T16:57:14.927 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2024-02-11T16:57:14.927 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2024-02-11T16:57:14.927 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-02-11T16:57:14.927 INFO:teuthology.orchestra.run.smithi037.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-02-11T16:57:14.928 INFO:teuthology.orchestra.run.smithi037.stdout: python3-routes-2.4.1-12.el8.noarch 2024-02-11T16:57:14.928 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:57:14.928 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-02-11T16:57:15.073 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-11T16:57:15.133 INFO:teuthology.orchestra.run.smithi178.stdout:(6/8): python3-scipy-1.0.0-21.module_el8+762+77 35 MB/s | 14 MB 00:00 2024-02-11T16:57:15.158 INFO:teuthology.orchestra.run.smithi178.stdout:(7/8): libquadmath-8.5.0-21.el8.x86_64.rpm 544 kB/s | 172 kB 00:00 2024-02-11T16:57:15.183 INFO:teuthology.orchestra.run.smithi178.stdout:(8/8): libgfortran-8.5.0-21.el8.x86_64.rpm 1.5 MB/s | 645 kB 00:00 2024-02-11T16:57:15.184 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-02-11T16:57:15.184 INFO:teuthology.orchestra.run.smithi178.stdout:Total 39 MB/s | 37 MB 00:00 2024-02-11T16:57:15.413 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-02-11T16:57:15.431 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-02-11T16:57:15.432 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-02-11T16:57:15.765 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-02-11T16:57:15.768 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-02-11T16:57:15.867 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:32 ago on Sun 11 Feb 2024 04:55:43 PM UTC. 2024-02-11T16:57:16.392 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-02-11T16:57:16.542 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-11T16:57:16.682 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-11T16:57:16.817 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-11T16:57:17.439 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-11T16:57:17.490 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-02-11T16:57:17.492 INFO:teuthology.orchestra.run.smithi037.stdout:============================================================================================ 2024-02-11T16:57:17.492 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-02-11T16:57:17.492 INFO:teuthology.orchestra.run.smithi037.stdout:============================================================================================ 2024-02-11T16:57:17.492 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-02-11T16:57:17.492 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2024-02-11T16:57:17.492 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-02-11T16:57:17.492 INFO:teuthology.orchestra.run.smithi037.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-02-11T16:57:17.492 INFO:teuthology.orchestra.run.smithi037.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-02-11T16:57:17.492 INFO:teuthology.orchestra.run.smithi037.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-02-11T16:57:17.492 INFO:teuthology.orchestra.run.smithi037.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-02-11T16:57:17.493 INFO:teuthology.orchestra.run.smithi037.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-02-11T16:57:17.493 INFO:teuthology.orchestra.run.smithi037.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-02-11T16:57:17.493 INFO:teuthology.orchestra.run.smithi037.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-02-11T16:57:17.493 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:57:17.493 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-02-11T16:57:17.493 INFO:teuthology.orchestra.run.smithi037.stdout:============================================================================================ 2024-02-11T16:57:17.493 INFO:teuthology.orchestra.run.smithi037.stdout:Install 8 Packages 2024-02-11T16:57:17.493 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:57:17.494 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 37 M 2024-02-11T16:57:17.494 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 210 M 2024-02-11T16:57:17.494 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-02-11T16:57:17.617 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-02-11T16:57:17.877 INFO:teuthology.orchestra.run.smithi037.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 58 MB/s | 8.6 MB 00:00 2024-02-11T16:57:18.044 INFO:teuthology.orchestra.run.smithi037.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 15 MB/s | 4.9 MB 00:00 2024-02-11T16:57:18.111 INFO:teuthology.orchestra.run.smithi037.stdout:(3/8): openblas-0.3.15-4.el8.x86_64.rpm 12 MB/s | 4.7 MB 00:00 2024-02-11T16:57:18.136 INFO:teuthology.orchestra.run.smithi037.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.4 MB/s | 225 kB 00:00 2024-02-11T16:57:18.203 INFO:teuthology.orchestra.run.smithi037.stdout:(5/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 11 MB/s | 3.7 MB 00:00 2024-02-11T16:57:18.243 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-02-11T16:57:18.353 INFO:teuthology.orchestra.run.smithi037.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 1.1 MB/s | 172 kB 00:00 2024-02-11T16:57:18.408 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-11T16:57:18.412 INFO:teuthology.orchestra.run.smithi037.stdout:(7/8): libgfortran-8.5.0-21.el8.x86_64.rpm 2.3 MB/s | 645 kB 00:00 2024-02-11T16:57:18.587 INFO:teuthology.orchestra.run.smithi037.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 30 MB/s | 14 MB 00:00 2024-02-11T16:57:18.588 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-02-11T16:57:18.588 INFO:teuthology.orchestra.run.smithi037.stdout:Total 34 MB/s | 37 MB 00:01 2024-02-11T16:57:18.825 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-02-11T16:57:18.855 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-02-11T16:57:18.855 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-02-11T16:57:19.088 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-11T16:57:19.207 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-02-11T16:57:19.210 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-02-11T16:57:19.462 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-02-11T16:57:19.692 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-11T16:57:19.977 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-02-11T16:57:20.123 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-11T16:57:20.241 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-11T16:57:20.407 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-11T16:57:21.040 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-11T16:57:21.214 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-02-11T16:57:21.464 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-11T16:57:21.851 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-02-11T16:57:21.981 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-11T16:57:22.741 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-11T16:57:22.756 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-02-11T16:57:22.957 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-02-11T16:57:23.055 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-02-11T16:57:23.273 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-02-11T16:57:23.273 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2024-02-11T16:57:23.273 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-02-11T16:57:23.273 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-02-11T16:57:23.273 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-02-11T16:57:23.273 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-02-11T16:57:23.274 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-02-11T16:57:23.274 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-02-11T16:57:23.306 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-11T16:57:23.699 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-02-11T16:57:23.699 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:57:23.699 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-02-11T16:57:23.699 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2024-02-11T16:57:23.699 INFO:teuthology.orchestra.run.smithi178.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-02-11T16:57:23.699 INFO:teuthology.orchestra.run.smithi178.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-02-11T16:57:23.699 INFO:teuthology.orchestra.run.smithi178.stdout: openblas-0.3.15-4.el8.x86_64 2024-02-11T16:57:23.699 INFO:teuthology.orchestra.run.smithi178.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-02-11T16:57:23.699 INFO:teuthology.orchestra.run.smithi178.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-02-11T16:57:23.699 INFO:teuthology.orchestra.run.smithi178.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-02-11T16:57:23.699 INFO:teuthology.orchestra.run.smithi178.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-02-11T16:57:23.700 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:57:23.700 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-02-11T16:57:23.857 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-mgr-rook 2024-02-11T16:57:24.610 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:43 ago on Sun 11 Feb 2024 04:55:41 PM UTC. 2024-02-11T16:57:25.195 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-11T16:57:26.127 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-02-11T16:57:26.129 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-02-11T16:57:26.129 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2024-02-11T16:57:26.129 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-02-11T16:57:26.129 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-02-11T16:57:26.129 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2024-02-11T16:57:26.129 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2024-02-11T16:57:26.129 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-02-11T16:57:26.129 INFO:teuthology.orchestra.run.smithi178.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-02-11T16:57:26.129 INFO:teuthology.orchestra.run.smithi178.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-02-11T16:57:26.129 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-02-11T16:57:26.129 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-02-11T16:57:26.130 INFO:teuthology.orchestra.run.smithi178.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-02-11T16:57:26.130 INFO:teuthology.orchestra.run.smithi178.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-02-11T16:57:26.130 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-02-11T16:57:26.130 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-02-11T16:57:26.130 INFO:teuthology.orchestra.run.smithi178.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-02-11T16:57:26.130 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-02-11T16:57:26.130 INFO:teuthology.orchestra.run.smithi178.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-02-11T16:57:26.130 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:57:26.130 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-02-11T16:57:26.130 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-02-11T16:57:26.130 INFO:teuthology.orchestra.run.smithi178.stdout:Install 13 Packages 2024-02-11T16:57:26.130 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:57:26.131 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 1.9 M 2024-02-11T16:57:26.131 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 24 M 2024-02-11T16:57:26.131 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-02-11T16:57:26.548 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-02-11T16:57:26.619 INFO:teuthology.orchestra.run.smithi178.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 551 kB/s | 27 kB 00:00 2024-02-11T16:57:26.644 INFO:teuthology.orchestra.run.smithi178.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 360 kB/s | 27 kB 00:00 2024-02-11T16:57:26.669 INFO:teuthology.orchestra.run.smithi178.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 202 kB/s | 20 kB 00:00 2024-02-11T16:57:26.744 INFO:teuthology.orchestra.run.smithi178.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 1.4 MB/s | 110 kB 00:00 2024-02-11T16:57:26.764 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-02-11T16:57:26.770 INFO:teuthology.orchestra.run.smithi178.stdout:(5/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 838 kB/s | 126 kB 00:00 2024-02-11T16:57:26.912 INFO:teuthology.orchestra.run.smithi178.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 638 kB/s | 155 kB 00:00 2024-02-11T16:57:26.937 INFO:teuthology.orchestra.run.smithi178.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 221 kB/s | 43 kB 00:00 2024-02-11T16:57:27.095 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-02-11T16:57:27.095 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2024-02-11T16:57:27.096 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-02-11T16:57:27.096 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-02-11T16:57:27.096 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-02-11T16:57:27.096 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-02-11T16:57:27.096 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-02-11T16:57:27.096 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-02-11T16:57:27.519 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-02-11T16:57:27.520 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:57:27.520 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-02-11T16:57:27.520 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2024-02-11T16:57:27.520 INFO:teuthology.orchestra.run.smithi037.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-02-11T16:57:27.520 INFO:teuthology.orchestra.run.smithi037.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-02-11T16:57:27.520 INFO:teuthology.orchestra.run.smithi037.stdout: openblas-0.3.15-4.el8.x86_64 2024-02-11T16:57:27.520 INFO:teuthology.orchestra.run.smithi037.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-02-11T16:57:27.520 INFO:teuthology.orchestra.run.smithi037.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-02-11T16:57:27.520 INFO:teuthology.orchestra.run.smithi037.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-02-11T16:57:27.520 INFO:teuthology.orchestra.run.smithi037.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-02-11T16:57:27.520 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:57:27.520 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-02-11T16:57:27.640 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-mgr-rook 2024-02-11T16:57:27.746 INFO:teuthology.orchestra.run.smithi178.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 34 kB/s | 33 kB 00:00 2024-02-11T16:57:27.771 INFO:teuthology.orchestra.run.smithi178.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 19 kB/s | 16 kB 00:00 2024-02-11T16:57:28.088 INFO:teuthology.orchestra.run.smithi178.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 71 kB/s | 81 kB 00:01 2024-02-11T16:57:28.426 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:45 ago on Sun 11 Feb 2024 04:55:43 PM UTC. 2024-02-11T16:57:28.763 INFO:teuthology.orchestra.run.smithi178.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 91 kB/s | 61 kB 00:00 2024-02-11T16:57:28.946 INFO:teuthology.orchestra.run.smithi178.stdout:(12/13): python3-rsa-4.9-2.el8.noarch.rpm 55 kB/s | 64 kB 00:01 2024-02-11T16:57:29.288 INFO:teuthology.orchestra.run.smithi178.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 774 kB/s | 1.2 MB 00:01 2024-02-11T16:57:29.291 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-02-11T16:57:29.291 INFO:teuthology.orchestra.run.smithi178.stdout:Total 619 kB/s | 1.9 MB 00:03 2024-02-11T16:57:29.347 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-02-11T16:57:29.362 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-02-11T16:57:29.362 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-02-11T16:57:29.451 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-02-11T16:57:29.454 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-02-11T16:57:30.062 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-02-11T16:57:30.064 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-02-11T16:57:30.064 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-02-11T16:57:30.064 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-02-11T16:57:30.064 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-02-11T16:57:30.064 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2024-02-11T16:57:30.064 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-02-11T16:57:30.064 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-02-11T16:57:30.065 INFO:teuthology.orchestra.run.smithi037.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-02-11T16:57:30.065 INFO:teuthology.orchestra.run.smithi037.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-02-11T16:57:30.065 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-02-11T16:57:30.065 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-02-11T16:57:30.065 INFO:teuthology.orchestra.run.smithi037.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-02-11T16:57:30.065 INFO:teuthology.orchestra.run.smithi037.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-02-11T16:57:30.065 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-02-11T16:57:30.065 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-02-11T16:57:30.065 INFO:teuthology.orchestra.run.smithi037.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-02-11T16:57:30.065 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-02-11T16:57:30.065 INFO:teuthology.orchestra.run.smithi037.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-02-11T16:57:30.065 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:57:30.065 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-02-11T16:57:30.065 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-02-11T16:57:30.066 INFO:teuthology.orchestra.run.smithi037.stdout:Install 13 Packages 2024-02-11T16:57:30.066 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:57:30.066 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 1.9 M 2024-02-11T16:57:30.066 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 24 M 2024-02-11T16:57:30.066 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-02-11T16:57:30.079 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-02-11T16:57:30.333 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-02-11T16:57:30.396 INFO:teuthology.orchestra.run.smithi037.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 614 kB/s | 27 kB 00:00 2024-02-11T16:57:30.421 INFO:teuthology.orchestra.run.smithi037.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 385 kB/s | 27 kB 00:00 2024-02-11T16:57:30.446 INFO:teuthology.orchestra.run.smithi037.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 212 kB/s | 20 kB 00:00 2024-02-11T16:57:30.497 INFO:teuthology.orchestra.run.smithi037.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 2.1 MB/s | 110 kB 00:00 2024-02-11T16:57:30.522 INFO:teuthology.orchestra.run.smithi037.stdout:(5/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.0 MB/s | 126 kB 00:00 2024-02-11T16:57:30.532 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-02-11T16:57:30.606 INFO:teuthology.orchestra.run.smithi037.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 975 kB/s | 155 kB 00:00 2024-02-11T16:57:30.623 INFO:teuthology.orchestra.run.smithi037.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 340 kB/s | 43 kB 00:00 2024-02-11T16:57:30.748 INFO:teuthology.orchestra.run.smithi037.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 146 kB/s | 33 kB 00:00 2024-02-11T16:57:30.791 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-02-11T16:57:30.823 INFO:teuthology.orchestra.run.smithi037.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 75 kB/s | 16 kB 00:00 2024-02-11T16:57:30.874 INFO:teuthology.orchestra.run.smithi037.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 325 kB/s | 81 kB 00:00 2024-02-11T16:57:30.933 INFO:teuthology.orchestra.run.smithi037.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 591 kB/s | 64 kB 00:00 2024-02-11T16:57:30.966 INFO:teuthology.orchestra.run.smithi037.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 667 kB/s | 61 kB 00:00 2024-02-11T16:57:30.996 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-02-11T16:57:31.178 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-02-11T16:57:31.374 INFO:teuthology.orchestra.run.smithi037.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 1.9 MB/s | 1.2 MB 00:00 2024-02-11T16:57:31.377 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-02-11T16:57:31.377 INFO:teuthology.orchestra.run.smithi037.stdout:Total 1.5 MB/s | 1.9 MB 00:01 2024-02-11T16:57:31.433 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-02-11T16:57:31.436 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-02-11T16:57:31.453 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-02-11T16:57:31.453 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-02-11T16:57:31.549 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-02-11T16:57:31.552 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-02-11T16:57:31.717 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-02-11T16:57:31.968 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-02-11T16:57:32.096 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-02-11T16:57:32.309 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-02-11T16:57:32.542 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-02-11T16:57:32.755 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-02-11T16:57:32.963 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-02-11T16:57:33.000 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-02-11T16:57:33.182 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-02-11T16:57:33.321 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-02-11T16:57:33.413 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-02-11T16:57:33.533 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-02-11T16:57:33.746 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-02-11T16:57:33.763 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-02-11T16:57:33.905 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-02-11T16:57:33.966 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-02-11T16:57:34.199 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-02-11T16:57:34.199 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2024-02-11T16:57:34.199 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-02-11T16:57:34.199 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-02-11T16:57:34.200 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-02-11T16:57:34.200 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-02-11T16:57:34.200 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-02-11T16:57:34.200 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-02-11T16:57:34.200 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-02-11T16:57:34.200 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-02-11T16:57:34.200 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-02-11T16:57:34.200 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-02-11T16:57:34.200 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-02-11T16:57:34.688 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-02-11T16:57:34.688 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:57:34.688 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-02-11T16:57:34.688 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2024-02-11T16:57:34.688 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-02-11T16:57:34.688 INFO:teuthology.orchestra.run.smithi178.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-02-11T16:57:34.689 INFO:teuthology.orchestra.run.smithi178.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-02-11T16:57:34.689 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-02-11T16:57:34.689 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-02-11T16:57:34.689 INFO:teuthology.orchestra.run.smithi178.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-02-11T16:57:34.689 INFO:teuthology.orchestra.run.smithi178.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-02-11T16:57:34.689 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-02-11T16:57:34.689 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-02-11T16:57:34.689 INFO:teuthology.orchestra.run.smithi178.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-02-11T16:57:34.689 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rsa-4.9-2.el8.noarch 2024-02-11T16:57:34.689 INFO:teuthology.orchestra.run.smithi178.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-02-11T16:57:34.689 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:57:34.689 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-02-11T16:57:34.820 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-mgr-cephadm 2024-02-11T16:57:35.216 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-02-11T16:57:35.504 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-02-11T16:57:35.570 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:54 ago on Sun 11 Feb 2024 04:55:41 PM UTC. 2024-02-11T16:57:35.739 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-02-11T16:57:35.956 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-02-11T16:57:36.113 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-02-11T16:57:36.420 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-02-11T16:57:36.420 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2024-02-11T16:57:36.420 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-02-11T16:57:36.420 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-02-11T16:57:36.420 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-02-11T16:57:36.420 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-02-11T16:57:36.420 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-02-11T16:57:36.420 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-02-11T16:57:36.420 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-02-11T16:57:36.420 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-02-11T16:57:36.420 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-02-11T16:57:36.421 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-02-11T16:57:36.421 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-02-11T16:57:36.915 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-02-11T16:57:36.916 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:57:36.916 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-02-11T16:57:36.916 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2024-02-11T16:57:36.916 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-02-11T16:57:36.916 INFO:teuthology.orchestra.run.smithi037.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-02-11T16:57:36.916 INFO:teuthology.orchestra.run.smithi037.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-02-11T16:57:36.916 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-02-11T16:57:36.916 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-02-11T16:57:36.916 INFO:teuthology.orchestra.run.smithi037.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-02-11T16:57:36.916 INFO:teuthology.orchestra.run.smithi037.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-02-11T16:57:36.916 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-02-11T16:57:36.916 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-02-11T16:57:36.916 INFO:teuthology.orchestra.run.smithi037.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-02-11T16:57:36.917 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rsa-4.9-2.el8.noarch 2024-02-11T16:57:36.917 INFO:teuthology.orchestra.run.smithi037.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-02-11T16:57:36.917 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:57:36.917 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-02-11T16:57:37.050 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-mgr-cephadm 2024-02-11T16:57:37.065 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-02-11T16:57:37.066 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-02-11T16:57:37.066 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2024-02-11T16:57:37.067 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-02-11T16:57:37.067 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-02-11T16:57:37.067 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2024-02-11T16:57:37.067 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2024-02-11T16:57:37.067 INFO:teuthology.orchestra.run.smithi178.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-02-11T16:57:37.067 INFO:teuthology.orchestra.run.smithi178.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-02-11T16:57:37.067 INFO:teuthology.orchestra.run.smithi178.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-02-11T16:57:37.067 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2024-02-11T16:57:37.067 INFO:teuthology.orchestra.run.smithi178.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-02-11T16:57:37.067 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:57:37.067 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-02-11T16:57:37.067 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-02-11T16:57:37.067 INFO:teuthology.orchestra.run.smithi178.stdout:Install 6 Packages 2024-02-11T16:57:37.068 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:57:37.068 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 5.8 M 2024-02-11T16:57:37.068 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 24 M 2024-02-11T16:57:37.068 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-02-11T16:57:37.229 INFO:teuthology.orchestra.run.smithi178.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 2.1 MB/s | 89 kB 00:00 2024-02-11T16:57:37.288 INFO:teuthology.orchestra.run.smithi178.stdout:(2/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 5.2 MB/s | 538 kB 00:00 2024-02-11T16:57:37.330 INFO:teuthology.orchestra.run.smithi178.stdout:(3/6): python3-apipkg-1.5-6.el8.noarch.rpm 212 kB/s | 21 kB 00:00 2024-02-11T16:57:37.371 INFO:teuthology.orchestra.run.smithi178.stdout:(4/6): python3-remoto-1.2.1-1.el8.noarch.rpm 1.2 MB/s | 49 kB 00:00 2024-02-11T16:57:37.430 INFO:teuthology.orchestra.run.smithi178.stdout:(5/6): python3-execnet-1.7.1-1.el8.noarch.rpm 2.5 MB/s | 355 kB 00:00 2024-02-11T16:57:37.505 INFO:teuthology.orchestra.run.smithi178.stdout:(6/6): python3-babel-2.5.1-7.el8.noarch.rpm 15 MB/s | 4.8 MB 00:00 2024-02-11T16:57:37.507 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-02-11T16:57:37.507 INFO:teuthology.orchestra.run.smithi178.stdout:Total 13 MB/s | 5.8 MB 00:00 2024-02-11T16:57:37.557 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-02-11T16:57:37.565 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-02-11T16:57:37.566 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-02-11T16:57:37.665 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-02-11T16:57:37.668 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-02-11T16:57:37.844 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:54 ago on Sun 11 Feb 2024 04:55:43 PM UTC. 2024-02-11T16:57:37.993 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-02-11T16:57:38.245 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-02-11T16:57:38.532 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-02-11T16:57:39.443 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-02-11T16:57:39.467 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-02-11T16:57:39.468 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-02-11T16:57:39.469 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-02-11T16:57:39.469 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-02-11T16:57:39.469 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-02-11T16:57:39.469 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2024-02-11T16:57:39.469 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-02-11T16:57:39.469 INFO:teuthology.orchestra.run.smithi037.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-02-11T16:57:39.469 INFO:teuthology.orchestra.run.smithi037.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-02-11T16:57:39.469 INFO:teuthology.orchestra.run.smithi037.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-02-11T16:57:39.469 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2024-02-11T16:57:39.469 INFO:teuthology.orchestra.run.smithi037.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-02-11T16:57:39.469 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:57:39.469 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-02-11T16:57:39.469 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-02-11T16:57:39.470 INFO:teuthology.orchestra.run.smithi037.stdout:Install 6 Packages 2024-02-11T16:57:39.470 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:57:39.470 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 5.8 M 2024-02-11T16:57:39.470 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 24 M 2024-02-11T16:57:39.470 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-02-11T16:57:39.741 INFO:teuthology.orchestra.run.smithi037.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 2.0 MB/s | 89 kB 00:00 2024-02-11T16:57:39.782 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-02-11T16:57:39.783 INFO:teuthology.orchestra.run.smithi037.stdout:(2/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 6.2 MB/s | 538 kB 00:00 2024-02-11T16:57:39.992 INFO:teuthology.orchestra.run.smithi037.stdout:(3/6): python3-babel-2.5.1-7.el8.noarch.rpm 16 MB/s | 4.8 MB 00:00 2024-02-11T16:57:40.025 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2024-02-11T16:57:40.067 INFO:teuthology.orchestra.run.smithi037.stdout:(4/6): python3-apipkg-1.5-6.el8.noarch.rpm 65 kB/s | 21 kB 00:00 2024-02-11T16:57:40.254 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-02-11T16:57:40.384 INFO:teuthology.orchestra.run.smithi037.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 125 kB/s | 49 kB 00:00 2024-02-11T16:57:40.418 INFO:teuthology.orchestra.run.smithi037.stdout:(6/6): python3-execnet-1.7.1-1.el8.noarch.rpm 560 kB/s | 355 kB 00:00 2024-02-11T16:57:40.420 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-02-11T16:57:40.420 INFO:teuthology.orchestra.run.smithi037.stdout:Total 6.1 MB/s | 5.8 MB 00:00 2024-02-11T16:57:40.475 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-02-11T16:57:40.486 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-02-11T16:57:40.486 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-02-11T16:57:40.548 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-02-11T16:57:40.548 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2024-02-11T16:57:40.549 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-02-11T16:57:40.549 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2024-02-11T16:57:40.549 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-02-11T16:57:40.549 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-02-11T16:57:40.595 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-02-11T16:57:40.598 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-02-11T16:57:40.877 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-02-11T16:57:40.961 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-02-11T16:57:40.961 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:57:40.961 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-02-11T16:57:40.961 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2024-02-11T16:57:40.961 INFO:teuthology.orchestra.run.smithi178.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2024-02-11T16:57:40.961 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2024-02-11T16:57:40.961 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:57:40.961 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-02-11T16:57:41.102 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-fuse 2024-02-11T16:57:41.135 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-02-11T16:57:41.401 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-02-11T16:57:41.852 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:02:00 ago on Sun 11 Feb 2024 04:55:41 PM UTC. 2024-02-11T16:57:42.379 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-02-11T16:57:42.710 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-02-11T16:57:42.961 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2024-02-11T16:57:43.153 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-02-11T16:57:43.350 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-02-11T16:57:43.350 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-02-11T16:57:43.350 INFO:teuthology.orchestra.run.smithi178.stdout: Package Architecture Version Repository Size 2024-02-11T16:57:43.351 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-02-11T16:57:43.351 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-02-11T16:57:43.351 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2024-02-11T16:57:43.351 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2024-02-11T16:57:43.351 INFO:teuthology.orchestra.run.smithi178.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-02-11T16:57:43.351 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:57:43.351 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-02-11T16:57:43.351 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-02-11T16:57:43.351 INFO:teuthology.orchestra.run.smithi178.stdout:Install 2 Packages 2024-02-11T16:57:43.351 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:57:43.352 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 968 k 2024-02-11T16:57:43.352 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 3.2 M 2024-02-11T16:57:43.352 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-02-11T16:57:43.458 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-02-11T16:57:43.458 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2024-02-11T16:57:43.458 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-02-11T16:57:43.458 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2024-02-11T16:57:43.458 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-02-11T16:57:43.458 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-02-11T16:57:43.639 INFO:teuthology.orchestra.run.smithi178.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 13 MB/s | 885 kB 00:00 2024-02-11T16:57:43.773 INFO:teuthology.orchestra.run.smithi178.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 417 kB/s | 83 kB 00:00 2024-02-11T16:57:43.773 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-02-11T16:57:43.773 INFO:teuthology.orchestra.run.smithi178.stdout:Total 2.2 MB/s | 968 kB 00:00 2024-02-11T16:57:43.783 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-02-11T16:57:43.804 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-02-11T16:57:43.804 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-02-11T16:57:43.866 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-02-11T16:57:43.866 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:57:43.866 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-02-11T16:57:43.866 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2024-02-11T16:57:43.866 INFO:teuthology.orchestra.run.smithi037.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2024-02-11T16:57:43.866 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2024-02-11T16:57:43.867 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:57:43.867 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-02-11T16:57:43.893 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-02-11T16:57:43.896 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-02-11T16:57:44.012 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-fuse 2024-02-11T16:57:44.077 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-02-11T16:57:44.329 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-02-11T16:57:44.529 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-02-11T16:57:44.797 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:02:01 ago on Sun 11 Feb 2024 04:55:43 PM UTC. 2024-02-11T16:57:44.910 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-02-11T16:57:44.911 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2024-02-11T16:57:45.489 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-02-11T16:57:45.490 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:57:45.490 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-02-11T16:57:45.490 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-02-11T16:57:45.490 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:57:45.490 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-02-11T16:57:46.053 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install librados-devel 2024-02-11T16:57:46.422 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-02-11T16:57:46.423 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-02-11T16:57:46.423 INFO:teuthology.orchestra.run.smithi037.stdout: Package Architecture Version Repository Size 2024-02-11T16:57:46.423 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-02-11T16:57:46.423 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-02-11T16:57:46.423 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2024-02-11T16:57:46.423 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-02-11T16:57:46.423 INFO:teuthology.orchestra.run.smithi037.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-02-11T16:57:46.423 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:57:46.423 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-02-11T16:57:46.423 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-02-11T16:57:46.423 INFO:teuthology.orchestra.run.smithi037.stdout:Install 2 Packages 2024-02-11T16:57:46.424 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:57:46.424 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 968 k 2024-02-11T16:57:46.424 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 3.2 M 2024-02-11T16:57:46.424 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-02-11T16:57:46.687 INFO:teuthology.orchestra.run.smithi037.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 13 MB/s | 885 kB 00:00 2024-02-11T16:57:46.787 INFO:teuthology.orchestra.run.smithi037.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 505 kB/s | 83 kB 00:00 2024-02-11T16:57:46.787 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-02-11T16:57:46.788 INFO:teuthology.orchestra.run.smithi037.stdout:Total 2.6 MB/s | 968 kB 00:00 2024-02-11T16:57:46.798 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-02-11T16:57:46.814 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:02:05 ago on Sun 11 Feb 2024 04:55:41 PM UTC. 2024-02-11T16:57:46.820 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-02-11T16:57:46.820 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-02-11T16:57:46.913 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-02-11T16:57:46.916 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-02-11T16:57:47.101 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-02-11T16:57:47.341 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-02-11T16:57:47.545 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-02-11T16:57:47.956 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-02-11T16:57:47.957 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2024-02-11T16:57:48.302 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-02-11T16:57:48.302 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-02-11T16:57:48.303 INFO:teuthology.orchestra.run.smithi178.stdout: Package Architecture Version Repository Size 2024-02-11T16:57:48.303 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-02-11T16:57:48.303 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-02-11T16:57:48.303 INFO:teuthology.orchestra.run.smithi178.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2024-02-11T16:57:48.303 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:57:48.303 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-02-11T16:57:48.303 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-02-11T16:57:48.303 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-02-11T16:57:48.303 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:57:48.303 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 131 k 2024-02-11T16:57:48.303 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 439 k 2024-02-11T16:57:48.303 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-02-11T16:57:48.353 INFO:teuthology.orchestra.run.smithi178.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 2.6 MB/s | 131 kB 00:00 2024-02-11T16:57:48.354 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-02-11T16:57:48.354 INFO:teuthology.orchestra.run.smithi178.stdout:Total 2.5 MB/s | 131 kB 00:00 2024-02-11T16:57:48.354 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-02-11T16:57:48.360 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-02-11T16:57:48.360 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-02-11T16:57:48.364 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-02-11T16:57:48.364 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:57:48.364 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-02-11T16:57:48.364 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-02-11T16:57:48.364 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:57:48.364 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-02-11T16:57:48.410 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-02-11T16:57:48.413 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-02-11T16:57:48.525 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-02-11T16:57:48.604 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install librados-devel 2024-02-11T16:57:48.735 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-11T16:57:49.055 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-11T16:57:49.392 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:02:06 ago on Sun 11 Feb 2024 04:55:43 PM UTC. 2024-02-11T16:57:49.409 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-11T16:57:49.409 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:57:49.409 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-02-11T16:57:49.409 INFO:teuthology.orchestra.run.smithi178.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2024-02-11T16:57:49.410 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:57:49.410 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-02-11T16:57:49.634 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install libcephfs2 2024-02-11T16:57:50.393 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:02:09 ago on Sun 11 Feb 2024 04:55:41 PM UTC. 2024-02-11T16:57:51.016 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-02-11T16:57:51.017 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-02-11T16:57:51.017 INFO:teuthology.orchestra.run.smithi037.stdout: Package Architecture Version Repository Size 2024-02-11T16:57:51.017 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-02-11T16:57:51.017 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-02-11T16:57:51.017 INFO:teuthology.orchestra.run.smithi037.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2024-02-11T16:57:51.017 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:57:51.017 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-02-11T16:57:51.017 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-02-11T16:57:51.017 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-02-11T16:57:51.017 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:57:51.018 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 131 k 2024-02-11T16:57:51.018 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 439 k 2024-02-11T16:57:51.018 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-02-11T16:57:51.141 INFO:teuthology.orchestra.run.smithi037.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 1.0 MB/s | 131 kB 00:00 2024-02-11T16:57:51.142 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-02-11T16:57:51.142 INFO:teuthology.orchestra.run.smithi037.stdout:Total 1.0 MB/s | 131 kB 00:00 2024-02-11T16:57:51.142 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-02-11T16:57:51.147 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-02-11T16:57:51.147 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-02-11T16:57:51.200 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-02-11T16:57:51.203 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-02-11T16:57:51.438 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-02-11T16:57:51.669 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-11T16:57:51.888 INFO:teuthology.orchestra.run.smithi178.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-11T16:57:51.934 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-02-11T16:57:51.935 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2024-02-11T16:57:51.935 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-02-11T16:57:51.972 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-11T16:57:52.337 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-11T16:57:52.337 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:57:52.337 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-02-11T16:57:52.337 INFO:teuthology.orchestra.run.smithi037.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2024-02-11T16:57:52.337 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:57:52.337 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-02-11T16:57:52.363 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install libcephfs-devel 2024-02-11T16:57:52.559 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install libcephfs2 2024-02-11T16:57:53.118 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:02:12 ago on Sun 11 Feb 2024 04:55:41 PM UTC. 2024-02-11T16:57:53.350 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:02:10 ago on Sun 11 Feb 2024 04:55:43 PM UTC. 2024-02-11T16:57:54.618 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-02-11T16:57:54.619 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-02-11T16:57:54.619 INFO:teuthology.orchestra.run.smithi178.stdout: Package Architecture Version Repository Size 2024-02-11T16:57:54.619 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-02-11T16:57:54.619 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-02-11T16:57:54.619 INFO:teuthology.orchestra.run.smithi178.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2024-02-11T16:57:54.619 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:57:54.619 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-02-11T16:57:54.619 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-02-11T16:57:54.619 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-02-11T16:57:54.619 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:57:54.620 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 24 k 2024-02-11T16:57:54.620 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 94 k 2024-02-11T16:57:54.620 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-02-11T16:57:54.687 INFO:teuthology.orchestra.run.smithi178.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 363 kB/s | 24 kB 00:00 2024-02-11T16:57:54.687 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-02-11T16:57:54.687 INFO:teuthology.orchestra.run.smithi178.stdout:Total 356 kB/s | 24 kB 00:00 2024-02-11T16:57:54.688 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-02-11T16:57:54.691 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-02-11T16:57:54.691 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-02-11T16:57:54.693 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-02-11T16:57:54.696 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-02-11T16:57:54.749 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-02-11T16:57:54.913 INFO:teuthology.orchestra.run.smithi037.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-11T16:57:54.934 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-11T16:57:54.962 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-02-11T16:57:54.964 INFO:teuthology.orchestra.run.smithi037.stdout:Nothing to do. 2024-02-11T16:57:54.964 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-02-11T16:57:55.099 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install libcephfs-devel 2024-02-11T16:57:55.204 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-11T16:57:55.558 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-11T16:57:55.558 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:57:55.558 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-02-11T16:57:55.559 INFO:teuthology.orchestra.run.smithi178.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2024-02-11T16:57:55.559 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:57:55.559 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-02-11T16:57:55.670 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install librados2 2024-02-11T16:57:55.889 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:02:12 ago on Sun 11 Feb 2024 04:55:43 PM UTC. 2024-02-11T16:57:56.429 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:02:15 ago on Sun 11 Feb 2024 04:55:41 PM UTC. 2024-02-11T16:57:57.513 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-02-11T16:57:57.514 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-02-11T16:57:57.514 INFO:teuthology.orchestra.run.smithi037.stdout: Package Architecture Version Repository Size 2024-02-11T16:57:57.514 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-02-11T16:57:57.514 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-02-11T16:57:57.514 INFO:teuthology.orchestra.run.smithi037.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2024-02-11T16:57:57.514 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:57:57.514 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-02-11T16:57:57.514 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-02-11T16:57:57.514 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-02-11T16:57:57.514 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:57:57.515 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 24 k 2024-02-11T16:57:57.515 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 94 k 2024-02-11T16:57:57.515 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-02-11T16:57:57.638 INFO:teuthology.orchestra.run.smithi037.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 197 kB/s | 24 kB 00:00 2024-02-11T16:57:57.639 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-02-11T16:57:57.639 INFO:teuthology.orchestra.run.smithi037.stdout:Total 195 kB/s | 24 kB 00:00 2024-02-11T16:57:57.639 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-02-11T16:57:57.643 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-02-11T16:57:57.643 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-02-11T16:57:57.645 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-02-11T16:57:57.649 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-02-11T16:57:57.702 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-02-11T16:57:57.893 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-11T16:57:57.893 INFO:teuthology.orchestra.run.smithi178.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-11T16:57:57.938 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-02-11T16:57:57.939 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2024-02-11T16:57:57.939 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-02-11T16:57:58.053 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install librbd1 2024-02-11T16:57:58.159 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-11T16:57:58.521 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-11T16:57:58.521 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:57:58.521 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-02-11T16:57:58.522 INFO:teuthology.orchestra.run.smithi037.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2024-02-11T16:57:58.522 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:57:58.522 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-02-11T16:57:58.680 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install librados2 2024-02-11T16:57:58.805 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:02:17 ago on Sun 11 Feb 2024 04:55:41 PM UTC. 2024-02-11T16:57:59.469 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:02:16 ago on Sun 11 Feb 2024 04:55:43 PM UTC. 2024-02-11T16:58:00.271 INFO:teuthology.orchestra.run.smithi178.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-11T16:58:00.314 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-02-11T16:58:00.315 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2024-02-11T16:58:00.315 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-02-11T16:58:00.420 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install python3-rados 2024-02-11T16:58:01.044 INFO:teuthology.orchestra.run.smithi037.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-11T16:58:01.096 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-02-11T16:58:01.096 INFO:teuthology.orchestra.run.smithi037.stdout:Nothing to do. 2024-02-11T16:58:01.096 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-02-11T16:58:01.171 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:02:20 ago on Sun 11 Feb 2024 04:55:41 PM UTC. 2024-02-11T16:58:01.213 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install librbd1 2024-02-11T16:58:01.997 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:02:18 ago on Sun 11 Feb 2024 04:55:43 PM UTC. 2024-02-11T16:58:02.632 INFO:teuthology.orchestra.run.smithi178.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-11T16:58:02.679 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-02-11T16:58:02.680 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2024-02-11T16:58:02.680 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-02-11T16:58:03.040 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install python3-rgw 2024-02-11T16:58:03.580 INFO:teuthology.orchestra.run.smithi037.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-11T16:58:03.631 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-02-11T16:58:03.632 INFO:teuthology.orchestra.run.smithi037.stdout:Nothing to do. 2024-02-11T16:58:03.632 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-02-11T16:58:03.753 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install python3-rados 2024-02-11T16:58:03.790 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:02:22 ago on Sun 11 Feb 2024 04:55:41 PM UTC. 2024-02-11T16:58:04.541 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:02:21 ago on Sun 11 Feb 2024 04:55:43 PM UTC. 2024-02-11T16:58:05.252 INFO:teuthology.orchestra.run.smithi178.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-11T16:58:05.297 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-02-11T16:58:05.298 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2024-02-11T16:58:05.298 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-02-11T16:58:05.403 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install python3-cephfs 2024-02-11T16:58:06.128 INFO:teuthology.orchestra.run.smithi037.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-11T16:58:06.156 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:02:25 ago on Sun 11 Feb 2024 04:55:41 PM UTC. 2024-02-11T16:58:06.177 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-02-11T16:58:06.180 INFO:teuthology.orchestra.run.smithi037.stdout:Nothing to do. 2024-02-11T16:58:06.180 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-02-11T16:58:06.294 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install python3-rgw 2024-02-11T16:58:07.085 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:02:24 ago on Sun 11 Feb 2024 04:55:43 PM UTC. 2024-02-11T16:58:07.619 INFO:teuthology.orchestra.run.smithi178.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-11T16:58:07.664 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-02-11T16:58:07.665 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2024-02-11T16:58:07.665 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-02-11T16:58:07.770 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install python3-rbd 2024-02-11T16:58:08.523 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:02:27 ago on Sun 11 Feb 2024 04:55:41 PM UTC. 2024-02-11T16:58:08.670 INFO:teuthology.orchestra.run.smithi037.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-11T16:58:08.719 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-02-11T16:58:08.720 INFO:teuthology.orchestra.run.smithi037.stdout:Nothing to do. 2024-02-11T16:58:08.720 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-02-11T16:58:08.848 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install python3-cephfs 2024-02-11T16:58:09.636 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:02:26 ago on Sun 11 Feb 2024 04:55:43 PM UTC. 2024-02-11T16:58:09.965 INFO:teuthology.orchestra.run.smithi178.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-11T16:58:10.011 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-02-11T16:58:10.012 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2024-02-11T16:58:10.012 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-02-11T16:58:10.128 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install rbd-fuse 2024-02-11T16:58:10.882 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:02:29 ago on Sun 11 Feb 2024 04:55:41 PM UTC. 2024-02-11T16:58:11.221 INFO:teuthology.orchestra.run.smithi037.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-11T16:58:11.273 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-02-11T16:58:11.275 INFO:teuthology.orchestra.run.smithi037.stdout:Nothing to do. 2024-02-11T16:58:11.275 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-02-11T16:58:11.457 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install python3-rbd 2024-02-11T16:58:12.247 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:02:29 ago on Sun 11 Feb 2024 04:55:43 PM UTC. 2024-02-11T16:58:12.390 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-02-11T16:58:12.391 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-02-11T16:58:12.391 INFO:teuthology.orchestra.run.smithi178.stdout: Package Architecture Version Repository Size 2024-02-11T16:58:12.391 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-02-11T16:58:12.391 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-02-11T16:58:12.391 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2024-02-11T16:58:12.391 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:58:12.391 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-02-11T16:58:12.391 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-02-11T16:58:12.392 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-02-11T16:58:12.392 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:58:12.392 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 89 k 2024-02-11T16:58:12.392 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 298 k 2024-02-11T16:58:12.392 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-02-11T16:58:12.431 INFO:teuthology.orchestra.run.smithi178.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 2.2 MB/s | 89 kB 00:00 2024-02-11T16:58:12.431 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-02-11T16:58:12.432 INFO:teuthology.orchestra.run.smithi178.stdout:Total 2.2 MB/s | 89 kB 00:00 2024-02-11T16:58:12.432 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-02-11T16:58:12.441 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-02-11T16:58:12.441 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-02-11T16:58:12.491 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-02-11T16:58:12.494 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-02-11T16:58:12.599 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-02-11T16:58:12.814 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-11T16:58:13.109 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-11T16:58:13.797 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-11T16:58:13.797 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:58:13.797 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-02-11T16:58:13.797 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2024-02-11T16:58:13.797 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:58:13.797 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-02-11T16:58:13.809 INFO:teuthology.orchestra.run.smithi037.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-11T16:58:13.859 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-02-11T16:58:13.861 INFO:teuthology.orchestra.run.smithi037.stdout:Nothing to do. 2024-02-11T16:58:13.861 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-02-11T16:58:13.980 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install rbd-fuse 2024-02-11T16:58:14.064 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install rbd-mirror 2024-02-11T16:58:14.769 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:02:31 ago on Sun 11 Feb 2024 04:55:43 PM UTC. 2024-02-11T16:58:14.829 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:02:33 ago on Sun 11 Feb 2024 04:55:41 PM UTC. 2024-02-11T16:58:16.330 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-02-11T16:58:16.331 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-02-11T16:58:16.331 INFO:teuthology.orchestra.run.smithi178.stdout: Package Architecture Version Repository Size 2024-02-11T16:58:16.331 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-02-11T16:58:16.331 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-02-11T16:58:16.331 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-02-11T16:58:16.331 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:58:16.331 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-02-11T16:58:16.331 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-02-11T16:58:16.331 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-02-11T16:58:16.331 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:58:16.332 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 3.7 M 2024-02-11T16:58:16.332 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 18 M 2024-02-11T16:58:16.332 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-02-11T16:58:16.388 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-02-11T16:58:16.389 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-02-11T16:58:16.389 INFO:teuthology.orchestra.run.smithi037.stdout: Package Architecture Version Repository Size 2024-02-11T16:58:16.389 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-02-11T16:58:16.389 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-02-11T16:58:16.390 INFO:teuthology.orchestra.run.smithi037.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2024-02-11T16:58:16.390 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:58:16.390 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-02-11T16:58:16.390 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-02-11T16:58:16.390 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-02-11T16:58:16.390 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:58:16.390 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 89 k 2024-02-11T16:58:16.390 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 298 k 2024-02-11T16:58:16.390 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-02-11T16:58:16.427 INFO:teuthology.orchestra.run.smithi178.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 39 MB/s | 3.7 MB 00:00 2024-02-11T16:58:16.427 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-02-11T16:58:16.428 INFO:teuthology.orchestra.run.smithi037.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 2.3 MB/s | 89 kB 00:00 2024-02-11T16:58:16.428 INFO:teuthology.orchestra.run.smithi178.stdout:Total 39 MB/s | 3.7 MB 00:00 2024-02-11T16:58:16.428 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-02-11T16:58:16.429 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-02-11T16:58:16.429 INFO:teuthology.orchestra.run.smithi037.stdout:Total 2.2 MB/s | 89 kB 00:00 2024-02-11T16:58:16.429 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-02-11T16:58:16.439 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-02-11T16:58:16.440 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-02-11T16:58:16.440 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-02-11T16:58:16.440 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-02-11T16:58:16.492 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-02-11T16:58:16.496 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-02-11T16:58:16.502 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-02-11T16:58:16.506 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-02-11T16:58:16.606 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-02-11T16:58:16.790 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-11T16:58:16.965 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-02-11T16:58:17.078 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-11T16:58:17.164 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-11T16:58:17.481 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-11T16:58:17.482 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:58:17.482 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-02-11T16:58:17.482 INFO:teuthology.orchestra.run.smithi037.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2024-02-11T16:58:17.482 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:58:17.482 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-02-11T16:58:17.585 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-11T16:58:17.683 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install rbd-mirror 2024-02-11T16:58:17.941 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-11T16:58:17.941 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:58:17.941 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-02-11T16:58:17.941 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2024-02-11T16:58:17.941 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:58:17.941 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-02-11T16:58:18.166 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install rbd-nbd 2024-02-11T16:58:18.470 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:02:35 ago on Sun 11 Feb 2024 04:55:43 PM UTC. 2024-02-11T16:58:18.932 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:02:37 ago on Sun 11 Feb 2024 04:55:41 PM UTC. 2024-02-11T16:58:20.083 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-02-11T16:58:20.083 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-02-11T16:58:20.083 INFO:teuthology.orchestra.run.smithi037.stdout: Package Architecture Version Repository Size 2024-02-11T16:58:20.084 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-02-11T16:58:20.084 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-02-11T16:58:20.084 INFO:teuthology.orchestra.run.smithi037.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-02-11T16:58:20.084 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:58:20.084 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-02-11T16:58:20.084 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-02-11T16:58:20.084 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-02-11T16:58:20.084 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:58:20.085 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 3.7 M 2024-02-11T16:58:20.085 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 18 M 2024-02-11T16:58:20.085 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-02-11T16:58:20.189 INFO:teuthology.orchestra.run.smithi037.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 36 MB/s | 3.7 MB 00:00 2024-02-11T16:58:20.189 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-02-11T16:58:20.190 INFO:teuthology.orchestra.run.smithi037.stdout:Total 35 MB/s | 3.7 MB 00:00 2024-02-11T16:58:20.190 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-02-11T16:58:20.203 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-02-11T16:58:20.203 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-02-11T16:58:20.270 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-02-11T16:58:20.273 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-02-11T16:58:20.416 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2024-02-11T16:58:20.417 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-02-11T16:58:20.417 INFO:teuthology.orchestra.run.smithi178.stdout: Package Architecture Version Repository Size 2024-02-11T16:58:20.417 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-02-11T16:58:20.417 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2024-02-11T16:58:20.417 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2024-02-11T16:58:20.417 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:58:20.418 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2024-02-11T16:58:20.418 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2024-02-11T16:58:20.418 INFO:teuthology.orchestra.run.smithi178.stdout:Install 1 Package 2024-02-11T16:58:20.418 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:58:20.418 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 173 k 2024-02-11T16:58:20.418 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 649 k 2024-02-11T16:58:20.418 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2024-02-11T16:58:20.464 INFO:teuthology.orchestra.run.smithi178.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 3.7 MB/s | 173 kB 00:00 2024-02-11T16:58:20.464 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2024-02-11T16:58:20.464 INFO:teuthology.orchestra.run.smithi178.stdout:Total 3.6 MB/s | 173 kB 00:00 2024-02-11T16:58:20.465 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2024-02-11T16:58:20.475 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2024-02-11T16:58:20.475 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2024-02-11T16:58:20.526 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2024-02-11T16:58:20.529 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2024-02-11T16:58:20.649 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2024-02-11T16:58:20.763 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-02-11T16:58:20.871 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-11T16:58:20.966 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-11T16:58:21.167 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-11T16:58:21.412 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-11T16:58:21.554 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-11T16:58:21.554 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:58:21.554 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2024-02-11T16:58:21.554 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2024-02-11T16:58:21.554 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T16:58:21.555 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2024-02-11T16:58:21.754 DEBUG:teuthology.parallel:result is None 2024-02-11T16:58:21.817 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-11T16:58:21.818 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:58:21.818 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-02-11T16:58:21.818 INFO:teuthology.orchestra.run.smithi037.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2024-02-11T16:58:21.818 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:58:21.818 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-02-11T16:58:22.027 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install rbd-nbd 2024-02-11T16:58:22.824 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:02:39 ago on Sun 11 Feb 2024 04:55:43 PM UTC. 2024-02-11T16:58:24.438 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-02-11T16:58:24.438 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-02-11T16:58:24.439 INFO:teuthology.orchestra.run.smithi037.stdout: Package Architecture Version Repository Size 2024-02-11T16:58:24.439 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-02-11T16:58:24.439 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-02-11T16:58:24.439 INFO:teuthology.orchestra.run.smithi037.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2024-02-11T16:58:24.439 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:58:24.439 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-02-11T16:58:24.439 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-02-11T16:58:24.439 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-02-11T16:58:24.439 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:58:24.440 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 173 k 2024-02-11T16:58:24.440 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 649 k 2024-02-11T16:58:24.440 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-02-11T16:58:24.483 INFO:teuthology.orchestra.run.smithi037.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 3.9 MB/s | 173 kB 00:00 2024-02-11T16:58:24.484 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-02-11T16:58:24.484 INFO:teuthology.orchestra.run.smithi037.stdout:Total 3.8 MB/s | 173 kB 00:00 2024-02-11T16:58:24.484 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-02-11T16:58:24.496 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-02-11T16:58:24.496 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-02-11T16:58:24.549 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-02-11T16:58:24.552 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-02-11T16:58:24.672 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-02-11T16:58:24.848 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-11T16:58:25.161 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-11T16:58:25.558 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-11T16:58:25.558 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:58:25.558 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-02-11T16:58:25.558 INFO:teuthology.orchestra.run.smithi037.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2024-02-11T16:58:25.558 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-02-11T16:58:25.558 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-02-11T16:58:25.802 DEBUG:teuthology.parallel:result is None 2024-02-11T16:58:25.803 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-11T16:58:25.803 INFO:teuthology.packaging:ref: None 2024-02-11T16:58:25.803 INFO:teuthology.packaging:tag: v16.2.4 2024-02-11T16:58:25.803 INFO:teuthology.packaging:branch: None 2024-02-11T16:58:25.803 INFO:teuthology.packaging:sha1: 5db25abe00e218b6920215c0bef70af7e47c7152 2024-02-11T16:58:25.803 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-11T16:58:26.023 DEBUG:teuthology.orchestra.run.smithi037:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-11T16:58:26.047 INFO:teuthology.orchestra.run.smithi037.stdout:16.2.4-0.el8 2024-02-11T16:58:26.047 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2024-02-11T16:58:26.048 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2024-02-11T16:58:26.049 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-11T16:58:26.049 INFO:teuthology.packaging:ref: None 2024-02-11T16:58:26.050 INFO:teuthology.packaging:tag: v16.2.4 2024-02-11T16:58:26.050 INFO:teuthology.packaging:branch: None 2024-02-11T16:58:26.050 INFO:teuthology.packaging:sha1: 5db25abe00e218b6920215c0bef70af7e47c7152 2024-02-11T16:58:26.050 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-11T16:58:26.267 DEBUG:teuthology.orchestra.run.smithi178:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-11T16:58:26.294 INFO:teuthology.orchestra.run.smithi178.stdout:16.2.4-0.el8 2024-02-11T16:58:26.294 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2024-02-11T16:58:26.294 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2024-02-11T16:58:26.296 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-02-11T16:58:26.296 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-02-11T16:58:26.297 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-11T16:58:26.326 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-02-11T16:58:26.326 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-11T16:58:26.366 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-02-11T16:58:26.366 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-02-11T16:58:26.366 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/usr/bin/daemon-helper 2024-02-11T16:58:26.394 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-11T16:58:26.462 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-02-11T16:58:26.462 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/usr/bin/daemon-helper 2024-02-11T16:58:26.491 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-11T16:58:26.560 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-02-11T16:58:26.560 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-02-11T16:58:26.560 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-11T16:58:26.587 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-11T16:58:26.655 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-02-11T16:58:26.655 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-11T16:58:26.684 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-11T16:58:26.752 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-02-11T16:58:26.753 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-02-11T16:58:26.753 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/usr/bin/stdin-killer 2024-02-11T16:58:26.780 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-11T16:58:26.848 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-02-11T16:58:26.848 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/usr/bin/stdin-killer 2024-02-11T16:58:26.877 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-11T16:58:26.945 INFO:teuthology.run_tasks:Running task print... 2024-02-11T16:58:26.956 INFO:teuthology.task.print:**** done install task... 2024-02-11T16:58:26.956 INFO:teuthology.run_tasks:Running task cephadm... 2024-02-11T16:58:27.027 INFO:tasks.cephadm:Config: {'avoid_pacific_features': True, 'cephadm_branch': 'v16.2.4', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'docker.io/ceph/ceph:v16.2.4', 'roleless': True, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '5db25abe00e218b6920215c0bef70af7e47c7152'} 2024-02-11T16:58:27.028 INFO:tasks.cephadm:Cluster image is docker.io/ceph/ceph:v16.2.4 2024-02-11T16:58:27.028 INFO:tasks.cephadm:Cluster fsid is c65cd888-c8fe-11ee-95b9-87774f69a715 2024-02-11T16:58:27.028 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-02-11T16:58:27.028 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-02-11T16:58:27.029 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi037': '172.21.15.37', 'mon.smithi178': '172.21.15.178'} 2024-02-11T16:58:27.029 INFO:tasks.cephadm:Normalizing hostnames... 2024-02-11T16:58:27.029 DEBUG:teuthology.orchestra.run.smithi037:> sudo hostname $(hostname -s) 2024-02-11T16:58:27.057 DEBUG:teuthology.orchestra.run.smithi178:> sudo hostname $(hostname -s) 2024-02-11T16:58:27.085 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v16.2.4)... 2024-02-11T16:58:27.085 DEBUG:teuthology.orchestra.run.smithi037:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-11T16:58:27.317 INFO:teuthology.orchestra.run.smithi037.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Feb 11 16:58 /home/ubuntu/cephtest/cephadm 2024-02-11T16:58:27.317 DEBUG:teuthology.orchestra.run.smithi178:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-11T16:58:27.452 INFO:teuthology.orchestra.run.smithi178.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Feb 11 16:58 /home/ubuntu/cephtest/cephadm 2024-02-11T16:58:27.452 DEBUG:teuthology.orchestra.run.smithi037:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-11T16:58:27.472 DEBUG:teuthology.orchestra.run.smithi178:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-11T16:58:27.502 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /etc/ceph 2024-02-11T16:58:27.539 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /etc/ceph 2024-02-11T16:58:27.568 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod 777 /etc/ceph 2024-02-11T16:58:27.607 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod 777 /etc/ceph 2024-02-11T16:58:27.637 INFO:tasks.cephadm:Writing seed config... 2024-02-11T16:58:27.638 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-02-11T16:58:27.638 INFO:tasks.cephadm: override: [client] debug client = 20 2024-02-11T16:58:27.638 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-02-11T16:58:27.638 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-02-11T16:58:27.638 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-02-11T16:58:27.638 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-02-11T16:58:27.638 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-02-11T16:58:27.638 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-02-11T16:58:27.638 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-02-11T16:58:27.638 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-02-11T16:58:27.638 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-02-11T16:58:27.638 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-02-11T16:58:27.639 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-02-11T16:58:27.639 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-02-11T16:58:27.639 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-02-11T16:58:27.639 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-02-11T16:58:27.639 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-02-11T16:58:27.639 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-02-11T16:58:27.639 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-02-11T16:58:27.639 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-02-11T16:58:27.639 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-02-11T16:58:27.639 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-02-11T16:58:27.639 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-02-11T16:58:27.639 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-02-11T16:58:27.639 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-02-11T16:58:27.639 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-02-11T16:58:27.639 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-02-11T16:58:27.639 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-02-11T16:58:27.639 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-02-11T16:58:27.639 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-02-11T16:58:27.640 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-02-11T16:58:27.640 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-02-11T16:58:27.640 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-02-11T16:58:27.640 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-02-11T16:58:27.640 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-02-11T16:58:27.640 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-02-11T16:58:27.640 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-02-11T16:58:27.640 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-02-11T16:58:27.641 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-02-11T16:58:27.641 DEBUG:teuthology.orchestra.run.smithi037:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-02-11T16:58:27.666 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = c65cd888-c8fe-11ee-95b9-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-02-11T16:58:27.666 DEBUG:teuthology.orchestra.run.smithi037:mon.smithi037> sudo journalctl -f -n 0 -u ceph-c65cd888-c8fe-11ee-95b9-87774f69a715@mon.smithi037.service 2024-02-11T16:58:27.708 INFO:tasks.cephadm:Bootstrapping... 2024-02-11T16:58:27.708 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 -v bootstrap --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.37 && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-02-11T16:58:27.736 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:-- Logs begin at Sun 2024-02-11 16:48:29 UTC. -- 2024-02-11T16:58:27.977 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-02-11T16:58:28.900 INFO:teuthology.orchestra.run.smithi037.stderr:/bin/podman: 4.8.3 2024-02-11T16:58:28.914 INFO:teuthology.orchestra.run.smithi037.stderr:Verifying podman|docker is present... 2024-02-11T16:58:28.914 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-02-11T16:58:28.938 INFO:teuthology.orchestra.run.smithi037.stderr:/bin/podman: 4.8.3 2024-02-11T16:58:28.948 INFO:teuthology.orchestra.run.smithi037.stderr:Verifying lvm2 is present... 2024-02-11T16:58:28.949 INFO:teuthology.orchestra.run.smithi037.stderr:Verifying time synchronization is in place... 2024-02-11T16:58:28.949 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: systemctl is-enabled chrony.service 2024-02-11T16:58:28.956 INFO:teuthology.orchestra.run.smithi037.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-02-11T16:58:28.957 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: systemctl is-active chrony.service 2024-02-11T16:58:28.963 INFO:teuthology.orchestra.run.smithi037.stderr:systemctl: inactive 2024-02-11T16:58:28.963 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: systemctl is-enabled chronyd.service 2024-02-11T16:58:28.970 INFO:teuthology.orchestra.run.smithi037.stderr:systemctl: enabled 2024-02-11T16:58:28.970 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: systemctl is-active chronyd.service 2024-02-11T16:58:28.976 INFO:teuthology.orchestra.run.smithi037.stderr:systemctl: active 2024-02-11T16:58:28.977 INFO:teuthology.orchestra.run.smithi037.stderr:Unit chronyd.service is enabled and running 2024-02-11T16:58:28.977 INFO:teuthology.orchestra.run.smithi037.stderr:Repeating the final host check... 2024-02-11T16:58:28.977 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-02-11T16:58:28.998 INFO:teuthology.orchestra.run.smithi037.stderr:/bin/podman: 4.8.3 2024-02-11T16:58:29.010 INFO:teuthology.orchestra.run.smithi037.stderr:podman|docker (/bin/podman) is present 2024-02-11T16:58:29.010 INFO:teuthology.orchestra.run.smithi037.stderr:systemctl is present 2024-02-11T16:58:29.010 INFO:teuthology.orchestra.run.smithi037.stderr:lvcreate is present 2024-02-11T16:58:29.010 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: systemctl is-enabled chrony.service 2024-02-11T16:58:29.016 INFO:teuthology.orchestra.run.smithi037.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-02-11T16:58:29.016 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: systemctl is-active chrony.service 2024-02-11T16:58:29.022 INFO:teuthology.orchestra.run.smithi037.stderr:systemctl: inactive 2024-02-11T16:58:29.022 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: systemctl is-enabled chronyd.service 2024-02-11T16:58:29.028 INFO:teuthology.orchestra.run.smithi037.stderr:systemctl: enabled 2024-02-11T16:58:29.029 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: systemctl is-active chronyd.service 2024-02-11T16:58:29.034 INFO:teuthology.orchestra.run.smithi037.stderr:systemctl: active 2024-02-11T16:58:29.035 INFO:teuthology.orchestra.run.smithi037.stderr:Unit chronyd.service is enabled and running 2024-02-11T16:58:29.035 INFO:teuthology.orchestra.run.smithi037.stderr:Host looks OK 2024-02-11T16:58:29.035 INFO:teuthology.orchestra.run.smithi037.stderr:Cluster fsid: c65cd888-c8fe-11ee-95b9-87774f69a715 2024-02-11T16:58:29.035 INFO:teuthology.orchestra.run.smithi037.stderr:Acquiring lock 140264508033512 on /run/cephadm/c65cd888-c8fe-11ee-95b9-87774f69a715.lock 2024-02-11T16:58:29.036 INFO:teuthology.orchestra.run.smithi037.stderr:Lock 140264508033512 acquired on /run/cephadm/c65cd888-c8fe-11ee-95b9-87774f69a715.lock 2024-02-11T16:58:29.036 INFO:teuthology.orchestra.run.smithi037.stderr:Verifying IP 172.21.15.37 port 3300 ... 2024-02-11T16:58:29.036 INFO:teuthology.orchestra.run.smithi037.stderr:Verifying IP 172.21.15.37 port 6789 ... 2024-02-11T16:58:29.037 INFO:teuthology.orchestra.run.smithi037.stderr:Base mon IP is 172.21.15.37, final addrv is [v2:172.21.15.37:3300,v1:172.21.15.37:6789] 2024-02-11T16:58:29.037 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: /sbin/ip route ls 2024-02-11T16:58:29.039 INFO:teuthology.orchestra.run.smithi037.stderr:/sbin/ip: default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.37 metric 100 2024-02-11T16:58:29.039 INFO:teuthology.orchestra.run.smithi037.stderr:/sbin/ip: 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.37 metric 100 2024-02-11T16:58:29.040 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: /sbin/ip -6 route ls 2024-02-11T16:58:29.042 INFO:teuthology.orchestra.run.smithi037.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-02-11T16:58:29.042 INFO:teuthology.orchestra.run.smithi037.stderr:/sbin/ip: fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2024-02-11T16:58:29.042 INFO:teuthology.orchestra.run.smithi037.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1792sec hoplimit 64 pref medium 2024-02-11T16:58:29.042 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: /sbin/ip -6 addr ls 2024-02-11T16:58:29.045 INFO:teuthology.orchestra.run.smithi037.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-02-11T16:58:29.045 INFO:teuthology.orchestra.run.smithi037.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-02-11T16:58:29.045 INFO:teuthology.orchestra.run.smithi037.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-02-11T16:58:29.045 INFO:teuthology.orchestra.run.smithi037.stderr:/sbin/ip: 4: ens1f0: mtu 1500 state UP qlen 1000 2024-02-11T16:58:29.045 INFO:teuthology.orchestra.run.smithi037.stderr:/sbin/ip: inet6 fe80::ec4:7aff:fe88:7794/64 scope link 2024-02-11T16:58:29.045 INFO:teuthology.orchestra.run.smithi037.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-02-11T16:58:29.046 INFO:teuthology.orchestra.run.smithi037.stderr:Mon IP 172.21.15.37 is in CIDR network 172.21.0.0/20 2024-02-11T16:58:29.046 INFO:teuthology.orchestra.run.smithi037.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-02-11T16:58:29.047 INFO:teuthology.orchestra.run.smithi037.stderr:Pulling container image docker.io/ceph/ceph:v16.2.4... 2024-02-11T16:58:29.048 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: /bin/podman pull docker.io/ceph/ceph:v16.2.4 2024-02-11T16:58:29.141 INFO:teuthology.orchestra.run.smithi037.stderr:/bin/podman: Trying to pull docker.io/ceph/ceph:v16.2.4... 2024-02-11T16:58:35.789 INFO:teuthology.orchestra.run.smithi037.stderr:/bin/podman: Getting image source signatures 2024-02-11T16:58:35.807 INFO:teuthology.orchestra.run.smithi037.stderr:/bin/podman: Copying blob sha256:551101690791055467b1840be133b2e6b029d263771bc6c46a96bb7acdacac39 2024-02-11T16:58:35.807 INFO:teuthology.orchestra.run.smithi037.stderr:/bin/podman: Copying blob sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621 2024-02-11T16:58:50.130 INFO:teuthology.orchestra.run.smithi037.stderr:/bin/podman: Copying config sha256:8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2024-02-11T16:58:50.280 INFO:teuthology.orchestra.run.smithi037.stderr:/bin/podman: Writing manifest to image destination 2024-02-11T16:58:50.584 INFO:teuthology.orchestra.run.smithi037.stderr:/bin/podman: 8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2024-02-11T16:58:50.595 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi037 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --version 2024-02-11T16:58:51.256 INFO:teuthology.orchestra.run.smithi037.stderr:ceph: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2024-02-11T16:58:51.789 INFO:teuthology.orchestra.run.smithi037.stderr:Ceph version: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2024-02-11T16:58:51.789 INFO:teuthology.orchestra.run.smithi037.stderr:Extracting ceph user uid/gid from container image... 2024-02-11T16:58:51.790 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi037 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 -c %u %g /var/lib/ceph 2024-02-11T16:58:52.309 INFO:teuthology.orchestra.run.smithi037.stderr:stat: 167 167 2024-02-11T16:58:52.946 INFO:teuthology.orchestra.run.smithi037.stderr:Creating initial keys... 2024-02-11T16:58:52.946 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi037 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-02-11T16:58:53.498 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-authtool: AQDN/MhlOGWMFxAAKL44o81ukgYBcYQ0AkaLnQ== 2024-02-11T16:58:54.175 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi037 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-02-11T16:58:54.708 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-authtool: AQDO/MhlQ8YPIxAALYs7vpFLlWP81zDWM27IeA== 2024-02-11T16:58:55.294 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi037 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-02-11T16:58:55.867 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-authtool: AQDP/MhldbEQLBAAZJKi180oZncwKh124pbTiw== 2024-02-11T16:58:56.316 INFO:teuthology.orchestra.run.smithi037.stderr:Creating initial monmap... 2024-02-11T16:58:56.316 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/monmaptool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi037 -e CEPH_USE_RANDOM_NONCE=1 -v /tmp/ceph-tmpikktqo7f:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --create --clobber --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 --addv smithi037 [v2:172.21.15.37:3300,v1:172.21.15.37:6789] /tmp/monmap 2024-02-11T16:58:56.857 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-11T16:58:56.857 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to c65cd888-c8fe-11ee-95b9-87774f69a715 2024-02-11T16:58:56.858 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-11T16:58:57.528 INFO:teuthology.orchestra.run.smithi037.stderr:monmaptool for smithi037 [v2:172.21.15.37:3300,v1:172.21.15.37:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-11T16:58:57.529 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/monmaptool: set fsid to c65cd888-c8fe-11ee-95b9-87774f69a715 2024-02-11T16:58:57.529 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-11T16:58:57.529 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-02-11T16:58:57.529 INFO:teuthology.orchestra.run.smithi037.stderr:Creating mon... 2024-02-11T16:58:57.529 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-mon --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi037 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715/mon.smithi037:/var/lib/ceph/mon/ceph-smithi037:z -v /tmp/ceph-tmp7ugtp7ds:/tmp/keyring:z -v /tmp/ceph-tmpikktqo7f:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --mkfs -i smithi037 --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -c /dev/null --monmap /tmp/monmap --keyring /tmp/keyring --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true 2024-02-11T16:58:58.106 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.002+0000 7f13c4611700 0 set uid:gid to 167:167 (ceph:ceph) 2024-02-11T16:58:58.106 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.002+0000 7f13c4611700 1 imported monmap: 2024-02-11T16:58:58.106 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: epoch 0 2024-02-11T16:58:58.107 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: fsid c65cd888-c8fe-11ee-95b9-87774f69a715 2024-02-11T16:58:58.107 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: last_changed 2024-02-11T16:58:56.764219+0000 2024-02-11T16:58:58.107 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: created 2024-02-11T16:58:56.764219+0000 2024-02-11T16:58:58.107 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2024-02-11T16:58:58.107 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: election_strategy: 1 2024-02-11T16:58:58.107 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.smithi037 2024-02-11T16:58:58.107 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: 2024-02-11T16:58:58.107 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.003+0000 7f13c4611700 0 /usr/bin/ceph-mon: set fsid to c65cd888-c8fe-11ee-95b9-87774f69a715 2024-02-11T16:58:58.107 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.003+0000 7f13c4611700 4 rocksdb: RocksDB version: 6.8.1 2024-02-11T16:58:58.107 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: 2024-02-11T16:58:58.107 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.003+0000 7f13c4611700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-11T16:58:58.107 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.003+0000 7f13c4611700 4 rocksdb: Compile date May 13 2021 2024-02-11T16:58:58.107 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.003+0000 7f13c4611700 4 rocksdb: DB SUMMARY 2024-02-11T16:58:58.107 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: 2024-02-11T16:58:58.107 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.003+0000 7f13c4611700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi037/store.db dir, Total Num: 0, files: 2024-02-11T16:58:58.108 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: 2024-02-11T16:58:58.108 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.003+0000 7f13c4611700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi037/store.db: 2024-02-11T16:58:58.108 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: 2024-02-11T16:58:58.108 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.003+0000 7f13c4611700 4 rocksdb: Options.error_if_exists: 0 2024-02-11T16:58:58.108 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.003+0000 7f13c4611700 4 rocksdb: Options.create_if_missing: 1 2024-02-11T16:58:58.108 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.003+0000 7f13c4611700 4 rocksdb: Options.paranoid_checks: 1 2024-02-11T16:58:58.108 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.003+0000 7f13c4611700 4 rocksdb: Options.env: 0x55fd1222b1c0 2024-02-11T16:58:58.108 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.003+0000 7f13c4611700 4 rocksdb: Options.fs: Posix File System 2024-02-11T16:58:58.108 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.003+0000 7f13c4611700 4 rocksdb: Options.info_log: 0x55fd14460ec0 2024-02-11T16:58:58.108 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.003+0000 7f13c4611700 4 rocksdb: Options.max_file_opening_threads: 16 2024-02-11T16:58:58.108 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.003+0000 7f13c4611700 4 rocksdb: Options.statistics: (nil) 2024-02-11T16:58:58.108 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.003+0000 7f13c4611700 4 rocksdb: Options.use_fsync: 0 2024-02-11T16:58:58.108 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.003+0000 7f13c4611700 4 rocksdb: Options.max_log_file_size: 0 2024-02-11T16:58:58.109 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.003+0000 7f13c4611700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-11T16:58:58.109 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.003+0000 7f13c4611700 4 rocksdb: Options.log_file_time_to_roll: 0 2024-02-11T16:58:58.109 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.003+0000 7f13c4611700 4 rocksdb: Options.keep_log_file_num: 1000 2024-02-11T16:58:58.109 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.003+0000 7f13c4611700 4 rocksdb: Options.recycle_log_file_num: 0 2024-02-11T16:58:58.109 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.003+0000 7f13c4611700 4 rocksdb: Options.allow_fallocate: 1 2024-02-11T16:58:58.109 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.003+0000 7f13c4611700 4 rocksdb: Options.allow_mmap_reads: 0 2024-02-11T16:58:58.109 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.003+0000 7f13c4611700 4 rocksdb: Options.allow_mmap_writes: 0 2024-02-11T16:58:58.109 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.003+0000 7f13c4611700 4 rocksdb: Options.use_direct_reads: 0 2024-02-11T16:58:58.109 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.003+0000 7f13c4611700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-11T16:58:58.109 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.003+0000 7f13c4611700 4 rocksdb: Options.create_missing_column_families: 0 2024-02-11T16:58:58.109 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.003+0000 7f13c4611700 4 rocksdb: Options.db_log_dir: 2024-02-11T16:58:58.109 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.003+0000 7f13c4611700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi037/store.db 2024-02-11T16:58:58.109 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.003+0000 7f13c4611700 4 rocksdb: Options.table_cache_numshardbits: 6 2024-02-11T16:58:58.109 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.003+0000 7f13c4611700 4 rocksdb: Options.max_subcompactions: 1 2024-02-11T16:58:58.110 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.003+0000 7f13c4611700 4 rocksdb: Options.max_background_flushes: -1 2024-02-11T16:58:58.110 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.003+0000 7f13c4611700 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-02-11T16:58:58.110 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.003+0000 7f13c4611700 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-02-11T16:58:58.110 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.003+0000 7f13c4611700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-11T16:58:58.110 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.003+0000 7f13c4611700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-11T16:58:58.110 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.003+0000 7f13c4611700 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-02-11T16:58:58.110 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.003+0000 7f13c4611700 4 rocksdb: Options.advise_random_on_open: 1 2024-02-11T16:58:58.110 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.003+0000 7f13c4611700 4 rocksdb: Options.db_write_buffer_size: 0 2024-02-11T16:58:58.110 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.003+0000 7f13c4611700 4 rocksdb: Options.write_buffer_manager: 0x55fd14467530 2024-02-11T16:58:58.110 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.004+0000 7f13c4611700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-11T16:58:58.110 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.004+0000 7f13c4611700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-11T16:58:58.110 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.004+0000 7f13c4611700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-11T16:58:58.110 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.004+0000 7f13c4611700 4 rocksdb: Options.use_adaptive_mutex: 0 2024-02-11T16:58:58.110 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.004+0000 7f13c4611700 4 rocksdb: Options.rate_limiter: (nil) 2024-02-11T16:58:58.111 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.004+0000 7f13c4611700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-11T16:58:58.111 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.004+0000 7f13c4611700 4 rocksdb: Options.wal_recovery_mode: 2 2024-02-11T16:58:58.111 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.004+0000 7f13c4611700 4 rocksdb: Options.enable_thread_tracking: 0 2024-02-11T16:58:58.111 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.004+0000 7f13c4611700 4 rocksdb: Options.enable_pipelined_write: 0 2024-02-11T16:58:58.111 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.004+0000 7f13c4611700 4 rocksdb: Options.unordered_write: 0 2024-02-11T16:58:58.111 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.004+0000 7f13c4611700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-11T16:58:58.111 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.004+0000 7f13c4611700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-11T16:58:58.111 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.004+0000 7f13c4611700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-11T16:58:58.111 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.004+0000 7f13c4611700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-11T16:58:58.111 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.004+0000 7f13c4611700 4 rocksdb: Options.row_cache: None 2024-02-11T16:58:58.111 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.004+0000 7f13c4611700 4 rocksdb: Options.wal_filter: None 2024-02-11T16:58:58.111 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.004+0000 7f13c4611700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-11T16:58:58.111 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.004+0000 7f13c4611700 4 rocksdb: Options.allow_ingest_behind: 0 2024-02-11T16:58:58.111 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.004+0000 7f13c4611700 4 rocksdb: Options.preserve_deletes: 0 2024-02-11T16:58:58.112 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.004+0000 7f13c4611700 4 rocksdb: Options.two_write_queues: 0 2024-02-11T16:58:58.112 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.004+0000 7f13c4611700 4 rocksdb: Options.manual_wal_flush: 0 2024-02-11T16:58:58.112 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.004+0000 7f13c4611700 4 rocksdb: Options.atomic_flush: 0 2024-02-11T16:58:58.112 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.004+0000 7f13c4611700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-11T16:58:58.112 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.004+0000 7f13c4611700 4 rocksdb: Options.persist_stats_to_disk: 0 2024-02-11T16:58:58.112 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.004+0000 7f13c4611700 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-02-11T16:58:58.112 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.004+0000 7f13c4611700 4 rocksdb: Options.log_readahead_size: 0 2024-02-11T16:58:58.112 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.004+0000 7f13c4611700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-02-11T16:58:58.112 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.004+0000 7f13c4611700 4 rocksdb: Options.max_background_jobs: 2 2024-02-11T16:58:58.112 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.004+0000 7f13c4611700 4 rocksdb: Options.max_background_compactions: -1 2024-02-11T16:58:58.112 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.004+0000 7f13c4611700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-11T16:58:58.112 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.004+0000 7f13c4611700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-11T16:58:58.112 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.004+0000 7f13c4611700 4 rocksdb: Options.delayed_write_rate : 16777216 2024-02-11T16:58:58.112 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.004+0000 7f13c4611700 4 rocksdb: Options.max_total_wal_size: 0 2024-02-11T16:58:58.113 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.004+0000 7f13c4611700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-11T16:58:58.113 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.004+0000 7f13c4611700 4 rocksdb: Options.stats_dump_period_sec: 600 2024-02-11T16:58:58.113 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.004+0000 7f13c4611700 4 rocksdb: Options.stats_persist_period_sec: 600 2024-02-11T16:58:58.113 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.004+0000 7f13c4611700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-11T16:58:58.113 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.004+0000 7f13c4611700 4 rocksdb: Options.max_open_files: -1 2024-02-11T16:58:58.113 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.004+0000 7f13c4611700 4 rocksdb: Options.bytes_per_sync: 0 2024-02-11T16:58:58.113 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.004+0000 7f13c4611700 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-02-11T16:58:58.113 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.004+0000 7f13c4611700 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-02-11T16:58:58.113 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.004+0000 7f13c4611700 4 rocksdb: Options.compaction_readahead_size: 0 2024-02-11T16:58:58.113 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.004+0000 7f13c4611700 4 rocksdb: Compression algorithms supported: 2024-02-11T16:58:58.113 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.004+0000 7f13c4611700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-11T16:58:58.113 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.004+0000 7f13c4611700 4 rocksdb: kZSTD supported: 0 2024-02-11T16:58:58.113 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.004+0000 7f13c4611700 4 rocksdb: kXpressCompression supported: 0 2024-02-11T16:58:58.113 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.004+0000 7f13c4611700 4 rocksdb: kLZ4HCCompression supported: 1 2024-02-11T16:58:58.113 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.004+0000 7f13c4611700 4 rocksdb: kLZ4Compression supported: 1 2024-02-11T16:58:58.113 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.004+0000 7f13c4611700 4 rocksdb: kBZip2Compression supported: 0 2024-02-11T16:58:58.114 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.004+0000 7f13c4611700 4 rocksdb: kZlibCompression supported: 1 2024-02-11T16:58:58.114 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.004+0000 7f13c4611700 4 rocksdb: kSnappyCompression supported: 1 2024-02-11T16:58:58.114 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.004+0000 7f13c4611700 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-02-11T16:58:58.114 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.004+0000 7f13c4611700 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2024-02-11T16:58:58.114 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: 2024-02-11T16:58:58.114 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.004+0000 7f13c4611700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi037/store.db/MANIFEST-000001 2024-02-11T16:58:58.114 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: 2024-02-11T16:58:58.114 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.004+0000 7f13c4611700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-02-11T16:58:58.114 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: 2024-02-11T16:58:58.114 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.004+0000 7f13c4611700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-11T16:58:58.114 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.004+0000 7f13c4611700 4 rocksdb: Options.merge_operator: 2024-02-11T16:58:58.114 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.004+0000 7f13c4611700 4 rocksdb: Options.compaction_filter: None 2024-02-11T16:58:58.114 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.004+0000 7f13c4611700 4 rocksdb: Options.compaction_filter_factory: None 2024-02-11T16:58:58.114 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.004+0000 7f13c4611700 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-02-11T16:58:58.114 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.004+0000 7f13c4611700 4 rocksdb: Options.table_factory: BlockBasedTable 2024-02-11T16:58:58.115 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55fd14379cb8) 2024-02-11T16:58:58.115 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2024-02-11T16:58:58.115 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-11T16:58:58.115 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-11T16:58:58.115 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2024-02-11T16:58:58.115 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: index_type: 0 2024-02-11T16:58:58.115 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2024-02-11T16:58:58.115 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: index_shortening: 1 2024-02-11T16:58:58.115 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2024-02-11T16:58:58.115 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2024-02-11T16:58:58.115 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: checksum: 1 2024-02-11T16:58:58.115 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2024-02-11T16:58:58.115 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: block_cache: 0x55fd143aed10 2024-02-11T16:58:58.115 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2024-02-11T16:58:58.115 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: block_cache_options: 2024-02-11T16:58:58.116 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: capacity : 536870912 2024-02-11T16:58:58.116 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2024-02-11T16:58:58.116 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2024-02-11T16:58:58.116 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2024-02-11T16:58:58.116 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2024-02-11T16:58:58.116 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2024-02-11T16:58:58.116 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: block_size: 4096 2024-02-11T16:58:58.116 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2024-02-11T16:58:58.116 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2024-02-11T16:58:58.116 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2024-02-11T16:58:58.116 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2024-02-11T16:58:58.116 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: partition_filters: 0 2024-02-11T16:58:58.116 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2024-02-11T16:58:58.116 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-11T16:58:58.117 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2024-02-11T16:58:58.117 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: verify_compression: 0 2024-02-11T16:58:58.117 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2024-02-11T16:58:58.117 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: format_version: 2 2024-02-11T16:58:58.117 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2024-02-11T16:58:58.117 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: block_align: 0 2024-02-11T16:58:58.117 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: 2024-02-11T16:58:58.117 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.write_buffer_size: 33554432 2024-02-11T16:58:58.117 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.max_write_buffer_number: 2 2024-02-11T16:58:58.117 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.compression: NoCompression 2024-02-11T16:58:58.117 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.bottommost_compression: Disabled 2024-02-11T16:58:58.117 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.prefix_extractor: nullptr 2024-02-11T16:58:58.117 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-11T16:58:58.117 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.num_levels: 7 2024-02-11T16:58:58.118 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-11T16:58:58.118 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-11T16:58:58.118 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-11T16:58:58.118 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-11T16:58:58.118 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-11T16:58:58.118 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-11T16:58:58.118 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-11T16:58:58.118 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-11T16:58:58.118 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-11T16:58:58.118 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.compression_opts.window_bits: -14 2024-02-11T16:58:58.118 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.compression_opts.level: 32767 2024-02-11T16:58:58.118 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.compression_opts.strategy: 0 2024-02-11T16:58:58.118 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-11T16:58:58.118 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-11T16:58:58.118 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.compression_opts.enabled: false 2024-02-11T16:58:58.119 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-11T16:58:58.119 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-11T16:58:58.119 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-11T16:58:58.119 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.target_file_size_base: 67108864 2024-02-11T16:58:58.119 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.target_file_size_multiplier: 1 2024-02-11T16:58:58.119 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-11T16:58:58.119 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-11T16:58:58.119 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-11T16:58:58.119 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-11T16:58:58.119 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-11T16:58:58.119 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-11T16:58:58.119 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-11T16:58:58.119 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-11T16:58:58.119 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-11T16:58:58.119 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-11T16:58:58.119 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-11T16:58:58.120 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-11T16:58:58.120 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.arena_block_size: 4194304 2024-02-11T16:58:58.120 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-11T16:58:58.120 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-11T16:58:58.120 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-11T16:58:58.120 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.disable_auto_compactions: 0 2024-02-11T16:58:58.120 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-11T16:58:58.120 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-11T16:58:58.120 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-11T16:58:58.120 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-11T16:58:58.120 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-11T16:58:58.120 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-11T16:58:58.120 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-11T16:58:58.120 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-11T16:58:58.121 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-11T16:58:58.121 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-11T16:58:58.121 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.table_properties_collectors: 2024-02-11T16:58:58.121 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.inplace_update_support: 0 2024-02-11T16:58:58.121 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-02-11T16:58:58.121 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-11T16:58:58.121 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-11T16:58:58.121 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.memtable_huge_page_size: 0 2024-02-11T16:58:58.121 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.bloom_locality: 0 2024-02-11T16:58:58.121 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.max_successive_merges: 0 2024-02-11T16:58:58.121 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-02-11T16:58:58.121 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.paranoid_file_checks: 0 2024-02-11T16:58:58.121 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.force_consistency_checks: 0 2024-02-11T16:58:58.121 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.report_bg_io_stats: 0 2024-02-11T16:58:58.122 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.ttl: 2592000 2024-02-11T16:58:58.122 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-02-11T16:58:58.122 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi037/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-11T16:58:58.122 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: 2024-02-11T16:58:58.122 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.005+0000 7f13c4611700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2024-02-11T16:58:58.122 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: 2024-02-11T16:58:58.122 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.006+0000 7f13c4611700 4 rocksdb: DB pointer 0x55fd14473800 2024-02-11T16:58:58.122 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.006+0000 7f13ad4aa700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-02-11T16:58:58.122 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.006+0000 7f13ad4aa700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-02-11T16:58:58.122 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2024-02-11T16:58:58.122 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-02-11T16:58:58.122 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-11T16:58:58.122 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-11T16:58:58.122 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-11T16:58:58.123 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-11T16:58:58.123 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-11T16:58:58.123 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-11T16:58:58.123 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: 2024-02-11T16:58:58.123 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-11T16:58:58.123 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-11T16:58:58.123 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-11T16:58:58.123 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-11T16:58:58.123 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-11T16:58:58.123 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: 2024-02-11T16:58:58.123 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-11T16:58:58.123 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-11T16:58:58.123 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-11T16:58:58.123 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-02-11T16:58:58.123 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-02-11T16:58:58.124 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-11T16:58:58.124 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-02-11T16:58:58.124 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-02-11T16:58:58.124 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-02-11T16:58:58.124 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-11T16:58:58.124 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-11T16:58:58.124 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-11T16:58:58.124 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: 2024-02-11T16:58:58.124 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-02-11T16:58:58.124 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: 2024-02-11T16:58:58.124 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-11T16:58:58.124 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-11T16:58:58.124 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-11T16:58:58.124 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-11T16:58:58.125 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-11T16:58:58.125 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: 2024-02-11T16:58:58.125 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-11T16:58:58.125 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-11T16:58:58.125 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-11T16:58:58.125 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-02-11T16:58:58.125 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-02-11T16:58:58.125 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-11T16:58:58.125 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-02-11T16:58:58.125 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-02-11T16:58:58.125 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-02-11T16:58:58.125 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-11T16:58:58.125 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-11T16:58:58.125 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-11T16:58:58.125 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: 2024-02-11T16:58:58.126 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-02-11T16:58:58.126 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: 2024-02-11T16:58:58.126 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.008+0000 7f13c4611700 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2024-02-11T16:58:58.126 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.008+0000 7f13c4611700 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2024-02-11T16:58:58.126 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph-mon: debug 2024-02-11T16:58:58.008+0000 7f13c4611700 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-smithi037 for mon.smithi037 2024-02-11T16:58:58.692 INFO:teuthology.orchestra.run.smithi037.stderr:create mon.smithi037 on 2024-02-11T16:58:58.693 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: install -d -m0770 -o 167 -g 167 /var/run/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715 2024-02-11T16:58:58.699 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: systemctl disable ceph.target 2024-02-11T16:58:58.707 INFO:teuthology.orchestra.run.smithi037.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-02-11T16:58:58.808 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: systemctl enable ceph.target 2024-02-11T16:58:58.815 INFO:teuthology.orchestra.run.smithi037.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-02-11T16:58:58.901 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: systemctl start ceph.target 2024-02-11T16:58:58.908 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: systemctl enable ceph-c65cd888-c8fe-11ee-95b9-87774f69a715.target 2024-02-11T16:58:58.915 INFO:teuthology.orchestra.run.smithi037.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-c65cd888-c8fe-11ee-95b9-87774f69a715.target → /etc/systemd/system/ceph-c65cd888-c8fe-11ee-95b9-87774f69a715.target. 2024-02-11T16:58:58.915 INFO:teuthology.orchestra.run.smithi037.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-c65cd888-c8fe-11ee-95b9-87774f69a715.target → /etc/systemd/system/ceph-c65cd888-c8fe-11ee-95b9-87774f69a715.target. 2024-02-11T16:58:59.000 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: systemctl start ceph-c65cd888-c8fe-11ee-95b9-87774f69a715.target 2024-02-11T16:58:59.007 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: systemctl daemon-reload 2024-02-11T16:58:59.098 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: systemctl stop ceph-c65cd888-c8fe-11ee-95b9-87774f69a715@mon.smithi037 2024-02-11T16:58:59.136 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: systemctl reset-failed ceph-c65cd888-c8fe-11ee-95b9-87774f69a715@mon.smithi037 2024-02-11T16:58:59.141 INFO:teuthology.orchestra.run.smithi037.stderr:systemctl: Failed to reset failed state of unit ceph-c65cd888-c8fe-11ee-95b9-87774f69a715@mon.smithi037.service: Unit ceph-c65cd888-c8fe-11ee-95b9-87774f69a715@mon.smithi037.service not loaded. 2024-02-11T16:58:59.142 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: systemctl enable ceph-c65cd888-c8fe-11ee-95b9-87774f69a715@mon.smithi037 2024-02-11T16:58:59.148 INFO:teuthology.orchestra.run.smithi037.stderr:systemctl: Created symlink /etc/systemd/system/ceph-c65cd888-c8fe-11ee-95b9-87774f69a715.target.wants/ceph-c65cd888-c8fe-11ee-95b9-87774f69a715@mon.smithi037.service → /etc/systemd/system/ceph-c65cd888-c8fe-11ee-95b9-87774f69a715@.service. 2024-02-11T16:58:59.241 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: systemctl start ceph-c65cd888-c8fe-11ee-95b9-87774f69a715@mon.smithi037 2024-02-11T16:58:59.283 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:58:59 smithi037 systemd[1]: Starting Ceph mon.smithi037 for c65cd888-c8fe-11ee-95b9-87774f69a715... 2024-02-11T16:59:00.112 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:58:59 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[38945]: cluster 2024-02-11T16:58:59.740379+0000 mon.smithi037 (mon.0) 0 : [INF] mkfs c65cd888-c8fe-11ee-95b9-87774f69a715 2024-02-11T16:59:00.112 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:58:59 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[38945]: cluster 2024-02-11T16:58:59.735473+0000 mon.smithi037 (mon.0) 1 : cluster [INF] mon.smithi037 is new leader, mons smithi037 in quorum (ranks 0) 2024-02-11T16:59:00.116 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: systemctl is-enabled firewalld.service 2024-02-11T16:59:00.125 INFO:teuthology.orchestra.run.smithi037.stderr:systemctl: disabled 2024-02-11T16:59:00.126 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: systemctl is-active firewalld.service 2024-02-11T16:59:00.134 INFO:teuthology.orchestra.run.smithi037.stderr:systemctl: inactive 2024-02-11T16:59:00.135 INFO:teuthology.orchestra.run.smithi037.stderr:firewalld.service is not enabled 2024-02-11T16:59:00.135 INFO:teuthology.orchestra.run.smithi037.stderr:Not possible to enable service . firewalld.service is not available 2024-02-11T16:59:00.135 INFO:teuthology.orchestra.run.smithi037.stderr:Not possible to open ports <[]>. firewalld.service is not available 2024-02-11T16:59:00.136 INFO:teuthology.orchestra.run.smithi037.stderr:Waiting for mon to start... 2024-02-11T16:59:00.136 INFO:teuthology.orchestra.run.smithi037.stderr:Waiting for mon... 2024-02-11T16:59:00.136 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi037 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715/mon.smithi037:/var/lib/ceph/mon/ceph-smithi037:z -v /tmp/ceph-tmpjftqjcr4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4ata0hlm:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status 2024-02-11T16:59:00.372 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:00 smithi037 bash[38930]: c1c4637732f6a22ae196ac8afa89c5b5b6b417a072e6083bfdb43ec12718ecfd 2024-02-11T16:59:00.372 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:00 smithi037 systemd[1]: Started Ceph mon.smithi037 for c65cd888-c8fe-11ee-95b9-87774f69a715. 2024-02-11T16:59:00.669 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:00.622+0000 7fe41bdd0700 1 Processor -- start 2024-02-11T16:59:00.669 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:00.623+0000 7fe41bdd0700 1 -- start start 2024-02-11T16:59:00.669 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:00.623+0000 7fe41bdd0700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4140fab80 0x7fe4140fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:00.669 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:00.623+0000 7fe41bdd0700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4140fb490 con 0x7fe4140fab80 2024-02-11T16:59:00.669 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:00.623+0000 7fe419b6c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4140fab80 0x7fe4140fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:00.669 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:00.623+0000 7fe419b6c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4140fab80 0x7fe4140fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:51460/0 (socket says 172.21.15.37:51460) 2024-02-11T16:59:00.669 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:00.623+0000 7fe419b6c700 1 -- 172.21.15.37:0/4171131962 learned_addr learned my addr 172.21.15.37:0/4171131962 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-02-11T16:59:00.669 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:00.624+0000 7fe419b6c700 1 -- 172.21.15.37:0/4171131962 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4140fb5d0 con 0x7fe4140fab80 2024-02-11T16:59:00.669 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:00.624+0000 7fe419b6c700 1 --2- 172.21.15.37:0/4171131962 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4140fab80 0x7fe4140fafa0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fe40400b5f0 tx=0x7fe404006ee0).ready entity=mon.0 client_cookie=ecb530f8a4731645 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:00.669 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:00.624+0000 7fe418b6a700 1 -- 172.21.15.37:0/4171131962 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe40400d600 con 0x7fe4140fab80 2024-02-11T16:59:00.669 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:00.624+0000 7fe418b6a700 1 -- 172.21.15.37:0/4171131962 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe40400d760 con 0x7fe4140fab80 2024-02-11T16:59:00.669 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:00.624+0000 7fe418b6a700 1 -- 172.21.15.37:0/4171131962 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe40400da30 con 0x7fe4140fab80 2024-02-11T16:59:00.670 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:00.624+0000 7fe41bdd0700 1 -- 172.21.15.37:0/4171131962 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4140fab80 msgr2=0x7fe4140fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:00.670 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:00.624+0000 7fe41bdd0700 1 --2- 172.21.15.37:0/4171131962 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4140fab80 0x7fe4140fafa0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fe40400b5f0 tx=0x7fe404006ee0).stop 2024-02-11T16:59:00.670 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:00.624+0000 7fe41bdd0700 1 -- 172.21.15.37:0/4171131962 shutdown_connections 2024-02-11T16:59:00.670 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:00.624+0000 7fe41bdd0700 1 --2- 172.21.15.37:0/4171131962 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4140fab80 0x7fe4140fafa0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:00.670 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:00.624+0000 7fe41bdd0700 1 -- 172.21.15.37:0/4171131962 >> 172.21.15.37:0/4171131962 conn(0x7fe4140f63c0 msgr2=0x7fe4140f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T16:59:00.670 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:00.625+0000 7fe41bdd0700 1 -- 172.21.15.37:0/4171131962 shutdown_connections 2024-02-11T16:59:00.670 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:00.625+0000 7fe41bdd0700 1 -- 172.21.15.37:0/4171131962 wait complete. 2024-02-11T16:59:00.670 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:00.625+0000 7fe41bdd0700 1 Processor -- start 2024-02-11T16:59:00.670 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:00.625+0000 7fe41bdd0700 1 -- start start 2024-02-11T16:59:00.670 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:00.625+0000 7fe41bdd0700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4140fab80 0x7fe41410cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:00.670 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:00.625+0000 7fe41bdd0700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4140704b0 con 0x7fe4140fab80 2024-02-11T16:59:00.670 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:00.625+0000 7fe419b6c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4140fab80 0x7fe41410cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:00.670 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:00.625+0000 7fe419b6c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4140fab80 0x7fe41410cf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:51466/0 (socket says 172.21.15.37:51466) 2024-02-11T16:59:00.670 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:00.625+0000 7fe419b6c700 1 -- 172.21.15.37:0/3603562370 learned_addr learned my addr 172.21.15.37:0/3603562370 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-02-11T16:59:00.671 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:00.626+0000 7fe419b6c700 1 -- 172.21.15.37:0/3603562370 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe40400b200 con 0x7fe4140fab80 2024-02-11T16:59:00.671 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:00.626+0000 7fe419b6c700 1 --2- 172.21.15.37:0/3603562370 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4140fab80 0x7fe41410cf10 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fe404006c70 tx=0x7fe404016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:00.671 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:00.626+0000 7fe4127fc700 1 -- 172.21.15.37:0/3603562370 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe404011b40 con 0x7fe4140fab80 2024-02-11T16:59:00.671 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:00.626+0000 7fe4127fc700 1 -- 172.21.15.37:0/3603562370 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe404011ca0 con 0x7fe4140fab80 2024-02-11T16:59:00.671 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:00.626+0000 7fe41bdd0700 1 -- 172.21.15.37:0/3603562370 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4140706b0 con 0x7fe4140fab80 2024-02-11T16:59:00.671 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:00.626+0000 7fe4127fc700 1 -- 172.21.15.37:0/3603562370 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe40401d430 con 0x7fe4140fab80 2024-02-11T16:59:00.671 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:00.626+0000 7fe41bdd0700 1 -- 172.21.15.37:0/3603562370 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe41410d5c0 con 0x7fe4140fab80 2024-02-11T16:59:00.671 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:00.627+0000 7fe41bdd0700 1 -- 172.21.15.37:0/3603562370 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe414045ad0 con 0x7fe4140fab80 2024-02-11T16:59:00.671 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:00.627+0000 7fe4127fc700 1 -- 172.21.15.37:0/3603562370 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fe40400f070 con 0x7fe4140fab80 2024-02-11T16:59:00.671 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:00.627+0000 7fe4127fc700 1 -- 172.21.15.37:0/3603562370 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fe404027460 con 0x7fe4140fab80 2024-02-11T16:59:00.671 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:00.629+0000 7fe4127fc700 1 -- 172.21.15.37:0/3603562370 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fe404016070 con 0x7fe4140fab80 2024-02-11T16:59:00.700 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:00.699+0000 7fe41bdd0700 1 -- 172.21.15.37:0/3603562370 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fe4140fbec0 con 0x7fe4140fab80 2024-02-11T16:59:00.702 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:00.699+0000 7fe4127fc700 1 -- 172.21.15.37:0/3603562370 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7fe40401d620 con 0x7fe4140fab80 2024-02-11T16:59:00.703 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: cluster: 2024-02-11T16:59:00.703 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: id: c65cd888-c8fe-11ee-95b9-87774f69a715 2024-02-11T16:59:00.703 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-02-11T16:59:00.703 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:00.703 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: services: 2024-02-11T16:59:00.703 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi037 (age 0.961439s) 2024-02-11T16:59:00.703 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: mgr: no daemons active 2024-02-11T16:59:00.703 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-02-11T16:59:00.703 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:00.703 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: data: 2024-02-11T16:59:00.703 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-02-11T16:59:00.703 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-02-11T16:59:00.703 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-02-11T16:59:00.703 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: pgs: 2024-02-11T16:59:00.703 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:00.704 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:00.700+0000 7fe41bdd0700 1 -- 172.21.15.37:0/3603562370 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4140fab80 msgr2=0x7fe41410cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:00.704 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:00.700+0000 7fe41bdd0700 1 --2- 172.21.15.37:0/3603562370 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4140fab80 0x7fe41410cf10 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fe404006c70 tx=0x7fe404016000).stop 2024-02-11T16:59:00.704 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:00.700+0000 7fe41bdd0700 1 -- 172.21.15.37:0/3603562370 shutdown_connections 2024-02-11T16:59:00.704 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:00.700+0000 7fe41bdd0700 1 --2- 172.21.15.37:0/3603562370 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4140fab80 0x7fe41410cf10 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:00.704 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:00.700+0000 7fe41bdd0700 1 -- 172.21.15.37:0/3603562370 >> 172.21.15.37:0/3603562370 conn(0x7fe4140f63c0 msgr2=0x7fe4140fd900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T16:59:00.704 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:00.700+0000 7fe41bdd0700 1 -- 172.21.15.37:0/3603562370 shutdown_connections 2024-02-11T16:59:00.704 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:00.700+0000 7fe41bdd0700 1 -- 172.21.15.37:0/3603562370 wait complete. 2024-02-11T16:59:01.052 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:00 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[38945]: cluster 2024-02-11T16:58:59.738958+0000 mon.smithi037 (mon.0) 2 : cluster [INF] mon.smithi037 is new leader, mons smithi037 in quorum (ranks 0) 2024-02-11T16:59:01.052 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:00 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[38945]: cluster 2024-02-11T16:58:59.739373+0000 mon.smithi037 (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {smithi037=[v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0]} 2024-02-11T16:59:01.052 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:00 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[38945]: cluster 2024-02-11T16:58:59.742399+0000 mon.smithi037 (mon.0) 4 : cluster [DBG] fsmap 2024-02-11T16:59:01.052 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:00 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[38945]: cluster 2024-02-11T16:58:59.745208+0000 mon.smithi037 (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-02-11T16:59:01.052 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:00 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[38945]: cluster 2024-02-11T16:58:59.745724+0000 mon.smithi037 (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2024-02-11T16:59:01.052 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:00 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[38945]: audit 2024-02-11T16:59:00.700357+0000 mon.smithi037 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.37:0/3603562370' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-11T16:59:01.330 INFO:teuthology.orchestra.run.smithi037.stderr:mon is available 2024-02-11T16:59:01.330 INFO:teuthology.orchestra.run.smithi037.stderr:Assimilating anything we can from ceph.conf... 2024-02-11T16:59:01.330 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi037 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpjftqjcr4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4ata0hlm:/etc/ceph/ceph.conf:z -v /var/lib/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715/mon.smithi037:/var/lib/ceph/mon/ceph-smithi037:z docker.io/ceph/ceph:v16.2.4 config assimilate-conf -i /var/lib/ceph/mon/ceph-smithi037/config 2024-02-11T16:59:01.881 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:01.879+0000 7fcef7c77700 1 Processor -- start 2024-02-11T16:59:01.882 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:01.880+0000 7fcef7c77700 1 -- start start 2024-02-11T16:59:01.882 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:01.880+0000 7fcef7c77700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcef00fab80 0x7fcef00fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:01.882 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:01.880+0000 7fcef7c77700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcef00fb490 con 0x7fcef00fab80 2024-02-11T16:59:01.882 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:01.880+0000 7fcef5a13700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcef00fab80 0x7fcef00fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:01.882 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:01.880+0000 7fcef5a13700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcef00fab80 0x7fcef00fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:51480/0 (socket says 172.21.15.37:51480) 2024-02-11T16:59:01.882 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:01.880+0000 7fcef5a13700 1 -- 172.21.15.37:0/3534144811 learned_addr learned my addr 172.21.15.37:0/3534144811 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-02-11T16:59:01.884 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:01.880+0000 7fcef5a13700 1 -- 172.21.15.37:0/3534144811 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcef00fb5d0 con 0x7fcef00fab80 2024-02-11T16:59:01.884 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:01.881+0000 7fcef5a13700 1 --2- 172.21.15.37:0/3534144811 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcef00fab80 0x7fcef00fafa0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7fceec00b5f0 tx=0x7fceec006ee0).ready entity=mon.0 client_cookie=2440ee67b4c861b server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:01.884 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:01.881+0000 7fcef4a11700 1 -- 172.21.15.37:0/3534144811 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fceec00d600 con 0x7fcef00fab80 2024-02-11T16:59:01.884 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:01.881+0000 7fcef4a11700 1 -- 172.21.15.37:0/3534144811 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fceec00d760 con 0x7fcef00fab80 2024-02-11T16:59:01.885 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:01.881+0000 7fcef4a11700 1 -- 172.21.15.37:0/3534144811 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fceec00da30 con 0x7fcef00fab80 2024-02-11T16:59:01.885 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:01.882+0000 7fcef7c77700 1 -- 172.21.15.37:0/3534144811 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcef00fab80 msgr2=0x7fcef00fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:01.885 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:01.882+0000 7fcef7c77700 1 --2- 172.21.15.37:0/3534144811 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcef00fab80 0x7fcef00fafa0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7fceec00b5f0 tx=0x7fceec006ee0).stop 2024-02-11T16:59:01.885 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:01.882+0000 7fcef7c77700 1 -- 172.21.15.37:0/3534144811 shutdown_connections 2024-02-11T16:59:01.885 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:01.882+0000 7fcef7c77700 1 --2- 172.21.15.37:0/3534144811 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcef00fab80 0x7fcef00fafa0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:01.885 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:01.882+0000 7fcef7c77700 1 -- 172.21.15.37:0/3534144811 >> 172.21.15.37:0/3534144811 conn(0x7fcef00f63c0 msgr2=0x7fcef00f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T16:59:01.885 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:01.882+0000 7fcef7c77700 1 -- 172.21.15.37:0/3534144811 shutdown_connections 2024-02-11T16:59:01.885 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:01.882+0000 7fcef7c77700 1 -- 172.21.15.37:0/3534144811 wait complete. 2024-02-11T16:59:01.886 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:01.882+0000 7fcef7c77700 1 Processor -- start 2024-02-11T16:59:01.886 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:01.883+0000 7fcef7c77700 1 -- start start 2024-02-11T16:59:01.886 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:01.883+0000 7fcef7c77700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcef00fab80 0x7fcef010a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:01.886 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:01.883+0000 7fcef7c77700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcef010a740 con 0x7fcef00fab80 2024-02-11T16:59:01.886 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:01.883+0000 7fcef5a13700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcef00fab80 0x7fcef010a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:01.886 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:01.883+0000 7fcef5a13700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcef00fab80 0x7fcef010a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:51482/0 (socket says 172.21.15.37:51482) 2024-02-11T16:59:01.887 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:01.883+0000 7fcef5a13700 1 -- 172.21.15.37:0/1143096835 learned_addr learned my addr 172.21.15.37:0/1143096835 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-02-11T16:59:01.887 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:01.883+0000 7fcef5a13700 1 -- 172.21.15.37:0/1143096835 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fceec00b200 con 0x7fcef00fab80 2024-02-11T16:59:01.887 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:01.883+0000 7fcef5a13700 1 --2- 172.21.15.37:0/1143096835 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcef00fab80 0x7fcef010a250 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7fceec006c70 tx=0x7fceec016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:01.887 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:01.884+0000 7fcee27fc700 1 -- 172.21.15.37:0/1143096835 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fceec011b40 con 0x7fcef00fab80 2024-02-11T16:59:01.887 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:01.884+0000 7fcee27fc700 1 -- 172.21.15.37:0/1143096835 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fceec011ca0 con 0x7fcef00fab80 2024-02-11T16:59:01.887 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:01.884+0000 7fcee27fc700 1 -- 172.21.15.37:0/1143096835 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fceec01d430 con 0x7fcef00fab80 2024-02-11T16:59:01.887 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:01.884+0000 7fcef7c77700 1 -- 172.21.15.37:0/1143096835 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcef010a940 con 0x7fcef00fab80 2024-02-11T16:59:01.887 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:01.884+0000 7fcef7c77700 1 -- 172.21.15.37:0/1143096835 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcef010ad50 con 0x7fcef00fab80 2024-02-11T16:59:01.887 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:01.884+0000 7fcee27fc700 1 -- 172.21.15.37:0/1143096835 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fceec00f070 con 0x7fcef00fab80 2024-02-11T16:59:01.887 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:01.884+0000 7fcee27fc700 1 -- 172.21.15.37:0/1143096835 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fceec027460 con 0x7fcef00fab80 2024-02-11T16:59:01.887 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:01.885+0000 7fcef7c77700 1 -- 172.21.15.37:0/1143096835 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcef0045ad0 con 0x7fcef00fab80 2024-02-11T16:59:01.888 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:01.887+0000 7fcee27fc700 1 -- 172.21.15.37:0/1143096835 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fceec016070 con 0x7fcef00fab80 2024-02-11T16:59:01.955 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:01.954+0000 7fcef7c77700 1 -- 172.21.15.37:0/1143096835 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fcef00fbec0 con 0x7fcef00fab80 2024-02-11T16:59:01.959 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:01.957+0000 7fcee27fc700 1 -- 172.21.15.37:0/1143096835 <== mon.0 v2:172.21.15.37:3300/0 7 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fceec01d620 con 0x7fcef00fab80 2024-02-11T16:59:01.959 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:01.958+0000 7fcee27fc700 1 -- 172.21.15.37:0/1143096835 <== mon.0 v2:172.21.15.37:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+464 (secure 0 0 0) 0x7fceec0308c0 con 0x7fcef00fab80 2024-02-11T16:59:01.959 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:01.959 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: [global] 2024-02-11T16:59:01.959 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: fsid = c65cd888-c8fe-11ee-95b9-87774f69a715 2024-02-11T16:59:01.959 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.37:3300,v1:172.21.15.37:6789] 2024-02-11T16:59:01.959 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-02-11T16:59:01.962 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-02-11T16:59:01.962 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-02-11T16:59:01.962 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-02-11T16:59:01.962 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:01.962 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: [mgr] 2024-02-11T16:59:01.962 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-02-11T16:59:01.962 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:01.962 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: [osd] 2024-02-11T16:59:01.962 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: mon_osd_backfillfull_ratio = 0.85 2024-02-11T16:59:01.962 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: mon_osd_full_ratio = 0.9 2024-02-11T16:59:01.962 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: mon_osd_nearfull_ratio = 0.8 2024-02-11T16:59:01.962 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-02-11T16:59:01.962 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: osd_memory_target_autotune = true 2024-02-11T16:59:01.963 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-02-11T16:59:01.963 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:01.958+0000 7fcef7c77700 1 -- 172.21.15.37:0/1143096835 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcef00fab80 msgr2=0x7fcef010a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:01.963 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:01.958+0000 7fcef7c77700 1 --2- 172.21.15.37:0/1143096835 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcef00fab80 0x7fcef010a250 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7fceec006c70 tx=0x7fceec016000).stop 2024-02-11T16:59:01.963 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:01.958+0000 7fcef7c77700 1 -- 172.21.15.37:0/1143096835 shutdown_connections 2024-02-11T16:59:01.963 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:01.958+0000 7fcef7c77700 1 --2- 172.21.15.37:0/1143096835 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcef00fab80 0x7fcef010a250 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:01.963 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:01.959+0000 7fcef7c77700 1 -- 172.21.15.37:0/1143096835 >> 172.21.15.37:0/1143096835 conn(0x7fcef00f63c0 msgr2=0x7fcef006ffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T16:59:01.963 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:01.959+0000 7fcef7c77700 1 -- 172.21.15.37:0/1143096835 shutdown_connections 2024-02-11T16:59:01.963 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:01.959+0000 7fcef7c77700 1 -- 172.21.15.37:0/1143096835 wait complete. 2024-02-11T16:59:02.337 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:02 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[38945]: audit 2024-02-11T16:59:01.955416+0000 mon.smithi037 (mon.0) 8 : audit [INF] from='client.? 172.21.15.37:0/1143096835' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-11T16:59:02.338 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:02 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[38945]: audit 2024-02-11T16:59:01.958517+0000 mon.smithi037 (mon.0) 9 : audit [INF] from='client.? 172.21.15.37:0/1143096835' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-02-11T16:59:02.594 INFO:teuthology.orchestra.run.smithi037.stderr:Generating new minimal ceph.conf... 2024-02-11T16:59:02.594 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi037 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpjftqjcr4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4ata0hlm:/etc/ceph/ceph.conf:z -v /var/lib/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715/mon.smithi037:/var/lib/ceph/mon/ceph-smithi037:z docker.io/ceph/ceph:v16.2.4 config generate-minimal-conf -o /var/lib/ceph/mon/ceph-smithi037/config 2024-02-11T16:59:03.066 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:03.062+0000 7f5afff77700 1 Processor -- start 2024-02-11T16:59:03.066 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:03.062+0000 7f5afff77700 1 -- start start 2024-02-11T16:59:03.066 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:03.062+0000 7f5afff77700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5af80fbcb0 0x7f5af80fc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:03.066 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:03.062+0000 7f5afff77700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af80fc5c0 con 0x7f5af80fbcb0 2024-02-11T16:59:03.066 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:03.063+0000 7f5afdd13700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5af80fbcb0 0x7f5af80fc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:03.066 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:03.063+0000 7f5afdd13700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5af80fbcb0 0x7f5af80fc0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:51496/0 (socket says 172.21.15.37:51496) 2024-02-11T16:59:03.066 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:03.063+0000 7f5afdd13700 1 -- 172.21.15.37:0/1616364412 learned_addr learned my addr 172.21.15.37:0/1616364412 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-02-11T16:59:03.066 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:03.063+0000 7f5afdd13700 1 -- 172.21.15.37:0/1616364412 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5af80fc700 con 0x7f5af80fbcb0 2024-02-11T16:59:03.066 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:03.063+0000 7f5afdd13700 1 --2- 172.21.15.37:0/1616364412 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5af80fbcb0 0x7f5af80fc0d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f5ae800b5f0 tx=0x7f5ae8006ee0).ready entity=mon.0 client_cookie=c3a18d2fb7f57763 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:03.066 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:03.063+0000 7f5afcd11700 1 -- 172.21.15.37:0/1616364412 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ae800d600 con 0x7f5af80fbcb0 2024-02-11T16:59:03.067 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:03.063+0000 7f5afcd11700 1 -- 172.21.15.37:0/1616364412 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f5ae800dbd0 con 0x7f5af80fbcb0 2024-02-11T16:59:03.067 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:03.063+0000 7f5afcd11700 1 -- 172.21.15.37:0/1616364412 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ae8011c00 con 0x7f5af80fbcb0 2024-02-11T16:59:03.067 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:03.064+0000 7f5afff77700 1 -- 172.21.15.37:0/1616364412 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5af80fbcb0 msgr2=0x7f5af80fc0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:03.067 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:03.064+0000 7f5afff77700 1 --2- 172.21.15.37:0/1616364412 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5af80fbcb0 0x7f5af80fc0d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f5ae800b5f0 tx=0x7f5ae8006ee0).stop 2024-02-11T16:59:03.067 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:03.064+0000 7f5afff77700 1 -- 172.21.15.37:0/1616364412 shutdown_connections 2024-02-11T16:59:03.067 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:03.064+0000 7f5afff77700 1 --2- 172.21.15.37:0/1616364412 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5af80fbcb0 0x7f5af80fc0d0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:03.067 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:03.064+0000 7f5afff77700 1 -- 172.21.15.37:0/1616364412 >> 172.21.15.37:0/1616364412 conn(0x7f5af80f74b0 msgr2=0x7f5af80f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T16:59:03.067 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:03.064+0000 7f5afff77700 1 -- 172.21.15.37:0/1616364412 shutdown_connections 2024-02-11T16:59:03.067 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:03.064+0000 7f5afff77700 1 -- 172.21.15.37:0/1616364412 wait complete. 2024-02-11T16:59:03.067 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:03.065+0000 7f5afff77700 1 Processor -- start 2024-02-11T16:59:03.069 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:03.065+0000 7f5afff77700 1 -- start start 2024-02-11T16:59:03.069 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:03.065+0000 7f5afff77700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5af80fbcb0 0x7f5af810aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:03.069 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:03.065+0000 7f5afff77700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af810af50 con 0x7f5af80fbcb0 2024-02-11T16:59:03.069 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:03.065+0000 7f5afdd13700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5af80fbcb0 0x7f5af810aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:03.069 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:03.065+0000 7f5afdd13700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5af80fbcb0 0x7f5af810aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:51510/0 (socket says 172.21.15.37:51510) 2024-02-11T16:59:03.069 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:03.065+0000 7f5afdd13700 1 -- 172.21.15.37:0/3726680689 learned_addr learned my addr 172.21.15.37:0/3726680689 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-02-11T16:59:03.069 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:03.066+0000 7f5afdd13700 1 -- 172.21.15.37:0/3726680689 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ae800b200 con 0x7f5af80fbcb0 2024-02-11T16:59:03.069 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:03.066+0000 7f5afdd13700 1 --2- 172.21.15.37:0/3726680689 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5af80fbcb0 0x7f5af810aa60 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f5ae800e040 tx=0x7f5ae8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:03.070 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:03.066+0000 7f5ae67fc700 1 -- 172.21.15.37:0/3726680689 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ae801b410 con 0x7f5af80fbcb0 2024-02-11T16:59:03.070 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:03.066+0000 7f5ae67fc700 1 -- 172.21.15.37:0/3726680689 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f5ae801b9e0 con 0x7f5af80fbcb0 2024-02-11T16:59:03.070 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:03.066+0000 7f5ae67fc700 1 -- 172.21.15.37:0/3726680689 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ae801a820 con 0x7f5af80fbcb0 2024-02-11T16:59:03.070 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:03.066+0000 7f5afff77700 1 -- 172.21.15.37:0/3726680689 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5af810b150 con 0x7f5af80fbcb0 2024-02-11T16:59:03.070 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:03.066+0000 7f5afff77700 1 -- 172.21.15.37:0/3726680689 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5af810b560 con 0x7f5af80fbcb0 2024-02-11T16:59:03.070 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:03.066+0000 7f5ae67fc700 1 -- 172.21.15.37:0/3726680689 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f5ae800f070 con 0x7f5af80fbcb0 2024-02-11T16:59:03.070 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:03.066+0000 7f5ae67fc700 1 -- 172.21.15.37:0/3726680689 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f5ae8025b30 con 0x7f5af80fbcb0 2024-02-11T16:59:03.070 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:03.068+0000 7f5afff77700 1 -- 172.21.15.37:0/3726680689 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5af8045ad0 con 0x7f5af80fbcb0 2024-02-11T16:59:03.071 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:03.070+0000 7f5ae67fc700 1 -- 172.21.15.37:0/3726680689 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f5ae8016070 con 0x7f5af80fbcb0 2024-02-11T16:59:03.139 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:03.137+0000 7f5afff77700 1 -- 172.21.15.37:0/3726680689 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f5af80fd9c0 con 0x7f5af80fbcb0 2024-02-11T16:59:03.139 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:03.138+0000 7f5ae67fc700 1 -- 172.21.15.37:0/3726680689 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f5ae801ad70 con 0x7f5af80fbcb0 2024-02-11T16:59:03.140 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:03.139+0000 7f5afff77700 1 -- 172.21.15.37:0/3726680689 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5af80fbcb0 msgr2=0x7f5af810aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:03.140 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:03.139+0000 7f5afff77700 1 --2- 172.21.15.37:0/3726680689 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5af80fbcb0 0x7f5af810aa60 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f5ae800e040 tx=0x7f5ae8006ee0).stop 2024-02-11T16:59:03.140 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:03.139+0000 7f5afff77700 1 -- 172.21.15.37:0/3726680689 shutdown_connections 2024-02-11T16:59:03.140 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:03.139+0000 7f5afff77700 1 --2- 172.21.15.37:0/3726680689 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5af80fbcb0 0x7f5af810aa60 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:03.140 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:03.139+0000 7f5afff77700 1 -- 172.21.15.37:0/3726680689 >> 172.21.15.37:0/3726680689 conn(0x7f5af80f74b0 msgr2=0x7f5af80f8f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T16:59:03.140 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:03.139+0000 7f5afff77700 1 -- 172.21.15.37:0/3726680689 shutdown_connections 2024-02-11T16:59:03.140 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:03.139+0000 7f5afff77700 1 -- 172.21.15.37:0/3726680689 wait complete. 2024-02-11T16:59:03.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:03 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[38945]: audit 2024-02-11T16:59:03.138662+0000 mon.smithi037 (mon.0) 10 : audit [DBG] from='client.? 172.21.15.37:0/3726680689' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-11T16:59:03.767 INFO:teuthology.orchestra.run.smithi037.stderr:Restarting the monitor... 2024-02-11T16:59:03.768 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: systemctl restart ceph-c65cd888-c8fe-11ee-95b9-87774f69a715@mon.smithi037 2024-02-11T16:59:04.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:03 smithi037 systemd[1]: Stopping Ceph mon.smithi037 for c65cd888-c8fe-11ee-95b9-87774f69a715... 2024-02-11T16:59:04.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:03 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[38945]: debug 2024-02-11T16:59:03.898+0000 7f85d9b5c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi037 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-02-11T16:59:04.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:03 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[38945]: debug 2024-02-11T16:59:03.898+0000 7f85d9b5c700 -1 mon.smithi037@0(leader) e1 *** Got Signal Terminated *** 2024-02-11T16:59:04.577 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:04 smithi037 podman[39269]: ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037 2024-02-11T16:59:04.911 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:04 smithi037 systemd[1]: ceph-c65cd888-c8fe-11ee-95b9-87774f69a715@mon.smithi037.service: Succeeded. 2024-02-11T16:59:04.911 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:04 smithi037 systemd[1]: Stopped Ceph mon.smithi037 for c65cd888-c8fe-11ee-95b9-87774f69a715. 2024-02-11T16:59:04.911 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:04 smithi037 systemd[1]: Starting Ceph mon.smithi037 for c65cd888-c8fe-11ee-95b9-87774f69a715... 2024-02-11T16:59:05.067 INFO:teuthology.orchestra.run.smithi037.stderr:Setting mon public_network to 172.21.0.0/20 2024-02-11T16:59:05.067 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi037 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpjftqjcr4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4ata0hlm:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mon public_network 172.21.0.0/20 2024-02-11T16:59:05.308 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.005+0000 7f907653a700 0 set uid:gid to 167:167 (ceph:ceph) 2024-02-11T16:59:05.309 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.005+0000 7f907653a700 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-mon, pid 7 2024-02-11T16:59:05.309 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.005+0000 7f907653a700 0 pidfile_write: ignore empty --pid-file 2024-02-11T16:59:05.309 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.009+0000 7f907653a700 0 load: jerasure load: lrc load: isa 2024-02-11T16:59:05.309 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: RocksDB version: 6.8.1 2024-02-11T16:59:05.309 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: 2024-02-11T16:59:05.309 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-11T16:59:05.309 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Compile date May 13 2021 2024-02-11T16:59:05.309 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: DB SUMMARY 2024-02-11T16:59:05.309 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: 2024-02-11T16:59:05.309 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: CURRENT file: CURRENT 2024-02-11T16:59:05.309 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: 2024-02-11T16:59:05.309 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: IDENTITY file: IDENTITY 2024-02-11T16:59:05.309 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: 2024-02-11T16:59:05.310 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2024-02-11T16:59:05.310 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: 2024-02-11T16:59:05.310 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi037/store.db dir, Total Num: 1, files: 000004.sst 2024-02-11T16:59:05.310 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: 2024-02-11T16:59:05.310 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi037/store.db: 000006.log size: 98185 ; 2024-02-11T16:59:05.310 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: 2024-02-11T16:59:05.310 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.error_if_exists: 0 2024-02-11T16:59:05.310 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.create_if_missing: 0 2024-02-11T16:59:05.310 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.paranoid_checks: 1 2024-02-11T16:59:05.310 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.env: 0x5618e9c7b1c0 2024-02-11T16:59:05.310 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.fs: Posix File System 2024-02-11T16:59:05.310 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.info_log: 0x5618eb2c9960 2024-02-11T16:59:05.310 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.max_file_opening_threads: 16 2024-02-11T16:59:05.310 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.statistics: (nil) 2024-02-11T16:59:05.311 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.use_fsync: 0 2024-02-11T16:59:05.311 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.max_log_file_size: 0 2024-02-11T16:59:05.311 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-11T16:59:05.311 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.log_file_time_to_roll: 0 2024-02-11T16:59:05.311 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.keep_log_file_num: 1000 2024-02-11T16:59:05.311 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.recycle_log_file_num: 0 2024-02-11T16:59:05.311 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.allow_fallocate: 1 2024-02-11T16:59:05.311 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.allow_mmap_reads: 0 2024-02-11T16:59:05.311 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.allow_mmap_writes: 0 2024-02-11T16:59:05.311 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.use_direct_reads: 0 2024-02-11T16:59:05.311 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-11T16:59:05.311 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.create_missing_column_families: 0 2024-02-11T16:59:05.311 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.db_log_dir: 2024-02-11T16:59:05.311 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi037/store.db 2024-02-11T16:59:05.311 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.table_cache_numshardbits: 6 2024-02-11T16:59:05.311 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.max_subcompactions: 1 2024-02-11T16:59:05.312 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.max_background_flushes: -1 2024-02-11T16:59:05.312 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-02-11T16:59:05.312 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-02-11T16:59:05.312 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-11T16:59:05.312 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-11T16:59:05.312 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-02-11T16:59:05.312 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.advise_random_on_open: 1 2024-02-11T16:59:05.312 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.db_write_buffer_size: 0 2024-02-11T16:59:05.312 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.write_buffer_manager: 0x5618ebf3e030 2024-02-11T16:59:05.312 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-11T16:59:05.312 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-11T16:59:05.312 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-11T16:59:05.312 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.use_adaptive_mutex: 0 2024-02-11T16:59:05.313 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.rate_limiter: (nil) 2024-02-11T16:59:05.313 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-11T16:59:05.313 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.wal_recovery_mode: 2 2024-02-11T16:59:05.313 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.enable_thread_tracking: 0 2024-02-11T16:59:05.313 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.enable_pipelined_write: 0 2024-02-11T16:59:05.313 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.unordered_write: 0 2024-02-11T16:59:05.313 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-11T16:59:05.313 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-11T16:59:05.313 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-11T16:59:05.313 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-11T16:59:05.313 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.row_cache: None 2024-02-11T16:59:05.313 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.wal_filter: None 2024-02-11T16:59:05.313 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-11T16:59:05.313 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.allow_ingest_behind: 0 2024-02-11T16:59:05.314 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.preserve_deletes: 0 2024-02-11T16:59:05.314 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.two_write_queues: 0 2024-02-11T16:59:05.314 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.manual_wal_flush: 0 2024-02-11T16:59:05.314 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.atomic_flush: 0 2024-02-11T16:59:05.314 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-11T16:59:05.314 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.persist_stats_to_disk: 0 2024-02-11T16:59:05.314 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-02-11T16:59:05.314 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.log_readahead_size: 0 2024-02-11T16:59:05.314 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-02-11T16:59:05.314 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.max_background_jobs: 2 2024-02-11T16:59:05.314 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.max_background_compactions: -1 2024-02-11T16:59:05.314 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-11T16:59:05.314 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-11T16:59:05.314 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.delayed_write_rate : 16777216 2024-02-11T16:59:05.315 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.max_total_wal_size: 0 2024-02-11T16:59:05.315 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-11T16:59:05.315 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.stats_dump_period_sec: 600 2024-02-11T16:59:05.315 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.stats_persist_period_sec: 600 2024-02-11T16:59:05.315 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-11T16:59:05.315 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.max_open_files: -1 2024-02-11T16:59:05.315 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.bytes_per_sync: 0 2024-02-11T16:59:05.315 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-02-11T16:59:05.315 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-02-11T16:59:05.315 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Options.compaction_readahead_size: 0 2024-02-11T16:59:05.315 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Compression algorithms supported: 2024-02-11T16:59:05.315 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-11T16:59:05.315 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: kZSTD supported: 0 2024-02-11T16:59:05.315 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: kXpressCompression supported: 0 2024-02-11T16:59:05.316 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: kLZ4HCCompression supported: 1 2024-02-11T16:59:05.316 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: kLZ4Compression supported: 1 2024-02-11T16:59:05.316 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: kBZip2Compression supported: 0 2024-02-11T16:59:05.316 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: kZlibCompression supported: 1 2024-02-11T16:59:05.316 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: kSnappyCompression supported: 1 2024-02-11T16:59:05.316 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.010+0000 7f907653a700 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-02-11T16:59:05.316 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi037/store.db/MANIFEST-000005 2024-02-11T16:59:05.316 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: 2024-02-11T16:59:05.316 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-02-11T16:59:05.316 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: 2024-02-11T16:59:05.316 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-11T16:59:05.316 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.merge_operator: 2024-02-11T16:59:05.316 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.compaction_filter: None 2024-02-11T16:59:05.316 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.compaction_filter_factory: None 2024-02-11T16:59:05.316 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-02-11T16:59:05.317 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.table_factory: BlockBasedTable 2024-02-11T16:59:05.317 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5618eb1e1cb0) 2024-02-11T16:59:05.317 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cache_index_and_filter_blocks: 1 2024-02-11T16:59:05.317 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-11T16:59:05.317 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-11T16:59:05.317 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: pin_top_level_index_and_filter: 1 2024-02-11T16:59:05.317 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: index_type: 0 2024-02-11T16:59:05.317 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: data_block_index_type: 0 2024-02-11T16:59:05.317 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: index_shortening: 1 2024-02-11T16:59:05.317 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: data_block_hash_table_util_ratio: 0.750000 2024-02-11T16:59:05.317 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: hash_index_allow_collision: 1 2024-02-11T16:59:05.317 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: checksum: 1 2024-02-11T16:59:05.317 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: no_block_cache: 0 2024-02-11T16:59:05.317 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: block_cache: 0x5618eb216f10 2024-02-11T16:59:05.317 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: block_cache_name: BinnedLRUCache 2024-02-11T16:59:05.318 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: block_cache_options: 2024-02-11T16:59:05.318 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: capacity : 536870912 2024-02-11T16:59:05.318 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: num_shard_bits : 4 2024-02-11T16:59:05.318 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: strict_capacity_limit : 0 2024-02-11T16:59:05.318 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: high_pri_pool_ratio: 0.000 2024-02-11T16:59:05.318 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: block_cache_compressed: (nil) 2024-02-11T16:59:05.318 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: persistent_cache: (nil) 2024-02-11T16:59:05.318 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: block_size: 4096 2024-02-11T16:59:05.318 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: block_size_deviation: 10 2024-02-11T16:59:05.318 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: block_restart_interval: 16 2024-02-11T16:59:05.318 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: index_block_restart_interval: 1 2024-02-11T16:59:05.318 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: metadata_block_size: 4096 2024-02-11T16:59:05.318 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: partition_filters: 0 2024-02-11T16:59:05.318 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: use_delta_encoding: 1 2024-02-11T16:59:05.319 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-11T16:59:05.319 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: whole_key_filtering: 1 2024-02-11T16:59:05.319 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: verify_compression: 0 2024-02-11T16:59:05.319 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: read_amp_bytes_per_bit: 0 2024-02-11T16:59:05.319 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: format_version: 2 2024-02-11T16:59:05.319 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: enable_index_compression: 1 2024-02-11T16:59:05.319 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: block_align: 0 2024-02-11T16:59:05.319 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: 2024-02-11T16:59:05.319 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.write_buffer_size: 33554432 2024-02-11T16:59:05.319 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.max_write_buffer_number: 2 2024-02-11T16:59:05.319 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.compression: NoCompression 2024-02-11T16:59:05.319 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.bottommost_compression: Disabled 2024-02-11T16:59:05.319 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.prefix_extractor: nullptr 2024-02-11T16:59:05.319 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-11T16:59:05.320 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.num_levels: 7 2024-02-11T16:59:05.320 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-11T16:59:05.320 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-11T16:59:05.320 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-11T16:59:05.320 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-11T16:59:05.321 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-11T16:59:05.321 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-11T16:59:05.321 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-11T16:59:05.321 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-11T16:59:05.321 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-11T16:59:05.322 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.compression_opts.window_bits: -14 2024-02-11T16:59:05.322 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.compression_opts.level: 32767 2024-02-11T16:59:05.322 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.compression_opts.strategy: 0 2024-02-11T16:59:05.322 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-11T16:59:05.322 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-11T16:59:05.322 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.compression_opts.enabled: false 2024-02-11T16:59:05.322 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-11T16:59:05.322 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-11T16:59:05.322 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-11T16:59:05.322 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.target_file_size_base: 67108864 2024-02-11T16:59:05.322 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.target_file_size_multiplier: 1 2024-02-11T16:59:05.322 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-11T16:59:05.322 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-11T16:59:05.322 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-11T16:59:05.322 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-11T16:59:05.323 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-11T16:59:05.323 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-11T16:59:05.323 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-11T16:59:05.323 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-11T16:59:05.323 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-11T16:59:05.323 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-11T16:59:05.323 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-11T16:59:05.323 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-11T16:59:05.323 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.arena_block_size: 4194304 2024-02-11T16:59:05.323 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-11T16:59:05.323 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-11T16:59:05.323 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-11T16:59:05.323 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.disable_auto_compactions: 0 2024-02-11T16:59:05.323 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-11T16:59:05.323 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-11T16:59:05.324 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-11T16:59:05.324 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-11T16:59:05.324 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-11T16:59:05.324 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-11T16:59:05.324 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-11T16:59:05.324 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-11T16:59:05.324 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-11T16:59:05.324 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-11T16:59:05.324 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.table_properties_collectors: 2024-02-11T16:59:05.324 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.inplace_update_support: 0 2024-02-11T16:59:05.324 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-02-11T16:59:05.324 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-11T16:59:05.324 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-11T16:59:05.325 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.memtable_huge_page_size: 0 2024-02-11T16:59:05.325 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.bloom_locality: 0 2024-02-11T16:59:05.325 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.max_successive_merges: 0 2024-02-11T16:59:05.325 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-02-11T16:59:05.325 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.paranoid_file_checks: 0 2024-02-11T16:59:05.325 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.force_consistency_checks: 0 2024-02-11T16:59:05.325 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.report_bg_io_stats: 0 2024-02-11T16:59:05.325 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.ttl: 2592000 2024-02-11T16:59:05.325 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.011+0000 7f907653a700 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-02-11T16:59:05.325 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.012+0000 7f907653a700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi037/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-11T16:59:05.325 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: 2024-02-11T16:59:05.325 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.012+0000 7f907653a700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2024-02-11T16:59:05.325 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: 2024-02-11T16:59:05.325 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.012+0000 7f907653a700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707670745012772, "job": 1, "event": "recovery_started", "log_files": [6]} 2024-02-11T16:59:05.326 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.012+0000 7f907653a700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2024-02-11T16:59:05.326 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.012+0000 7f907653a700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-02-11T16:59:05.326 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.013+0000 7f907653a700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707670745013725, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 93835, "table_properties": {"data_size": 91923, "index_size": 389, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 709, "raw_key_size": 12432, "raw_average_key_size": 49, "raw_value_size": 85006, "raw_average_value_size": 337, "num_data_blocks": 12, "num_entries": 252, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1707670745, "oldest_key_time": 3, "file_creation_time": 0}} 2024-02-11T16:59:05.326 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.013+0000 7f907653a700 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2024-02-11T16:59:05.326 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: 2024-02-11T16:59:05.326 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.013+0000 7f907653a700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707670745014156, "job": 1, "event": "recovery_finished"} 2024-02-11T16:59:05.326 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.014+0000 7f907653a700 4 rocksdb: DB pointer 0x5618eb2db800 2024-02-11T16:59:05.326 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.014+0000 7f905d352700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-02-11T16:59:05.326 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.014+0000 7f905d352700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-02-11T16:59:05.326 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: ** DB Stats ** 2024-02-11T16:59:05.326 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-11T16:59:05.326 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-11T16:59:05.326 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-11T16:59:05.326 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-11T16:59:05.327 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-11T16:59:05.327 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-11T16:59:05.327 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-11T16:59:05.327 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: 2024-02-11T16:59:05.327 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: ** Compaction Stats [default] ** 2024-02-11T16:59:05.327 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-11T16:59:05.327 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-11T16:59:05.327 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: L0 2/0 93.33 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 144.8 0.00 0.00 1 0.001 0 0 2024-02-11T16:59:05.327 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: Sum 2/0 93.33 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 144.8 0.00 0.00 1 0.001 0 0 2024-02-11T16:59:05.327 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 144.8 0.00 0.00 1 0.001 0 0 2024-02-11T16:59:05.327 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: 2024-02-11T16:59:05.327 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: ** Compaction Stats [default] ** 2024-02-11T16:59:05.327 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-11T16:59:05.327 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-11T16:59:05.327 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 144.8 0.00 0.00 1 0.001 0 0 2024-02-11T16:59:05.328 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-11T16:59:05.328 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-11T16:59:05.328 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-11T16:59:05.328 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: AddFile(Total Files): cumulative 0, interval 0 2024-02-11T16:59:05.328 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-11T16:59:05.328 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: AddFile(Keys): cumulative 0, interval 0 2024-02-11T16:59:05.328 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: Cumulative compaction: 0.00 GB write, 27.71 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-11T16:59:05.328 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: Interval compaction: 0.00 GB write, 27.71 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-11T16:59:05.328 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-11T16:59:05.328 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: 2024-02-11T16:59:05.328 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: ** File Read Latency Histogram By Level [default] ** 2024-02-11T16:59:05.328 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: 2024-02-11T16:59:05.328 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: ** Compaction Stats [default] ** 2024-02-11T16:59:05.328 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-11T16:59:05.329 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-11T16:59:05.329 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: L0 2/0 93.33 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 144.8 0.00 0.00 1 0.001 0 0 2024-02-11T16:59:05.329 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: Sum 2/0 93.33 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 144.8 0.00 0.00 1 0.001 0 0 2024-02-11T16:59:05.329 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-11T16:59:05.329 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: 2024-02-11T16:59:05.329 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: ** Compaction Stats [default] ** 2024-02-11T16:59:05.329 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-11T16:59:05.329 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-11T16:59:05.329 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 144.8 0.00 0.00 1 0.001 0 0 2024-02-11T16:59:05.329 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-11T16:59:05.329 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-11T16:59:05.329 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-11T16:59:05.329 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: AddFile(Total Files): cumulative 0, interval 0 2024-02-11T16:59:05.329 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-11T16:59:05.329 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: AddFile(Keys): cumulative 0, interval 0 2024-02-11T16:59:05.330 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: Cumulative compaction: 0.00 GB write, 27.46 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-11T16:59:05.330 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-11T16:59:05.330 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-11T16:59:05.330 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: 2024-02-11T16:59:05.330 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: ** File Read Latency Histogram By Level [default] ** 2024-02-11T16:59:05.330 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: 2024-02-11T16:59:05.330 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.015+0000 7f907653a700 0 starting mon.smithi037 rank 0 at public addrs [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] at bind addrs [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi037 fsid c65cd888-c8fe-11ee-95b9-87774f69a715 2024-02-11T16:59:05.330 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.016+0000 7f907653a700 1 mon.smithi037@-1(???) e1 preinit fsid c65cd888-c8fe-11ee-95b9-87774f69a715 2024-02-11T16:59:05.330 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.016+0000 7f907653a700 0 mon.smithi037@-1(???).mds e1 new map 2024-02-11T16:59:05.330 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.016+0000 7f907653a700 0 mon.smithi037@-1(???).mds e1 print_map 2024-02-11T16:59:05.330 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: e1 2024-02-11T16:59:05.330 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: enable_multiple, ever_enabled_multiple: 1,1 2024-02-11T16:59:05.330 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-11T16:59:05.330 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: legacy client fscid: -1 2024-02-11T16:59:05.331 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: 2024-02-11T16:59:05.331 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: No filesystems configured 2024-02-11T16:59:05.331 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: 2024-02-11T16:59:05.331 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.016+0000 7f907653a700 0 mon.smithi037@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-11T16:59:05.331 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.016+0000 7f907653a700 0 mon.smithi037@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-11T16:59:05.331 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.016+0000 7f907653a700 0 mon.smithi037@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-11T16:59:05.331 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.016+0000 7f907653a700 0 mon.smithi037@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-11T16:59:05.331 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:03.138662+0000 mon.smithi037 (mon.0) 10 : audit [DBG] from='client.? 172.21.15.37:0/3726680689' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-11T16:59:05.331 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.016+0000 7f907653a700 1 mon.smithi037@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-02-11T16:59:05.331 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.017+0000 7f907653a700 20 expand_channel_meta expand map: {default=false} 2024-02-11T16:59:05.331 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.017+0000 7f907653a700 20 expand_channel_meta from 'false' to 'false' 2024-02-11T16:59:05.331 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.017+0000 7f907653a700 20 expand_channel_meta expanded map: {default=false} 2024-02-11T16:59:05.331 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.017+0000 7f907653a700 20 expand_channel_meta expand map: {default=info} 2024-02-11T16:59:05.331 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.017+0000 7f907653a700 20 expand_channel_meta from 'info' to 'info' 2024-02-11T16:59:05.331 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.017+0000 7f907653a700 20 expand_channel_meta expanded map: {default=info} 2024-02-11T16:59:05.332 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.017+0000 7f907653a700 20 expand_channel_meta expand map: {default=daemon} 2024-02-11T16:59:05.332 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.017+0000 7f907653a700 20 expand_channel_meta from 'daemon' to 'daemon' 2024-02-11T16:59:05.332 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.017+0000 7f907653a700 20 expand_channel_meta expanded map: {default=daemon} 2024-02-11T16:59:05.332 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.017+0000 7f907653a700 20 expand_channel_meta expand map: {default=debug} 2024-02-11T16:59:05.332 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.017+0000 7f907653a700 20 expand_channel_meta from 'debug' to 'debug' 2024-02-11T16:59:05.332 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T16:59:05.017+0000 7f907653a700 20 expand_channel_meta expanded map: {default=debug} 2024-02-11T16:59:05.332 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 bash[39341]: b77d24f889c10ba2735d39d1708192b88a4a91fcfe86e51de86e135d7a7163e6 2024-02-11T16:59:05.332 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 systemd[1]: Started Ceph mon.smithi037 for c65cd888-c8fe-11ee-95b9-87774f69a715. 2024-02-11T16:59:05.332 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T16:59:05.033461+0000 mon.smithi037 (mon.0) 1 : cluster [INF] mon.smithi037 is new leader, mons smithi037 in quorum (ranks 0) 2024-02-11T16:59:05.332 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T16:59:05.033512+0000 mon.smithi037 (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {smithi037=[v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0]} 2024-02-11T16:59:05.332 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T16:59:05.033543+0000 mon.smithi037 (mon.0) 3 : cluster [DBG] fsmap 2024-02-11T16:59:05.332 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T16:59:05.033581+0000 mon.smithi037 (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-02-11T16:59:05.332 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T16:59:05.033975+0000 mon.smithi037 (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2024-02-11T16:59:05.615 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:05.612+0000 7f5e88caf700 1 Processor -- start 2024-02-11T16:59:05.615 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:05.612+0000 7f5e88caf700 1 -- start start 2024-02-11T16:59:05.615 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:05.612+0000 7f5e88caf700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e840703d0 0x7f5e840fe7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:05.615 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:05.612+0000 7f5e88caf700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e840feca0 con 0x7f5e840703d0 2024-02-11T16:59:05.615 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:05.612+0000 7f5e837fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e840703d0 0x7f5e840fe7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:05.615 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:05.612+0000 7f5e837fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e840703d0 0x7f5e840fe7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:51522/0 (socket says 172.21.15.37:51522) 2024-02-11T16:59:05.615 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:05.612+0000 7f5e837fe700 1 -- 172.21.15.37:0/1864902551 learned_addr learned my addr 172.21.15.37:0/1864902551 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-02-11T16:59:05.615 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:05.613+0000 7f5e837fe700 1 -- 172.21.15.37:0/1864902551 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e840fede0 con 0x7f5e840703d0 2024-02-11T16:59:05.615 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:05.613+0000 7f5e837fe700 1 --2- 172.21.15.37:0/1864902551 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e840703d0 0x7f5e840fe7b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f5e7400b5f0 tx=0x7f5e74006ee0).ready entity=mon.0 client_cookie=9aa9f135ab90c5cd server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:05.615 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:05.613+0000 7f5e827fc700 1 -- 172.21.15.37:0/1864902551 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e7400d600 con 0x7f5e840703d0 2024-02-11T16:59:05.615 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:05.613+0000 7f5e827fc700 1 -- 172.21.15.37:0/1864902551 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f5e7400dbd0 con 0x7f5e840703d0 2024-02-11T16:59:05.616 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:05.613+0000 7f5e827fc700 1 -- 172.21.15.37:0/1864902551 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e74011c00 con 0x7f5e840703d0 2024-02-11T16:59:05.616 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:05.614+0000 7f5e88caf700 1 -- 172.21.15.37:0/1864902551 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e840703d0 msgr2=0x7f5e840fe7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:05.616 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:05.614+0000 7f5e88caf700 1 --2- 172.21.15.37:0/1864902551 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e840703d0 0x7f5e840fe7b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f5e7400b5f0 tx=0x7f5e74006ee0).stop 2024-02-11T16:59:05.618 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:05.614+0000 7f5e88caf700 1 -- 172.21.15.37:0/1864902551 shutdown_connections 2024-02-11T16:59:05.618 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:05.614+0000 7f5e88caf700 1 --2- 172.21.15.37:0/1864902551 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e840703d0 0x7f5e840fe7b0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:05.618 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:05.614+0000 7f5e88caf700 1 -- 172.21.15.37:0/1864902551 >> 172.21.15.37:0/1864902551 conn(0x7f5e840f6210 msgr2=0x7f5e840f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T16:59:05.618 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:05.614+0000 7f5e88caf700 1 -- 172.21.15.37:0/1864902551 shutdown_connections 2024-02-11T16:59:05.618 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:05.614+0000 7f5e88caf700 1 -- 172.21.15.37:0/1864902551 wait complete. 2024-02-11T16:59:05.618 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:05.615+0000 7f5e88caf700 1 Processor -- start 2024-02-11T16:59:05.618 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:05.615+0000 7f5e88caf700 1 -- start start 2024-02-11T16:59:05.618 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:05.615+0000 7f5e88caf700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e840703d0 0x7f5e8410a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:05.618 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:05.615+0000 7f5e88caf700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e8410adc0 con 0x7f5e840703d0 2024-02-11T16:59:05.618 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:05.616+0000 7f5e837fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e840703d0 0x7f5e8410a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:05.619 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:05.616+0000 7f5e837fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e840703d0 0x7f5e8410a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:51528/0 (socket says 172.21.15.37:51528) 2024-02-11T16:59:05.619 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:05.616+0000 7f5e837fe700 1 -- 172.21.15.37:0/3385753972 learned_addr learned my addr 172.21.15.37:0/3385753972 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-02-11T16:59:05.619 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:05.616+0000 7f5e837fe700 1 -- 172.21.15.37:0/3385753972 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e7400b200 con 0x7f5e840703d0 2024-02-11T16:59:05.619 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:05.616+0000 7f5e837fe700 1 --2- 172.21.15.37:0/3385753972 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e840703d0 0x7f5e8410a8d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f5e7400e040 tx=0x7f5e74006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:05.619 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:05.616+0000 7f5e6bfff700 1 -- 172.21.15.37:0/3385753972 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e7400d9f0 con 0x7f5e840703d0 2024-02-11T16:59:05.619 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:05.616+0000 7f5e6bfff700 1 -- 172.21.15.37:0/3385753972 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f5e7401a9f0 con 0x7f5e840703d0 2024-02-11T16:59:05.619 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:05.616+0000 7f5e88caf700 1 -- 172.21.15.37:0/3385753972 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e8410afc0 con 0x7f5e840703d0 2024-02-11T16:59:05.619 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:05.616+0000 7f5e6bfff700 1 -- 172.21.15.37:0/3385753972 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e7401b8c0 con 0x7f5e840703d0 2024-02-11T16:59:05.619 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:05.616+0000 7f5e88caf700 1 -- 172.21.15.37:0/3385753972 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e8410b3d0 con 0x7f5e840703d0 2024-02-11T16:59:05.620 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:05.617+0000 7f5e6bfff700 1 -- 172.21.15.37:0/3385753972 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f5e7400f070 con 0x7f5e840703d0 2024-02-11T16:59:05.620 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:05.617+0000 7f5e6bfff700 1 -- 172.21.15.37:0/3385753972 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f5e74025b30 con 0x7f5e840703d0 2024-02-11T16:59:05.620 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:05.617+0000 7f5e88caf700 1 -- 172.21.15.37:0/3385753972 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e84045ad0 con 0x7f5e840703d0 2024-02-11T16:59:05.621 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:05.620+0000 7f5e6bfff700 1 -- 172.21.15.37:0/3385753972 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f5e74016070 con 0x7f5e840703d0 2024-02-11T16:59:05.687 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:05.685+0000 7f5e88caf700 1 -- 172.21.15.37:0/3385753972 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f5e840ffa00 con 0x7f5e840703d0 2024-02-11T16:59:05.689 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:05.688+0000 7f5e6bfff700 1 -- 172.21.15.37:0/3385753972 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f5e7401bdc0 con 0x7f5e840703d0 2024-02-11T16:59:05.690 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:05.688+0000 7f5e88caf700 1 -- 172.21.15.37:0/3385753972 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e840703d0 msgr2=0x7f5e8410a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:05.690 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:05.688+0000 7f5e88caf700 1 --2- 172.21.15.37:0/3385753972 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e840703d0 0x7f5e8410a8d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f5e7400e040 tx=0x7f5e74006ee0).stop 2024-02-11T16:59:05.690 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:05.688+0000 7f5e88caf700 1 -- 172.21.15.37:0/3385753972 shutdown_connections 2024-02-11T16:59:05.690 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:05.688+0000 7f5e88caf700 1 --2- 172.21.15.37:0/3385753972 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e840703d0 0x7f5e8410a8d0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:05.690 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:05.688+0000 7f5e88caf700 1 -- 172.21.15.37:0/3385753972 >> 172.21.15.37:0/3385753972 conn(0x7f5e840f6210 msgr2=0x7f5e840f7d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T16:59:05.690 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:05.688+0000 7f5e88caf700 1 -- 172.21.15.37:0/3385753972 shutdown_connections 2024-02-11T16:59:05.690 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:05.688+0000 7f5e88caf700 1 -- 172.21.15.37:0/3385753972 wait complete. 2024-02-11T16:59:06.272 INFO:teuthology.orchestra.run.smithi037.stderr:Wrote config to /etc/ceph/ceph.conf 2024-02-11T16:59:06.272 INFO:teuthology.orchestra.run.smithi037.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-02-11T16:59:06.273 INFO:teuthology.orchestra.run.smithi037.stderr:Creating mgr... 2024-02-11T16:59:06.273 INFO:teuthology.orchestra.run.smithi037.stderr:Verifying port 9283 ... 2024-02-11T16:59:06.275 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: systemctl daemon-reload 2024-02-11T16:59:06.381 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: systemctl stop ceph-c65cd888-c8fe-11ee-95b9-87774f69a715@mgr.smithi037.eofnaq 2024-02-11T16:59:06.407 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: systemctl reset-failed ceph-c65cd888-c8fe-11ee-95b9-87774f69a715@mgr.smithi037.eofnaq 2024-02-11T16:59:06.415 INFO:teuthology.orchestra.run.smithi037.stderr:systemctl: Failed to reset failed state of unit ceph-c65cd888-c8fe-11ee-95b9-87774f69a715@mgr.smithi037.eofnaq.service: Unit ceph-c65cd888-c8fe-11ee-95b9-87774f69a715@mgr.smithi037.eofnaq.service not loaded. 2024-02-11T16:59:06.416 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: systemctl enable ceph-c65cd888-c8fe-11ee-95b9-87774f69a715@mgr.smithi037.eofnaq 2024-02-11T16:59:06.420 INFO:teuthology.orchestra.run.smithi037.stderr:systemctl: Created symlink /etc/systemd/system/ceph-c65cd888-c8fe-11ee-95b9-87774f69a715.target.wants/ceph-c65cd888-c8fe-11ee-95b9-87774f69a715@mgr.smithi037.eofnaq.service → /etc/systemd/system/ceph-c65cd888-c8fe-11ee-95b9-87774f69a715@.service. 2024-02-11T16:59:06.507 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: systemctl start ceph-c65cd888-c8fe-11ee-95b9-87774f69a715@mgr.smithi037.eofnaq 2024-02-11T16:59:07.123 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:06 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:05.688622+0000 mon.smithi037 (mon.0) 6 : audit [INF] from='client.? 172.21.15.37:0/3385753972' entity='client.admin' 2024-02-11T16:59:07.127 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: systemctl is-enabled firewalld.service 2024-02-11T16:59:07.136 INFO:teuthology.orchestra.run.smithi037.stderr:systemctl: disabled 2024-02-11T16:59:07.136 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: systemctl is-active firewalld.service 2024-02-11T16:59:07.144 INFO:teuthology.orchestra.run.smithi037.stderr:systemctl: inactive 2024-02-11T16:59:07.145 INFO:teuthology.orchestra.run.smithi037.stderr:firewalld.service is not enabled 2024-02-11T16:59:07.145 INFO:teuthology.orchestra.run.smithi037.stderr:Not possible to enable service . firewalld.service is not available 2024-02-11T16:59:07.145 INFO:teuthology.orchestra.run.smithi037.stderr:Not possible to open ports <[]>. firewalld.service is not available 2024-02-11T16:59:07.145 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: systemctl is-enabled firewalld.service 2024-02-11T16:59:07.153 INFO:teuthology.orchestra.run.smithi037.stderr:systemctl: disabled 2024-02-11T16:59:07.153 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: systemctl is-active firewalld.service 2024-02-11T16:59:07.161 INFO:teuthology.orchestra.run.smithi037.stderr:systemctl: inactive 2024-02-11T16:59:07.162 INFO:teuthology.orchestra.run.smithi037.stderr:firewalld.service is not enabled 2024-02-11T16:59:07.162 INFO:teuthology.orchestra.run.smithi037.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-02-11T16:59:07.162 INFO:teuthology.orchestra.run.smithi037.stderr:Waiting for mgr to start... 2024-02-11T16:59:07.162 INFO:teuthology.orchestra.run.smithi037.stderr:Waiting for mgr... 2024-02-11T16:59:07.162 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi037 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpjftqjcr4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4ata0hlm:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-02-11T16:59:07.691 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:07.686+0000 7f857dde1700 1 Processor -- start 2024-02-11T16:59:07.691 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:07.686+0000 7f857dde1700 1 -- start start 2024-02-11T16:59:07.691 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:07.686+0000 7f857dde1700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f85780703f0 0x7f85780fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:07.691 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:07.686+0000 7f857dde1700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85780fee10 con 0x7f85780703f0 2024-02-11T16:59:07.691 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:07.687+0000 7f85777fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f85780703f0 0x7f85780fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:07.691 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:07.687+0000 7f85777fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f85780703f0 0x7f85780fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:51536/0 (socket says 172.21.15.37:51536) 2024-02-11T16:59:07.691 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:07.687+0000 7f85777fe700 1 -- 172.21.15.37:0/3682423713 learned_addr learned my addr 172.21.15.37:0/3682423713 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-02-11T16:59:07.691 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:07.687+0000 7f85777fe700 1 -- 172.21.15.37:0/3682423713 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85780fef50 con 0x7f85780703f0 2024-02-11T16:59:07.691 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:07.687+0000 7f85777fe700 1 --2- 172.21.15.37:0/3682423713 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f85780703f0 0x7f85780fe920 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f856800b5f0 tx=0x7f8568006ee0).ready entity=mon.0 client_cookie=56c8d75adc4330ec server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:07.691 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:07.687+0000 7f85767fc700 1 -- 172.21.15.37:0/3682423713 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f856800d600 con 0x7f85780703f0 2024-02-11T16:59:07.691 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:07.688+0000 7f85767fc700 1 -- 172.21.15.37:0/3682423713 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f856800dbd0 con 0x7f85780703f0 2024-02-11T16:59:07.691 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:07.688+0000 7f85767fc700 1 -- 172.21.15.37:0/3682423713 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8568011c00 con 0x7f85780703f0 2024-02-11T16:59:07.691 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:07.688+0000 7f857dde1700 1 -- 172.21.15.37:0/3682423713 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f85780703f0 msgr2=0x7f85780fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:07.692 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:07.688+0000 7f857dde1700 1 --2- 172.21.15.37:0/3682423713 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f85780703f0 0x7f85780fe920 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f856800b5f0 tx=0x7f8568006ee0).stop 2024-02-11T16:59:07.692 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:07.688+0000 7f857dde1700 1 -- 172.21.15.37:0/3682423713 shutdown_connections 2024-02-11T16:59:07.692 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:07.688+0000 7f857dde1700 1 --2- 172.21.15.37:0/3682423713 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f85780703f0 0x7f85780fe920 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:07.692 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:07.688+0000 7f857dde1700 1 -- 172.21.15.37:0/3682423713 >> 172.21.15.37:0/3682423713 conn(0x7f85780f63c0 msgr2=0x7f85780f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T16:59:07.692 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:07.688+0000 7f857dde1700 1 -- 172.21.15.37:0/3682423713 shutdown_connections 2024-02-11T16:59:07.692 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:07.688+0000 7f857dde1700 1 -- 172.21.15.37:0/3682423713 wait complete. 2024-02-11T16:59:07.692 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:07.689+0000 7f857dde1700 1 Processor -- start 2024-02-11T16:59:07.693 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:07.689+0000 7f857dde1700 1 -- start start 2024-02-11T16:59:07.693 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:07.689+0000 7f857dde1700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f85780703f0 0x7f85780fc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:07.693 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:07.689+0000 7f857dde1700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85780fa820 con 0x7f85780703f0 2024-02-11T16:59:07.693 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:07.689+0000 7f85777fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f85780703f0 0x7f85780fc3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:07.693 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:07.689+0000 7f85777fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f85780703f0 0x7f85780fc3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:51544/0 (socket says 172.21.15.37:51544) 2024-02-11T16:59:07.694 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:07.689+0000 7f85777fe700 1 -- 172.21.15.37:0/3883374187 learned_addr learned my addr 172.21.15.37:0/3883374187 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-02-11T16:59:07.694 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:07.690+0000 7f85777fe700 1 -- 172.21.15.37:0/3883374187 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f856800b200 con 0x7f85780703f0 2024-02-11T16:59:07.694 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:07.690+0000 7f85777fe700 1 --2- 172.21.15.37:0/3883374187 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f85780703f0 0x7f85780fc3a0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f856800e040 tx=0x7f8568006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:07.694 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:07.690+0000 7f855ffff700 1 -- 172.21.15.37:0/3883374187 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f856800d9f0 con 0x7f85780703f0 2024-02-11T16:59:07.694 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:07.690+0000 7f855ffff700 1 -- 172.21.15.37:0/3883374187 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f856801a9f0 con 0x7f85780703f0 2024-02-11T16:59:07.694 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:07.690+0000 7f857dde1700 1 -- 172.21.15.37:0/3883374187 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85780fa9c0 con 0x7f85780703f0 2024-02-11T16:59:07.694 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:07.690+0000 7f855ffff700 1 -- 172.21.15.37:0/3883374187 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f856801b8c0 con 0x7f85780703f0 2024-02-11T16:59:07.694 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:07.690+0000 7f857dde1700 1 -- 172.21.15.37:0/3883374187 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85780fadd0 con 0x7f85780703f0 2024-02-11T16:59:07.694 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:07.691+0000 7f855ffff700 1 -- 172.21.15.37:0/3883374187 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f856800f070 con 0x7f85780703f0 2024-02-11T16:59:07.694 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:07.691+0000 7f855ffff700 1 -- 172.21.15.37:0/3883374187 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f85680257d0 con 0x7f85780703f0 2024-02-11T16:59:07.694 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:07.691+0000 7f857dde1700 1 -- 172.21.15.37:0/3883374187 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8578045ad0 con 0x7f85780703f0 2024-02-11T16:59:07.695 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:07.693+0000 7f855ffff700 1 -- 172.21.15.37:0/3883374187 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f8568016070 con 0x7f85780703f0 2024-02-11T16:59:07.771 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:07.762+0000 7f857dde1700 1 -- 172.21.15.37:0/3883374187 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f85781048c0 con 0x7f85780703f0 2024-02-11T16:59:07.771 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:07.763+0000 7f855ffff700 1 -- 172.21.15.37:0/3883374187 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f856801ba20 con 0x7f85780703f0 2024-02-11T16:59:07.771 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:07.771 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: { 2024-02-11T16:59:07.771 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "fsid": "c65cd888-c8fe-11ee-95b9-87774f69a715", 2024-02-11T16:59:07.771 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "health": { 2024-02-11T16:59:07.771 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-11T16:59:07.771 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "checks": {}, 2024-02-11T16:59:07.771 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "mutes": [] 2024-02-11T16:59:07.771 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: }, 2024-02-11T16:59:07.771 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-11T16:59:07.771 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "quorum": [ 2024-02-11T16:59:07.772 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 0 2024-02-11T16:59:07.772 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: ], 2024-02-11T16:59:07.772 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-11T16:59:07.772 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "smithi037" 2024-02-11T16:59:07.772 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: ], 2024-02-11T16:59:07.772 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "quorum_age": 2, 2024-02-11T16:59:07.772 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "monmap": { 2024-02-11T16:59:07.772 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-11T16:59:07.772 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-02-11T16:59:07.772 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-11T16:59:07.772 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: }, 2024-02-11T16:59:07.772 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "osdmap": { 2024-02-11T16:59:07.772 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-11T16:59:07.772 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-11T16:59:07.773 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-11T16:59:07.773 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-11T16:59:07.773 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-11T16:59:07.773 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-11T16:59:07.773 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-11T16:59:07.773 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: }, 2024-02-11T16:59:07.773 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "pgmap": { 2024-02-11T16:59:07.773 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-11T16:59:07.773 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-11T16:59:07.773 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-11T16:59:07.773 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-11T16:59:07.773 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-11T16:59:07.773 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-11T16:59:07.773 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-11T16:59:07.773 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-11T16:59:07.774 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: }, 2024-02-11T16:59:07.774 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "fsmap": { 2024-02-11T16:59:07.774 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-11T16:59:07.774 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-11T16:59:07.774 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-11T16:59:07.774 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: }, 2024-02-11T16:59:07.774 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-11T16:59:07.774 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "available": false, 2024-02-11T16:59:07.774 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-11T16:59:07.774 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "modules": [ 2024-02-11T16:59:07.774 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "iostat", 2024-02-11T16:59:07.774 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "restful" 2024-02-11T16:59:07.774 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: ], 2024-02-11T16:59:07.774 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "services": {} 2024-02-11T16:59:07.774 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: }, 2024-02-11T16:59:07.775 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "servicemap": { 2024-02-11T16:59:07.775 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-11T16:59:07.775 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "modified": "2024-02-11T16:58:59.740742+0000", 2024-02-11T16:59:07.775 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "services": {} 2024-02-11T16:59:07.775 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: }, 2024-02-11T16:59:07.775 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-11T16:59:07.775 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: } 2024-02-11T16:59:07.775 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:07.764+0000 7f857dde1700 1 -- 172.21.15.37:0/3883374187 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f85780703f0 msgr2=0x7f85780fc3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:07.775 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:07.764+0000 7f857dde1700 1 --2- 172.21.15.37:0/3883374187 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f85780703f0 0x7f85780fc3a0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f856800e040 tx=0x7f8568006ee0).stop 2024-02-11T16:59:07.775 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:07.764+0000 7f857dde1700 1 -- 172.21.15.37:0/3883374187 shutdown_connections 2024-02-11T16:59:07.775 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:07.764+0000 7f857dde1700 1 --2- 172.21.15.37:0/3883374187 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f85780703f0 0x7f85780fc3a0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:07.775 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:07.764+0000 7f857dde1700 1 -- 172.21.15.37:0/3883374187 >> 172.21.15.37:0/3883374187 conn(0x7f85780f63c0 msgr2=0x7f85780f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T16:59:07.775 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:07.764+0000 7f857dde1700 1 -- 172.21.15.37:0/3883374187 shutdown_connections 2024-02-11T16:59:07.775 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:07.764+0000 7f857dde1700 1 -- 172.21.15.37:0/3883374187 wait complete. 2024-02-11T16:59:08.154 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:07 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:07.764764+0000 mon.smithi037 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.37:0/3883374187' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-11T16:59:08.440 INFO:teuthology.orchestra.run.smithi037.stderr:mgr not available, waiting (1/15)... 2024-02-11T16:59:10.443 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi037 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpjftqjcr4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4ata0hlm:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-02-11T16:59:12.684 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:12 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T16:59:12.176606+0000 mon.smithi037 (mon.0) 8 : cluster [INF] Activating manager daemon smithi037.eofnaq 2024-02-11T16:59:12.684 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:12 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T16:59:12.179331+0000 mon.smithi037 (mon.0) 9 : cluster [DBG] mgrmap e2: smithi037.eofnaq(active, starting, since 0.00284026s) 2024-02-11T16:59:12.684 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:12 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:12.180317+0000 mon.smithi037 (mon.0) 10 : audit [DBG] from='mgr.14100 172.21.15.37:0/3888328983' entity='mgr.smithi037.eofnaq' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-11T16:59:12.684 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:12 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:12.180502+0000 mon.smithi037 (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.37:0/3888328983' entity='mgr.smithi037.eofnaq' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-11T16:59:12.684 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:12 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:12.180668+0000 mon.smithi037 (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.37:0/3888328983' entity='mgr.smithi037.eofnaq' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-11T16:59:12.684 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:12 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:12.180848+0000 mon.smithi037 (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.37:0/3888328983' entity='mgr.smithi037.eofnaq' cmd=[{"prefix": "mon metadata", "id": "smithi037"}]: dispatch 2024-02-11T16:59:12.684 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:12 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:12.181009+0000 mon.smithi037 (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.37:0/3888328983' entity='mgr.smithi037.eofnaq' cmd=[{"prefix": "mgr metadata", "who": "smithi037.eofnaq", "id": "smithi037.eofnaq"}]: dispatch 2024-02-11T16:59:12.684 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:12 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T16:59:12.195103+0000 mon.smithi037 (mon.0) 15 : cluster [INF] Manager daemon smithi037.eofnaq is now available 2024-02-11T16:59:12.684 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:12 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:12.202101+0000 mon.smithi037 (mon.0) 16 : audit [INF] from='mgr.14100 172.21.15.37:0/3888328983' entity='mgr.smithi037.eofnaq' 2024-02-11T16:59:12.684 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:12 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:12.208545+0000 mon.smithi037 (mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.37:0/3888328983' entity='mgr.smithi037.eofnaq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi037.eofnaq/mirror_snapshot_schedule"}]: dispatch 2024-02-11T16:59:12.685 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:12 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:12.209705+0000 mon.smithi037 (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.37:0/3888328983' entity='mgr.smithi037.eofnaq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi037.eofnaq/trash_purge_schedule"}]: dispatch 2024-02-11T16:59:12.685 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:12 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:12.215221+0000 mon.smithi037 (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.37:0/3888328983' entity='mgr.smithi037.eofnaq' 2024-02-11T16:59:12.685 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:12 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:12.217608+0000 mon.smithi037 (mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.37:0/3888328983' entity='mgr.smithi037.eofnaq' 2024-02-11T16:59:12.998 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:12.996+0000 7f3a12f89700 1 Processor -- start 2024-02-11T16:59:13.002 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:12.996+0000 7f3a12f89700 1 -- start start 2024-02-11T16:59:13.002 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:12.996+0000 7f3a12f89700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3a0c0fab80 0x7f3a0c0fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:13.003 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:12.996+0000 7f3a12f89700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a0c0fb490 con 0x7f3a0c0fab80 2024-02-11T16:59:13.003 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:12.996+0000 7f3a10d25700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3a0c0fab80 0x7f3a0c0fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:13.003 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:12.996+0000 7f3a10d25700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3a0c0fab80 0x7f3a0c0fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:32880/0 (socket says 172.21.15.37:32880) 2024-02-11T16:59:13.003 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:12.996+0000 7f3a10d25700 1 -- 172.21.15.37:0/1594856315 learned_addr learned my addr 172.21.15.37:0/1594856315 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-02-11T16:59:13.003 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:12.997+0000 7f3a10d25700 1 -- 172.21.15.37:0/1594856315 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a0c0fb5d0 con 0x7f3a0c0fab80 2024-02-11T16:59:13.003 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:12.997+0000 7f3a10d25700 1 --2- 172.21.15.37:0/1594856315 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3a0c0fab80 0x7f3a0c0fafa0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f3a0000b5f0 tx=0x7f3a00006ee0).ready entity=mon.0 client_cookie=5f813e1dbe26b73c server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:13.003 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:12.997+0000 7f3a0b7fe700 1 -- 172.21.15.37:0/1594856315 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a0000d600 con 0x7f3a0c0fab80 2024-02-11T16:59:13.003 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:12.997+0000 7f3a0b7fe700 1 -- 172.21.15.37:0/1594856315 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f3a0000dbd0 con 0x7f3a0c0fab80 2024-02-11T16:59:13.003 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:12.997+0000 7f3a0b7fe700 1 -- 172.21.15.37:0/1594856315 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a00011c00 con 0x7f3a0c0fab80 2024-02-11T16:59:13.003 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:12.997+0000 7f3a12f89700 1 -- 172.21.15.37:0/1594856315 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3a0c0fab80 msgr2=0x7f3a0c0fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:13.003 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:12.998+0000 7f3a12f89700 1 --2- 172.21.15.37:0/1594856315 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3a0c0fab80 0x7f3a0c0fafa0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f3a0000b5f0 tx=0x7f3a00006ee0).stop 2024-02-11T16:59:13.003 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:12.998+0000 7f3a12f89700 1 -- 172.21.15.37:0/1594856315 shutdown_connections 2024-02-11T16:59:13.003 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:12.998+0000 7f3a12f89700 1 --2- 172.21.15.37:0/1594856315 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3a0c0fab80 0x7f3a0c0fafa0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:13.003 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:12.998+0000 7f3a12f89700 1 -- 172.21.15.37:0/1594856315 >> 172.21.15.37:0/1594856315 conn(0x7f3a0c0f63c0 msgr2=0x7f3a0c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T16:59:13.004 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:12.998+0000 7f3a12f89700 1 -- 172.21.15.37:0/1594856315 shutdown_connections 2024-02-11T16:59:13.004 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:12.998+0000 7f3a12f89700 1 -- 172.21.15.37:0/1594856315 wait complete. 2024-02-11T16:59:13.004 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:12.998+0000 7f3a12f89700 1 Processor -- start 2024-02-11T16:59:13.004 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:12.998+0000 7f3a12f89700 1 -- start start 2024-02-11T16:59:13.004 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:12.999+0000 7f3a12f89700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3a0c0fab80 0x7f3a0c10fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:13.004 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:12.999+0000 7f3a12f89700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a0c10b510 con 0x7f3a0c0fab80 2024-02-11T16:59:13.004 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:12.999+0000 7f3a10d25700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3a0c0fab80 0x7f3a0c10fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:13.004 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:12.999+0000 7f3a10d25700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3a0c0fab80 0x7f3a0c10fb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:32882/0 (socket says 172.21.15.37:32882) 2024-02-11T16:59:13.004 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:12.999+0000 7f3a10d25700 1 -- 172.21.15.37:0/3616856319 learned_addr learned my addr 172.21.15.37:0/3616856319 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-02-11T16:59:13.004 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:12.999+0000 7f3a10d25700 1 -- 172.21.15.37:0/3616856319 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a0000b200 con 0x7f3a0c0fab80 2024-02-11T16:59:13.004 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:12.999+0000 7f3a10d25700 1 --2- 172.21.15.37:0/3616856319 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3a0c0fab80 0x7f3a0c10fb20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f3a0000b1d0 tx=0x7f3a00006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:13.004 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:12.999+0000 7f3a097fa700 1 -- 172.21.15.37:0/3616856319 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a0001b410 con 0x7f3a0c0fab80 2024-02-11T16:59:13.004 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:12.999+0000 7f3a097fa700 1 -- 172.21.15.37:0/3616856319 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f3a0001b9e0 con 0x7f3a0c0fab80 2024-02-11T16:59:13.005 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:12.999+0000 7f3a097fa700 1 -- 172.21.15.37:0/3616856319 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a0001a840 con 0x7f3a0c0fab80 2024-02-11T16:59:13.005 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:12.999+0000 7f3a12f89700 1 -- 172.21.15.37:0/3616856319 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a0c070450 con 0x7f3a0c0fab80 2024-02-11T16:59:13.005 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:12.999+0000 7f3a12f89700 1 -- 172.21.15.37:0/3616856319 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a0c110010 con 0x7f3a0c0fab80 2024-02-11T16:59:13.005 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:13.000+0000 7f3a097fa700 1 -- 172.21.15.37:0/3616856319 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 2) v1 ==== 35959+0+0 (secure 0 0 0) 0x7f3a0000f070 con 0x7f3a0c0fab80 2024-02-11T16:59:13.005 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:13.000+0000 7f3a097fa700 1 -- 172.21.15.37:0/3616856319 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f3a00041010 con 0x7f3a0c0fab80 2024-02-11T16:59:13.005 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:13.000+0000 7f3a12f89700 1 -- 172.21.15.37:0/3616856319 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a0c045ad0 con 0x7f3a0c0fab80 2024-02-11T16:59:13.005 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:13.003+0000 7f3a097fa700 1 -- 172.21.15.37:0/3616856319 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f3a00016080 con 0x7f3a0c0fab80 2024-02-11T16:59:13.081 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:13.072+0000 7f3a12f89700 1 -- 172.21.15.37:0/3616856319 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f3a0c027a00 con 0x7f3a0c0fab80 2024-02-11T16:59:13.081 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:13.072+0000 7f3a097fa700 1 -- 172.21.15.37:0/3616856319 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f3a00019cd0 con 0x7f3a0c0fab80 2024-02-11T16:59:13.081 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:13.081 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: { 2024-02-11T16:59:13.081 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "fsid": "c65cd888-c8fe-11ee-95b9-87774f69a715", 2024-02-11T16:59:13.081 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "health": { 2024-02-11T16:59:13.081 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-11T16:59:13.081 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "checks": {}, 2024-02-11T16:59:13.081 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "mutes": [] 2024-02-11T16:59:13.082 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: }, 2024-02-11T16:59:13.082 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-11T16:59:13.082 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "quorum": [ 2024-02-11T16:59:13.082 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 0 2024-02-11T16:59:13.082 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: ], 2024-02-11T16:59:13.082 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-11T16:59:13.082 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "smithi037" 2024-02-11T16:59:13.082 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: ], 2024-02-11T16:59:13.082 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "quorum_age": 8, 2024-02-11T16:59:13.082 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "monmap": { 2024-02-11T16:59:13.082 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-11T16:59:13.082 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-02-11T16:59:13.082 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-11T16:59:13.082 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: }, 2024-02-11T16:59:13.082 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "osdmap": { 2024-02-11T16:59:13.083 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-11T16:59:13.083 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-11T16:59:13.083 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-11T16:59:13.083 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-11T16:59:13.083 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-11T16:59:13.083 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-11T16:59:13.083 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-11T16:59:13.083 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: }, 2024-02-11T16:59:13.083 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "pgmap": { 2024-02-11T16:59:13.083 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-11T16:59:13.083 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-11T16:59:13.083 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-11T16:59:13.083 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-11T16:59:13.083 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-11T16:59:13.083 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-11T16:59:13.083 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-11T16:59:13.084 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-11T16:59:13.084 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: }, 2024-02-11T16:59:13.084 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "fsmap": { 2024-02-11T16:59:13.084 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-11T16:59:13.084 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-11T16:59:13.084 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-11T16:59:13.084 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: }, 2024-02-11T16:59:13.084 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-11T16:59:13.084 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "available": false, 2024-02-11T16:59:13.084 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-11T16:59:13.084 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "modules": [ 2024-02-11T16:59:13.084 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "iostat", 2024-02-11T16:59:13.084 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "restful" 2024-02-11T16:59:13.084 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: ], 2024-02-11T16:59:13.085 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "services": {} 2024-02-11T16:59:13.085 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: }, 2024-02-11T16:59:13.085 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "servicemap": { 2024-02-11T16:59:13.085 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-11T16:59:13.085 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "modified": "2024-02-11T16:58:59.740742+0000", 2024-02-11T16:59:13.085 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "services": {} 2024-02-11T16:59:13.085 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: }, 2024-02-11T16:59:13.085 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-11T16:59:13.085 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: } 2024-02-11T16:59:13.085 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:13.073+0000 7f3a12f89700 1 -- 172.21.15.37:0/3616856319 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3a0c0fab80 msgr2=0x7f3a0c10fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:13.085 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:13.073+0000 7f3a12f89700 1 --2- 172.21.15.37:0/3616856319 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3a0c0fab80 0x7f3a0c10fb20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f3a0000b1d0 tx=0x7f3a00006ee0).stop 2024-02-11T16:59:13.085 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:13.073+0000 7f3a12f89700 1 -- 172.21.15.37:0/3616856319 shutdown_connections 2024-02-11T16:59:13.085 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:13.073+0000 7f3a12f89700 1 --2- 172.21.15.37:0/3616856319 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3a0c0fab80 0x7f3a0c10fb20 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:13.085 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:13.073+0000 7f3a12f89700 1 -- 172.21.15.37:0/3616856319 >> 172.21.15.37:0/3616856319 conn(0x7f3a0c0f63c0 msgr2=0x7f3a0c0f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T16:59:13.086 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:13.073+0000 7f3a12f89700 1 -- 172.21.15.37:0/3616856319 shutdown_connections 2024-02-11T16:59:13.086 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:13.073+0000 7f3a12f89700 1 -- 172.21.15.37:0/3616856319 wait complete. 2024-02-11T16:59:13.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:13.074425+0000 mon.smithi037 (mon.0) 21 : audit [DBG] from='client.? 172.21.15.37:0/3616856319' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-11T16:59:13.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T16:59:13.181803+0000 mon.smithi037 (mon.0) 22 : cluster [DBG] mgrmap e3: smithi037.eofnaq(active, since 1.00531s) 2024-02-11T16:59:13.709 INFO:teuthology.orchestra.run.smithi037.stderr:mgr not available, waiting (2/15)... 2024-02-11T16:59:15.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:15 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T16:59:14.233869+0000 mon.smithi037 (mon.0) 23 : cluster [DBG] mgrmap e4: smithi037.eofnaq(active, since 2s) 2024-02-11T16:59:15.711 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi037 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpjftqjcr4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4ata0hlm:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-02-11T16:59:16.280 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:16.270+0000 7f7c76700700 1 Processor -- start 2024-02-11T16:59:16.280 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:16.270+0000 7f7c76700700 1 -- start start 2024-02-11T16:59:16.280 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:16.270+0000 7f7c76700700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c70070430 0x7f7c700fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:16.280 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:16.270+0000 7f7c76700700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c700fedd0 con 0x7f7c70070430 2024-02-11T16:59:16.281 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:16.271+0000 7f7c6ffff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c70070430 0x7f7c700fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:16.281 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:16.271+0000 7f7c6ffff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c70070430 0x7f7c700fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:32886/0 (socket says 172.21.15.37:32886) 2024-02-11T16:59:16.281 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:16.271+0000 7f7c6ffff700 1 -- 172.21.15.37:0/1818292415 learned_addr learned my addr 172.21.15.37:0/1818292415 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-02-11T16:59:16.281 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:16.271+0000 7f7c6ffff700 1 -- 172.21.15.37:0/1818292415 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c700fef10 con 0x7f7c70070430 2024-02-11T16:59:16.281 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:16.271+0000 7f7c6ffff700 1 --2- 172.21.15.37:0/1818292415 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c70070430 0x7f7c700fe8e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f7c6000b5f0 tx=0x7f7c60006ee0).ready entity=mon.0 client_cookie=64e01337d29ca442 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:16.281 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:16.271+0000 7f7c6effd700 1 -- 172.21.15.37:0/1818292415 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c6000d600 con 0x7f7c70070430 2024-02-11T16:59:16.281 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:16.271+0000 7f7c6effd700 1 -- 172.21.15.37:0/1818292415 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f7c6000dbd0 con 0x7f7c70070430 2024-02-11T16:59:16.281 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:16.271+0000 7f7c6effd700 1 -- 172.21.15.37:0/1818292415 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c60011c00 con 0x7f7c70070430 2024-02-11T16:59:16.281 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:16.272+0000 7f7c76700700 1 -- 172.21.15.37:0/1818292415 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c70070430 msgr2=0x7f7c700fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:16.281 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:16.272+0000 7f7c76700700 1 --2- 172.21.15.37:0/1818292415 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c70070430 0x7f7c700fe8e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f7c6000b5f0 tx=0x7f7c60006ee0).stop 2024-02-11T16:59:16.281 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:16.272+0000 7f7c76700700 1 -- 172.21.15.37:0/1818292415 shutdown_connections 2024-02-11T16:59:16.281 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:16.272+0000 7f7c76700700 1 --2- 172.21.15.37:0/1818292415 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c70070430 0x7f7c700fe8e0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:16.282 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:16.272+0000 7f7c76700700 1 -- 172.21.15.37:0/1818292415 >> 172.21.15.37:0/1818292415 conn(0x7f7c700f63c0 msgr2=0x7f7c700f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T16:59:16.282 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:16.272+0000 7f7c76700700 1 -- 172.21.15.37:0/1818292415 shutdown_connections 2024-02-11T16:59:16.282 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:16.272+0000 7f7c76700700 1 -- 172.21.15.37:0/1818292415 wait complete. 2024-02-11T16:59:16.282 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:16.272+0000 7f7c76700700 1 Processor -- start 2024-02-11T16:59:16.282 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:16.272+0000 7f7c76700700 1 -- start start 2024-02-11T16:59:16.282 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:16.273+0000 7f7c76700700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c70070430 0x7f7c7010aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:16.282 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:16.273+0000 7f7c76700700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7010af30 con 0x7f7c70070430 2024-02-11T16:59:16.282 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:16.273+0000 7f7c6ffff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c70070430 0x7f7c7010aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:16.282 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:16.273+0000 7f7c6ffff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c70070430 0x7f7c7010aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:32888/0 (socket says 172.21.15.37:32888) 2024-02-11T16:59:16.282 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:16.273+0000 7f7c6ffff700 1 -- 172.21.15.37:0/3751181993 learned_addr learned my addr 172.21.15.37:0/3751181993 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-02-11T16:59:16.282 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:16.273+0000 7f7c6ffff700 1 -- 172.21.15.37:0/3751181993 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c6000b200 con 0x7f7c70070430 2024-02-11T16:59:16.283 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:16.273+0000 7f7c6ffff700 1 --2- 172.21.15.37:0/3751181993 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c70070430 0x7f7c7010aa40 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f7c6000e040 tx=0x7f7c60006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:16.283 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:16.273+0000 7f7c6cff9700 1 -- 172.21.15.37:0/3751181993 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c6000d9f0 con 0x7f7c70070430 2024-02-11T16:59:16.283 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:16.273+0000 7f7c76700700 1 -- 172.21.15.37:0/3751181993 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c7010b130 con 0x7f7c70070430 2024-02-11T16:59:16.283 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:16.273+0000 7f7c6cff9700 1 -- 172.21.15.37:0/3751181993 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f7c6001a9f0 con 0x7f7c70070430 2024-02-11T16:59:16.283 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:16.273+0000 7f7c6cff9700 1 -- 172.21.15.37:0/3751181993 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c6001b8c0 con 0x7f7c70070430 2024-02-11T16:59:16.283 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:16.273+0000 7f7c76700700 1 -- 172.21.15.37:0/3751181993 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c7010b540 con 0x7f7c70070430 2024-02-11T16:59:16.283 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:16.274+0000 7f7c6cff9700 1 -- 172.21.15.37:0/3751181993 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 4) v1 ==== 36109+0+0 (secure 0 0 0) 0x7f7c6000f070 con 0x7f7c70070430 2024-02-11T16:59:16.283 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:16.274+0000 7f7c6cff9700 1 --2- 172.21.15.37:0/3751181993 >> [v2:172.21.15.37:6800/183682512,v1:172.21.15.37:6801/183682512] conn(0x7f7c5c02fd50 0x7f7c5c032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:16.283 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:16.274+0000 7f7c76700700 1 -- 172.21.15.37:0/3751181993 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c70045ad0 con 0x7f7c70070430 2024-02-11T16:59:16.283 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:16.274+0000 7f7c6cff9700 1 -- 172.21.15.37:0/3751181993 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f7c600411b0 con 0x7f7c70070430 2024-02-11T16:59:16.283 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:16.274+0000 7f7c6f7fe700 1 --2- 172.21.15.37:0/3751181993 >> [v2:172.21.15.37:6800/183682512,v1:172.21.15.37:6801/183682512] conn(0x7f7c5c02fd50 0x7f7c5c032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:16.283 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:16.275+0000 7f7c6f7fe700 1 --2- 172.21.15.37:0/3751181993 >> [v2:172.21.15.37:6800/183682512,v1:172.21.15.37:6801/183682512] conn(0x7f7c5c02fd50 0x7f7c5c032210 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f7c640096a0 tx=0x7f7c64006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:16.284 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:16.279+0000 7f7c6cff9700 1 -- 172.21.15.37:0/3751181993 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7c60022b90 con 0x7f7c70070430 2024-02-11T16:59:16.502 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:16.493+0000 7f7c76700700 1 -- 172.21.15.37:0/3751181993 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f7c700ffb90 con 0x7f7c70070430 2024-02-11T16:59:16.502 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:16.493+0000 7f7c6cff9700 1 -- 172.21.15.37:0/3751181993 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f7c60016090 con 0x7f7c70070430 2024-02-11T16:59:16.502 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:16.502 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: { 2024-02-11T16:59:16.502 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "fsid": "c65cd888-c8fe-11ee-95b9-87774f69a715", 2024-02-11T16:59:16.502 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "health": { 2024-02-11T16:59:16.502 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-11T16:59:16.502 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "checks": {}, 2024-02-11T16:59:16.502 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "mutes": [] 2024-02-11T16:59:16.502 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: }, 2024-02-11T16:59:16.502 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-11T16:59:16.502 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "quorum": [ 2024-02-11T16:59:16.502 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 0 2024-02-11T16:59:16.503 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: ], 2024-02-11T16:59:16.503 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-11T16:59:16.503 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "smithi037" 2024-02-11T16:59:16.503 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: ], 2024-02-11T16:59:16.503 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "quorum_age": 11, 2024-02-11T16:59:16.503 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "monmap": { 2024-02-11T16:59:16.503 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-11T16:59:16.503 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-02-11T16:59:16.503 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-11T16:59:16.503 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: }, 2024-02-11T16:59:16.503 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "osdmap": { 2024-02-11T16:59:16.503 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-11T16:59:16.503 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-11T16:59:16.503 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-11T16:59:16.504 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-11T16:59:16.504 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-11T16:59:16.504 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-11T16:59:16.504 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-11T16:59:16.504 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: }, 2024-02-11T16:59:16.504 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "pgmap": { 2024-02-11T16:59:16.504 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-11T16:59:16.504 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-11T16:59:16.504 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-11T16:59:16.504 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-11T16:59:16.504 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-11T16:59:16.504 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-11T16:59:16.504 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-11T16:59:16.504 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-11T16:59:16.505 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: }, 2024-02-11T16:59:16.505 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "fsmap": { 2024-02-11T16:59:16.505 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-11T16:59:16.505 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-11T16:59:16.505 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-11T16:59:16.505 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: }, 2024-02-11T16:59:16.505 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-11T16:59:16.505 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "available": true, 2024-02-11T16:59:16.505 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-11T16:59:16.505 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "modules": [ 2024-02-11T16:59:16.505 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "iostat", 2024-02-11T16:59:16.505 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "restful" 2024-02-11T16:59:16.505 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: ], 2024-02-11T16:59:16.505 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "services": {} 2024-02-11T16:59:16.505 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: }, 2024-02-11T16:59:16.505 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "servicemap": { 2024-02-11T16:59:16.506 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-11T16:59:16.506 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "modified": "2024-02-11T16:58:59.740742+0000", 2024-02-11T16:59:16.506 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "services": {} 2024-02-11T16:59:16.506 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: }, 2024-02-11T16:59:16.506 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-11T16:59:16.506 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: } 2024-02-11T16:59:16.507 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:16.496+0000 7f7c76700700 1 -- 172.21.15.37:0/3751181993 >> [v2:172.21.15.37:6800/183682512,v1:172.21.15.37:6801/183682512] conn(0x7f7c5c02fd50 msgr2=0x7f7c5c032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:16.507 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:16.496+0000 7f7c76700700 1 --2- 172.21.15.37:0/3751181993 >> [v2:172.21.15.37:6800/183682512,v1:172.21.15.37:6801/183682512] conn(0x7f7c5c02fd50 0x7f7c5c032210 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f7c640096a0 tx=0x7f7c64006b40).stop 2024-02-11T16:59:16.507 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:16.496+0000 7f7c76700700 1 -- 172.21.15.37:0/3751181993 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c70070430 msgr2=0x7f7c7010aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:16.507 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:16.496+0000 7f7c76700700 1 --2- 172.21.15.37:0/3751181993 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c70070430 0x7f7c7010aa40 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f7c6000e040 tx=0x7f7c60006ee0).stop 2024-02-11T16:59:16.507 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:16.496+0000 7f7c76700700 1 -- 172.21.15.37:0/3751181993 shutdown_connections 2024-02-11T16:59:16.508 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:16.496+0000 7f7c76700700 1 --2- 172.21.15.37:0/3751181993 >> [v2:172.21.15.37:6800/183682512,v1:172.21.15.37:6801/183682512] conn(0x7f7c5c02fd50 0x7f7c5c032210 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:16.508 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:16.496+0000 7f7c76700700 1 --2- 172.21.15.37:0/3751181993 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c70070430 0x7f7c7010aa40 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:16.508 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:16.496+0000 7f7c76700700 1 -- 172.21.15.37:0/3751181993 >> 172.21.15.37:0/3751181993 conn(0x7f7c700f63c0 msgr2=0x7f7c700f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T16:59:16.508 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:16.496+0000 7f7c76700700 1 -- 172.21.15.37:0/3751181993 shutdown_connections 2024-02-11T16:59:16.508 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:16.497+0000 7f7c76700700 1 -- 172.21.15.37:0/3751181993 wait complete. 2024-02-11T16:59:16.837 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:16 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:16.495183+0000 mon.smithi037 (mon.0) 24 : audit [DBG] from='client.? 172.21.15.37:0/3751181993' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-11T16:59:17.057 INFO:teuthology.orchestra.run.smithi037.stderr:mgr is available 2024-02-11T16:59:17.057 INFO:teuthology.orchestra.run.smithi037.stderr:Enabling cephadm module... 2024-02-11T16:59:17.057 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi037 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpjftqjcr4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4ata0hlm:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable cephadm 2024-02-11T16:59:17.632 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:17.629+0000 7f554e4a2700 1 Processor -- start 2024-02-11T16:59:17.633 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:17.629+0000 7f554e4a2700 1 -- start start 2024-02-11T16:59:17.633 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:17.629+0000 7f554e4a2700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f55480fab80 0x7f55480fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:17.633 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:17.629+0000 7f554e4a2700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55480fb490 con 0x7f55480fab80 2024-02-11T16:59:17.633 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:17.629+0000 7f5547fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f55480fab80 0x7f55480fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:17.633 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:17.629+0000 7f5547fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f55480fab80 0x7f55480fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:32900/0 (socket says 172.21.15.37:32900) 2024-02-11T16:59:17.633 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:17.629+0000 7f5547fff700 1 -- 172.21.15.37:0/1269057885 learned_addr learned my addr 172.21.15.37:0/1269057885 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-02-11T16:59:17.633 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:17.630+0000 7f5547fff700 1 -- 172.21.15.37:0/1269057885 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55480fb5d0 con 0x7f55480fab80 2024-02-11T16:59:17.633 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:17.630+0000 7f5547fff700 1 --2- 172.21.15.37:0/1269057885 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f55480fab80 0x7f55480fafa0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f553800b5f0 tx=0x7f5538006ee0).ready entity=mon.0 client_cookie=f63d1b5d72ae903e server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:17.633 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:17.630+0000 7f55477fe700 1 -- 172.21.15.37:0/1269057885 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f553800d600 con 0x7f55480fab80 2024-02-11T16:59:17.633 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:17.630+0000 7f55477fe700 1 -- 172.21.15.37:0/1269057885 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f553800dbd0 con 0x7f55480fab80 2024-02-11T16:59:17.633 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:17.630+0000 7f55477fe700 1 -- 172.21.15.37:0/1269057885 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5538011c00 con 0x7f55480fab80 2024-02-11T16:59:17.635 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:17.631+0000 7f554e4a2700 1 -- 172.21.15.37:0/1269057885 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f55480fab80 msgr2=0x7f55480fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:17.635 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:17.631+0000 7f554e4a2700 1 --2- 172.21.15.37:0/1269057885 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f55480fab80 0x7f55480fafa0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f553800b5f0 tx=0x7f5538006ee0).stop 2024-02-11T16:59:17.635 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:17.631+0000 7f554e4a2700 1 -- 172.21.15.37:0/1269057885 shutdown_connections 2024-02-11T16:59:17.635 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:17.631+0000 7f554e4a2700 1 --2- 172.21.15.37:0/1269057885 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f55480fab80 0x7f55480fafa0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:17.635 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:17.631+0000 7f554e4a2700 1 -- 172.21.15.37:0/1269057885 >> 172.21.15.37:0/1269057885 conn(0x7f55480f63c0 msgr2=0x7f55480f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T16:59:17.635 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:17.631+0000 7f554e4a2700 1 -- 172.21.15.37:0/1269057885 shutdown_connections 2024-02-11T16:59:17.635 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:17.631+0000 7f554e4a2700 1 -- 172.21.15.37:0/1269057885 wait complete. 2024-02-11T16:59:17.635 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:17.632+0000 7f554e4a2700 1 Processor -- start 2024-02-11T16:59:17.635 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:17.632+0000 7f554e4a2700 1 -- start start 2024-02-11T16:59:17.635 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:17.632+0000 7f554e4a2700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f55480fab80 0x7f554810d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:17.635 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:17.632+0000 7f554e4a2700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5548070470 con 0x7f55480fab80 2024-02-11T16:59:17.635 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:17.632+0000 7f5547fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f55480fab80 0x7f554810d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:17.636 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:17.632+0000 7f5547fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f55480fab80 0x7f554810d730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:32916/0 (socket says 172.21.15.37:32916) 2024-02-11T16:59:17.637 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:17.632+0000 7f5547fff700 1 -- 172.21.15.37:0/3319881931 learned_addr learned my addr 172.21.15.37:0/3319881931 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-02-11T16:59:17.637 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:17.633+0000 7f5547fff700 1 -- 172.21.15.37:0/3319881931 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f553800b200 con 0x7f55480fab80 2024-02-11T16:59:17.637 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:17.633+0000 7f5547fff700 1 --2- 172.21.15.37:0/3319881931 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f55480fab80 0x7f554810d730 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f553800e040 tx=0x7f5538006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:17.637 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:17.633+0000 7f55457fa700 1 -- 172.21.15.37:0/3319881931 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f553801b410 con 0x7f55480fab80 2024-02-11T16:59:17.637 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:17.633+0000 7f55457fa700 1 -- 172.21.15.37:0/3319881931 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f553801b9e0 con 0x7f55480fab80 2024-02-11T16:59:17.637 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:17.633+0000 7f554e4a2700 1 -- 172.21.15.37:0/3319881931 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5548070670 con 0x7f55480fab80 2024-02-11T16:59:17.637 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:17.633+0000 7f55457fa700 1 -- 172.21.15.37:0/3319881931 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f553801a820 con 0x7f55480fab80 2024-02-11T16:59:17.637 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:17.633+0000 7f554e4a2700 1 -- 172.21.15.37:0/3319881931 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f554810dde0 con 0x7f55480fab80 2024-02-11T16:59:17.637 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:17.634+0000 7f55457fa700 1 -- 172.21.15.37:0/3319881931 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 4) v1 ==== 36109+0+0 (secure 0 0 0) 0x7f553800f070 con 0x7f55480fab80 2024-02-11T16:59:17.637 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:17.634+0000 7f55457fa700 1 --2- 172.21.15.37:0/3319881931 >> [v2:172.21.15.37:6800/183682512,v1:172.21.15.37:6801/183682512] conn(0x7f552802fd50 0x7f5528032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:17.637 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:17.634+0000 7f55457fa700 1 -- 172.21.15.37:0/3319881931 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f5538041250 con 0x7f55480fab80 2024-02-11T16:59:17.637 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:17.634+0000 7f553ffff700 1 --2- 172.21.15.37:0/3319881931 >> [v2:172.21.15.37:6800/183682512,v1:172.21.15.37:6801/183682512] conn(0x7f552802fd50 0x7f5528032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:17.637 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:17.634+0000 7f553ffff700 1 --2- 172.21.15.37:0/3319881931 >> [v2:172.21.15.37:6800/183682512,v1:172.21.15.37:6801/183682512] conn(0x7f552802fd50 0x7f5528032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f55300096a0 tx=0x7f5530006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:17.637 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:17.634+0000 7f554e4a2700 1 -- 172.21.15.37:0/3319881931 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5548045ad0 con 0x7f55480fab80 2024-02-11T16:59:17.641 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:17.639+0000 7f55457fa700 1 -- 172.21.15.37:0/3319881931 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5538016020 con 0x7f55480fab80 2024-02-11T16:59:17.837 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:17.835+0000 7f554e4a2700 1 -- 172.21.15.37:0/3319881931 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f55480fc990 con 0x7f55480fab80 2024-02-11T16:59:17.889 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:17.887+0000 7f55457fa700 1 -- 172.21.15.37:0/3319881931 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f5538025500 con 0x7f55480fab80 2024-02-11T16:59:17.890 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:17.888+0000 7f55457fa700 1 -- 172.21.15.37:0/3319881931 <== mon.0 v2:172.21.15.37:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f553802f6c0 con 0x7f55480fab80 2024-02-11T16:59:17.893 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:17.890+0000 7f554e4a2700 1 -- 172.21.15.37:0/3319881931 >> [v2:172.21.15.37:6800/183682512,v1:172.21.15.37:6801/183682512] conn(0x7f552802fd50 msgr2=0x7f5528032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:17.893 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:17.890+0000 7f554e4a2700 1 --2- 172.21.15.37:0/3319881931 >> [v2:172.21.15.37:6800/183682512,v1:172.21.15.37:6801/183682512] conn(0x7f552802fd50 0x7f5528032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f55300096a0 tx=0x7f5530006b40).stop 2024-02-11T16:59:17.893 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:17.890+0000 7f554e4a2700 1 -- 172.21.15.37:0/3319881931 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f55480fab80 msgr2=0x7f554810d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:17.893 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:17.890+0000 7f554e4a2700 1 --2- 172.21.15.37:0/3319881931 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f55480fab80 0x7f554810d730 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f553800e040 tx=0x7f5538006ee0).stop 2024-02-11T16:59:17.894 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:17.890+0000 7f554e4a2700 1 -- 172.21.15.37:0/3319881931 shutdown_connections 2024-02-11T16:59:17.894 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:17.890+0000 7f554e4a2700 1 --2- 172.21.15.37:0/3319881931 >> [v2:172.21.15.37:6800/183682512,v1:172.21.15.37:6801/183682512] conn(0x7f552802fd50 0x7f5528032210 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:17.894 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:17.890+0000 7f554e4a2700 1 --2- 172.21.15.37:0/3319881931 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f55480fab80 0x7f554810d730 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:17.894 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:17.890+0000 7f554e4a2700 1 -- 172.21.15.37:0/3319881931 >> 172.21.15.37:0/3319881931 conn(0x7f55480f63c0 msgr2=0x7f55480f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T16:59:17.894 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:17.890+0000 7f554e4a2700 1 -- 172.21.15.37:0/3319881931 shutdown_connections 2024-02-11T16:59:17.894 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:17.890+0000 7f554e4a2700 1 -- 172.21.15.37:0/3319881931 wait complete. 2024-02-11T16:59:18.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:17 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:17.837115+0000 mon.smithi037 (mon.0) 25 : audit [INF] from='client.? 172.21.15.37:0/3319881931' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-02-11T16:59:19.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:18 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:17.890191+0000 mon.smithi037 (mon.0) 26 : audit [INF] from='client.? 172.21.15.37:0/3319881931' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-02-11T16:59:19.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:18 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T16:59:17.890240+0000 mon.smithi037 (mon.0) 27 : cluster [DBG] mgrmap e5: smithi037.eofnaq(active, since 5s) 2024-02-11T16:59:21.342 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi037 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpjftqjcr4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4ata0hlm:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2024-02-11T16:59:24.434 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:24 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T16:59:23.012491+0000 mon.smithi037 (mon.0) 28 : cluster [INF] Active manager daemon smithi037.eofnaq restarted 2024-02-11T16:59:24.434 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:24 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T16:59:23.012969+0000 mon.smithi037 (mon.0) 29 : cluster [INF] Activating manager daemon smithi037.eofnaq 2024-02-11T16:59:24.434 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:24 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T16:59:23.016921+0000 mon.smithi037 (mon.0) 30 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2024-02-11T16:59:24.434 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:24 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T16:59:23.065285+0000 mon.smithi037 (mon.0) 31 : cluster [DBG] mgrmap e6: smithi037.eofnaq(active, starting, since 0.0524126s) 2024-02-11T16:59:24.434 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:24 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:23.066432+0000 mon.smithi037 (mon.0) 32 : audit [DBG] from='mgr.14114 172.21.15.37:0/1469767164' entity='mgr.smithi037.eofnaq' cmd=[{"prefix": "mon metadata", "id": "smithi037"}]: dispatch 2024-02-11T16:59:24.434 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:24 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:23.067009+0000 mon.smithi037 (mon.0) 33 : audit [DBG] from='mgr.14114 172.21.15.37:0/1469767164' entity='mgr.smithi037.eofnaq' cmd=[{"prefix": "mgr metadata", "who": "smithi037.eofnaq", "id": "smithi037.eofnaq"}]: dispatch 2024-02-11T16:59:24.434 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:24 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:23.067695+0000 mon.smithi037 (mon.0) 34 : audit [DBG] from='mgr.14114 172.21.15.37:0/1469767164' entity='mgr.smithi037.eofnaq' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-11T16:59:24.434 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:24 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:23.067821+0000 mon.smithi037 (mon.0) 35 : audit [DBG] from='mgr.14114 172.21.15.37:0/1469767164' entity='mgr.smithi037.eofnaq' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-11T16:59:24.434 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:24 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:23.067956+0000 mon.smithi037 (mon.0) 36 : audit [DBG] from='mgr.14114 172.21.15.37:0/1469767164' entity='mgr.smithi037.eofnaq' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-11T16:59:24.434 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:24 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T16:59:23.081226+0000 mon.smithi037 (mon.0) 37 : cluster [INF] Manager daemon smithi037.eofnaq is now available 2024-02-11T16:59:24.435 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:24 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:23.094727+0000 mon.smithi037 (mon.0) 38 : audit [INF] from='mgr.14114 172.21.15.37:0/1469767164' entity='mgr.smithi037.eofnaq' 2024-02-11T16:59:24.435 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:24 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:23.097579+0000 mon.smithi037 (mon.0) 39 : audit [INF] from='mgr.14114 172.21.15.37:0/1469767164' entity='mgr.smithi037.eofnaq' 2024-02-11T16:59:24.435 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:24 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:23.100043+0000 mon.smithi037 (mon.0) 40 : audit [INF] from='mgr.14114 172.21.15.37:0/1469767164' entity='mgr.smithi037.eofnaq' 2024-02-11T16:59:24.435 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:24 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:23.194015+0000 mon.smithi037 (mon.0) 41 : audit [INF] from='mgr.14114 172.21.15.37:0/1469767164' entity='mgr.smithi037.eofnaq' 2024-02-11T16:59:24.435 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:24 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:23.194693+0000 mon.smithi037 (mon.0) 42 : audit [DBG] from='mgr.14114 172.21.15.37:0/1469767164' entity='mgr.smithi037.eofnaq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-11T16:59:24.435 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:24 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:23.195794+0000 mon.smithi037 (mon.0) 43 : audit [DBG] from='mgr.14114 172.21.15.37:0/1469767164' entity='mgr.smithi037.eofnaq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-11T16:59:24.435 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:24 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:23.197250+0000 mon.smithi037 (mon.0) 44 : audit [DBG] from='mgr.14114 172.21.15.37:0/1469767164' entity='mgr.smithi037.eofnaq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-11T16:59:24.435 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:24 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:23.207653+0000 mon.smithi037 (mon.0) 45 : audit [INF] from='mgr.14114 172.21.15.37:0/1469767164' entity='mgr.smithi037.eofnaq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi037.eofnaq/mirror_snapshot_schedule"}]: dispatch 2024-02-11T16:59:24.435 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:24 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:23.208783+0000 mon.smithi037 (mon.0) 46 : audit [INF] from='mgr.14114 172.21.15.37:0/1469767164' entity='mgr.smithi037.eofnaq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi037.eofnaq/trash_purge_schedule"}]: dispatch 2024-02-11T16:59:25.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:25 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T16:59:24.067715+0000 mon.smithi037 (mon.0) 47 : cluster [DBG] mgrmap e7: smithi037.eofnaq(active, since 1.05484s) 2024-02-11T16:59:25.449 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:24.930+0000 7f7eae1cf700 1 Processor -- start 2024-02-11T16:59:25.449 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:24.930+0000 7f7eae1cf700 1 -- start start 2024-02-11T16:59:25.449 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:24.930+0000 7f7eae1cf700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7ea80703f0 0x7f7ea80fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:25.449 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:24.930+0000 7f7eae1cf700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea80fee10 con 0x7f7ea80703f0 2024-02-11T16:59:25.450 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:24.930+0000 7f7ea77fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7ea80703f0 0x7f7ea80fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:25.450 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:24.930+0000 7f7ea77fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7ea80703f0 0x7f7ea80fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:54458/0 (socket says 172.21.15.37:54458) 2024-02-11T16:59:25.450 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:24.930+0000 7f7ea77fe700 1 -- 172.21.15.37:0/1072365291 learned_addr learned my addr 172.21.15.37:0/1072365291 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-02-11T16:59:25.450 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:24.930+0000 7f7ea77fe700 1 -- 172.21.15.37:0/1072365291 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ea80fef50 con 0x7f7ea80703f0 2024-02-11T16:59:25.450 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:24.931+0000 7f7ea77fe700 1 --2- 172.21.15.37:0/1072365291 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7ea80703f0 0x7f7ea80fe920 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f7e9800b5f0 tx=0x7f7e98006ee0).ready entity=mon.0 client_cookie=4a7385bda2d5f43e server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:25.450 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:24.931+0000 7f7ea67fc700 1 -- 172.21.15.37:0/1072365291 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e9800d600 con 0x7f7ea80703f0 2024-02-11T16:59:25.450 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:24.931+0000 7f7ea67fc700 1 -- 172.21.15.37:0/1072365291 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f7e9800dbd0 con 0x7f7ea80703f0 2024-02-11T16:59:25.450 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:24.931+0000 7f7ea67fc700 1 -- 172.21.15.37:0/1072365291 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e98011c00 con 0x7f7ea80703f0 2024-02-11T16:59:25.450 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:24.931+0000 7f7eae1cf700 1 -- 172.21.15.37:0/1072365291 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7ea80703f0 msgr2=0x7f7ea80fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:25.450 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:24.931+0000 7f7eae1cf700 1 --2- 172.21.15.37:0/1072365291 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7ea80703f0 0x7f7ea80fe920 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f7e9800b5f0 tx=0x7f7e98006ee0).stop 2024-02-11T16:59:25.450 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:24.931+0000 7f7eae1cf700 1 -- 172.21.15.37:0/1072365291 shutdown_connections 2024-02-11T16:59:25.450 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:24.931+0000 7f7eae1cf700 1 --2- 172.21.15.37:0/1072365291 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7ea80703f0 0x7f7ea80fe920 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:25.450 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:24.931+0000 7f7eae1cf700 1 -- 172.21.15.37:0/1072365291 >> 172.21.15.37:0/1072365291 conn(0x7f7ea80f63c0 msgr2=0x7f7ea80f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T16:59:25.450 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:24.931+0000 7f7eae1cf700 1 -- 172.21.15.37:0/1072365291 shutdown_connections 2024-02-11T16:59:25.451 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:24.931+0000 7f7eae1cf700 1 -- 172.21.15.37:0/1072365291 wait complete. 2024-02-11T16:59:25.451 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:24.932+0000 7f7eae1cf700 1 Processor -- start 2024-02-11T16:59:25.451 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:24.932+0000 7f7eae1cf700 1 -- start start 2024-02-11T16:59:25.451 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:24.932+0000 7f7eae1cf700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7ea80703f0 0x7f7ea810a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:25.451 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:24.932+0000 7f7eae1cf700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea810aeb0 con 0x7f7ea80703f0 2024-02-11T16:59:25.451 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:24.932+0000 7f7ea77fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7ea80703f0 0x7f7ea810a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:25.451 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:24.932+0000 7f7ea77fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7ea80703f0 0x7f7ea810a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:54474/0 (socket says 172.21.15.37:54474) 2024-02-11T16:59:25.451 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:24.932+0000 7f7ea77fe700 1 -- 172.21.15.37:0/1991980733 learned_addr learned my addr 172.21.15.37:0/1991980733 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-02-11T16:59:25.451 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:24.933+0000 7f7ea77fe700 1 -- 172.21.15.37:0/1991980733 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e9800b200 con 0x7f7ea80703f0 2024-02-11T16:59:25.451 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:24.933+0000 7f7ea77fe700 1 --2- 172.21.15.37:0/1991980733 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7ea80703f0 0x7f7ea810a9c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f7e9800b190 tx=0x7f7e98006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:25.451 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:24.933+0000 7f7e8ffff700 1 -- 172.21.15.37:0/1991980733 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e9801b410 con 0x7f7ea80703f0 2024-02-11T16:59:25.451 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:24.933+0000 7f7e8ffff700 1 -- 172.21.15.37:0/1991980733 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f7e9801b9e0 con 0x7f7ea80703f0 2024-02-11T16:59:25.451 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:24.933+0000 7f7eae1cf700 1 -- 172.21.15.37:0/1991980733 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ea810b0b0 con 0x7f7ea80703f0 2024-02-11T16:59:25.451 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:24.933+0000 7f7e8ffff700 1 -- 172.21.15.37:0/1991980733 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e9801a880 con 0x7f7ea80703f0 2024-02-11T16:59:25.451 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:24.933+0000 7f7eae1cf700 1 -- 172.21.15.37:0/1991980733 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ea810b4c0 con 0x7f7ea80703f0 2024-02-11T16:59:25.451 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:24.934+0000 7f7e8ffff700 1 -- 172.21.15.37:0/1991980733 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 7) v1 ==== 36040+0+0 (secure 0 0 0) 0x7f7e9800f070 con 0x7f7ea80703f0 2024-02-11T16:59:25.452 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:24.934+0000 7f7e8ffff700 1 --2- 172.21.15.37:0/1991980733 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7f7e9402fcf0 0x7f7e940321b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:25.452 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:24.934+0000 7f7e8ffff700 1 -- 172.21.15.37:0/1991980733 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f7e98041100 con 0x7f7ea80703f0 2024-02-11T16:59:25.452 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:24.934+0000 7f7eae1cf700 1 -- 172.21.15.37:0/1991980733 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ea8045ad0 con 0x7f7ea80703f0 2024-02-11T16:59:25.452 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:24.934+0000 7f7ea6ffd700 1 --2- 172.21.15.37:0/1991980733 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7f7e9402fcf0 0x7f7e940321b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:25.452 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:24.934+0000 7f7ea6ffd700 1 --2- 172.21.15.37:0/1991980733 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7f7e9402fcf0 0x7f7e940321b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f7e9c0096a0 tx=0x7f7e9c006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:25.452 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:24.939+0000 7f7e8ffff700 1 -- 172.21.15.37:0/1991980733 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7e98016070 con 0x7f7ea80703f0 2024-02-11T16:59:25.452 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:25.137+0000 7f7eae1cf700 1 -- 172.21.15.37:0/1991980733 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f7ea8104890 con 0x7f7ea80703f0 2024-02-11T16:59:25.452 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:25.137+0000 7f7e8ffff700 1 -- 172.21.15.37:0/1991980733 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v7) v1 ==== 56+0+103 (secure 0 0 0) 0x7f7e98025460 con 0x7f7ea80703f0 2024-02-11T16:59:25.452 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:25.138+0000 7f7eae1cf700 1 -- 172.21.15.37:0/1991980733 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7f7e9402fcf0 msgr2=0x7f7e940321b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:25.452 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:25.139+0000 7f7eae1cf700 1 --2- 172.21.15.37:0/1991980733 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7f7e9402fcf0 0x7f7e940321b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f7e9c0096a0 tx=0x7f7e9c006b40).stop 2024-02-11T16:59:25.452 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:25.139+0000 7f7eae1cf700 1 -- 172.21.15.37:0/1991980733 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7ea80703f0 msgr2=0x7f7ea810a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:25.452 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:25.139+0000 7f7eae1cf700 1 --2- 172.21.15.37:0/1991980733 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7ea80703f0 0x7f7ea810a9c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f7e9800b190 tx=0x7f7e98006ee0).stop 2024-02-11T16:59:25.452 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:25.139+0000 7f7eae1cf700 1 -- 172.21.15.37:0/1991980733 shutdown_connections 2024-02-11T16:59:25.452 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:25.139+0000 7f7eae1cf700 1 --2- 172.21.15.37:0/1991980733 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7f7e9402fcf0 0x7f7e940321b0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:25.453 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:25.139+0000 7f7eae1cf700 1 --2- 172.21.15.37:0/1991980733 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7ea80703f0 0x7f7ea810a9c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:25.453 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:25.139+0000 7f7eae1cf700 1 -- 172.21.15.37:0/1991980733 >> 172.21.15.37:0/1991980733 conn(0x7f7ea80f63c0 msgr2=0x7f7ea80f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T16:59:25.453 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:25.139+0000 7f7eae1cf700 1 -- 172.21.15.37:0/1991980733 shutdown_connections 2024-02-11T16:59:25.453 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:25.139+0000 7f7eae1cf700 1 -- 172.21.15.37:0/1991980733 wait complete. 2024-02-11T16:59:25.453 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: { 2024-02-11T16:59:25.453 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "epoch": 7, 2024-02-11T16:59:25.453 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "available": true, 2024-02-11T16:59:25.453 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "active_name": "smithi037.eofnaq", 2024-02-11T16:59:25.453 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "num_standby": 0 2024-02-11T16:59:25.453 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: } 2024-02-11T16:59:26.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:26 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:25.138948+0000 mon.smithi037 (mon.0) 48 : audit [DBG] from='client.? 172.21.15.37:0/1991980733' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-11T16:59:26.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:26 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T16:59:26.026706+0000 mon.smithi037 (mon.0) 49 : cluster [DBG] mgrmap e8: smithi037.eofnaq(active, since 3s) 2024-02-11T16:59:27.418 INFO:teuthology.orchestra.run.smithi037.stderr:Waiting for the mgr to restart... 2024-02-11T16:59:27.418 INFO:teuthology.orchestra.run.smithi037.stderr:Waiting for mgr epoch 7... 2024-02-11T16:59:27.419 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi037 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpjftqjcr4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4ata0hlm:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2024-02-11T16:59:27.932 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:27.928+0000 7fcb14b5f700 1 Processor -- start 2024-02-11T16:59:27.932 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:27.928+0000 7fcb14b5f700 1 -- start start 2024-02-11T16:59:27.932 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:27.929+0000 7fcb14b5f700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcb100703f0 0x7fcb10100d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:27.932 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:27.929+0000 7fcb14b5f700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb100fc710 con 0x7fcb100703f0 2024-02-11T16:59:27.932 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:27.929+0000 7fcb0e59c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcb100703f0 0x7fcb10100d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:27.932 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:27.929+0000 7fcb0e59c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcb100703f0 0x7fcb10100d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:54476/0 (socket says 172.21.15.37:54476) 2024-02-11T16:59:27.932 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:27.929+0000 7fcb0e59c700 1 -- 172.21.15.37:0/153026623 learned_addr learned my addr 172.21.15.37:0/153026623 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-02-11T16:59:27.933 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:27.929+0000 7fcb0e59c700 1 -- 172.21.15.37:0/153026623 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb10101210 con 0x7fcb100703f0 2024-02-11T16:59:27.933 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:27.929+0000 7fcb0e59c700 1 --2- 172.21.15.37:0/153026623 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcb100703f0 0x7fcb10100d20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fcb0000b5f0 tx=0x7fcb00006ee0).ready entity=mon.0 client_cookie=7c8ddd99196962e1 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:27.933 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:27.929+0000 7fcb0d59a700 1 -- 172.21.15.37:0/153026623 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb0000d600 con 0x7fcb100703f0 2024-02-11T16:59:27.933 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:27.929+0000 7fcb0d59a700 1 -- 172.21.15.37:0/153026623 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fcb0000dbd0 con 0x7fcb100703f0 2024-02-11T16:59:27.933 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:27.929+0000 7fcb0d59a700 1 -- 172.21.15.37:0/153026623 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb00011c00 con 0x7fcb100703f0 2024-02-11T16:59:27.933 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:27.930+0000 7fcb14b5f700 1 -- 172.21.15.37:0/153026623 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcb100703f0 msgr2=0x7fcb10100d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:27.933 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:27.930+0000 7fcb14b5f700 1 --2- 172.21.15.37:0/153026623 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcb100703f0 0x7fcb10100d20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fcb0000b5f0 tx=0x7fcb00006ee0).stop 2024-02-11T16:59:27.935 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:27.930+0000 7fcb14b5f700 1 -- 172.21.15.37:0/153026623 shutdown_connections 2024-02-11T16:59:27.935 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:27.930+0000 7fcb14b5f700 1 --2- 172.21.15.37:0/153026623 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcb100703f0 0x7fcb10100d20 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:27.936 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:27.930+0000 7fcb14b5f700 1 -- 172.21.15.37:0/153026623 >> 172.21.15.37:0/153026623 conn(0x7fcb100f63c0 msgr2=0x7fcb100f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T16:59:27.936 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:27.930+0000 7fcb14b5f700 1 -- 172.21.15.37:0/153026623 shutdown_connections 2024-02-11T16:59:27.936 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:27.930+0000 7fcb14b5f700 1 -- 172.21.15.37:0/153026623 wait complete. 2024-02-11T16:59:27.936 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:27.931+0000 7fcb14b5f700 1 Processor -- start 2024-02-11T16:59:27.936 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:27.931+0000 7fcb14b5f700 1 -- start start 2024-02-11T16:59:27.936 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:27.931+0000 7fcb14b5f700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcb100703f0 0x7fcb1010cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:27.936 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:27.931+0000 7fcb14b5f700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb1010d0b0 con 0x7fcb100703f0 2024-02-11T16:59:27.936 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:27.931+0000 7fcb0e59c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcb100703f0 0x7fcb1010cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:27.936 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:27.931+0000 7fcb0e59c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcb100703f0 0x7fcb1010cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:54488/0 (socket says 172.21.15.37:54488) 2024-02-11T16:59:27.936 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:27.931+0000 7fcb0e59c700 1 -- 172.21.15.37:0/2095697043 learned_addr learned my addr 172.21.15.37:0/2095697043 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-02-11T16:59:27.936 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:27.932+0000 7fcb0e59c700 1 -- 172.21.15.37:0/2095697043 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb0000b200 con 0x7fcb100703f0 2024-02-11T16:59:27.936 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:27.932+0000 7fcb0e59c700 1 --2- 172.21.15.37:0/2095697043 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcb100703f0 0x7fcb1010cbc0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7fcb0000b190 tx=0x7fcb00006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:27.936 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:27.932+0000 7fcb06ffd700 1 -- 172.21.15.37:0/2095697043 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb0001b410 con 0x7fcb100703f0 2024-02-11T16:59:27.937 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:27.932+0000 7fcb06ffd700 1 -- 172.21.15.37:0/2095697043 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fcb0001b9e0 con 0x7fcb100703f0 2024-02-11T16:59:27.937 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:27.932+0000 7fcb14b5f700 1 -- 172.21.15.37:0/2095697043 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb1010d2b0 con 0x7fcb100703f0 2024-02-11T16:59:27.937 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:27.932+0000 7fcb06ffd700 1 -- 172.21.15.37:0/2095697043 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb0001a880 con 0x7fcb100703f0 2024-02-11T16:59:27.937 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:27.932+0000 7fcb14b5f700 1 -- 172.21.15.37:0/2095697043 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb1010d6c0 con 0x7fcb100703f0 2024-02-11T16:59:27.937 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:27.933+0000 7fcb06ffd700 1 -- 172.21.15.37:0/2095697043 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fcb0000f070 con 0x7fcb100703f0 2024-02-11T16:59:27.937 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:27.933+0000 7fcb06ffd700 1 --2- 172.21.15.37:0/2095697043 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7fcaf002fdb0 0x7fcaf0032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:27.937 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:27.933+0000 7fcb06ffd700 1 -- 172.21.15.37:0/2095697043 --> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcaf0032930 con 0x7fcaf002fdb0 2024-02-11T16:59:27.937 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:27.933+0000 7fcb06ffd700 1 -- 172.21.15.37:0/2095697043 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fcb000411d0 con 0x7fcb100703f0 2024-02-11T16:59:27.937 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:27.933+0000 7fcb0dd9b700 1 --2- 172.21.15.37:0/2095697043 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7fcaf002fdb0 0x7fcaf0032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:27.937 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:27.934+0000 7fcb0dd9b700 1 --2- 172.21.15.37:0/2095697043 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7fcaf002fdb0 0x7fcaf0032270 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fcaf80096a0 tx=0x7fcaf8006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:27.937 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:27.934+0000 7fcb06ffd700 1 -- 172.21.15.37:0/2095697043 <== mgr.14114 v2:172.21.15.37:6800/2256954126 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7fcaf0032930 con 0x7fcaf002fdb0 2024-02-11T16:59:27.943 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:27.940+0000 7fcb14b5f700 1 -- 172.21.15.37:0/2095697043 --> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fcb10059100 con 0x7fcaf002fdb0 2024-02-11T16:59:27.943 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:27.940+0000 7fcb06ffd700 1 -- 172.21.15.37:0/2095697043 <== mgr.14114 v2:172.21.15.37:6800/2256954126 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fcb10059100 con 0x7fcaf002fdb0 2024-02-11T16:59:27.943 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: { 2024-02-11T16:59:27.943 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "mgrmap_epoch": 8, 2024-02-11T16:59:27.943 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "initialized": true 2024-02-11T16:59:27.943 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: } 2024-02-11T16:59:27.943 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:27.940+0000 7fcb14b5f700 1 -- 172.21.15.37:0/2095697043 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7fcaf002fdb0 msgr2=0x7fcaf0032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:27.944 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:27.940+0000 7fcb14b5f700 1 --2- 172.21.15.37:0/2095697043 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7fcaf002fdb0 0x7fcaf0032270 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fcaf80096a0 tx=0x7fcaf8006b40).stop 2024-02-11T16:59:27.944 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:27.940+0000 7fcb14b5f700 1 -- 172.21.15.37:0/2095697043 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcb100703f0 msgr2=0x7fcb1010cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:27.944 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:27.940+0000 7fcb14b5f700 1 --2- 172.21.15.37:0/2095697043 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcb100703f0 0x7fcb1010cbc0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7fcb0000b190 tx=0x7fcb00006ee0).stop 2024-02-11T16:59:27.944 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:27.940+0000 7fcb14b5f700 1 -- 172.21.15.37:0/2095697043 shutdown_connections 2024-02-11T16:59:27.944 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:27.940+0000 7fcb14b5f700 1 --2- 172.21.15.37:0/2095697043 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7fcaf002fdb0 0x7fcaf0032270 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:27.944 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:27.940+0000 7fcb14b5f700 1 --2- 172.21.15.37:0/2095697043 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcb100703f0 0x7fcb1010cbc0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:27.944 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:27.940+0000 7fcb14b5f700 1 -- 172.21.15.37:0/2095697043 >> 172.21.15.37:0/2095697043 conn(0x7fcb100f63c0 msgr2=0x7fcb100f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T16:59:27.944 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:27.940+0000 7fcb14b5f700 1 -- 172.21.15.37:0/2095697043 shutdown_connections 2024-02-11T16:59:27.944 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:27.940+0000 7fcb14b5f700 1 -- 172.21.15.37:0/2095697043 wait complete. 2024-02-11T16:59:28.567 INFO:teuthology.orchestra.run.smithi037.stderr:mgr epoch 7 is available 2024-02-11T16:59:28.567 INFO:teuthology.orchestra.run.smithi037.stderr:Setting orchestrator backend to cephadm... 2024-02-11T16:59:28.567 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi037 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpjftqjcr4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4ata0hlm:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch set backend cephadm 2024-02-11T16:59:29.130 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:29.125+0000 7fce85ecc700 1 Processor -- start 2024-02-11T16:59:29.130 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:29.125+0000 7fce85ecc700 1 -- start start 2024-02-11T16:59:29.130 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:29.125+0000 7fce85ecc700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce800fbb00 0x7fce800fbf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:29.130 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:29.125+0000 7fce85ecc700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce800fc410 con 0x7fce800fbb00 2024-02-11T16:59:29.130 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:29.125+0000 7fce84eca700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce800fbb00 0x7fce800fbf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:29.131 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:29.125+0000 7fce84eca700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce800fbb00 0x7fce800fbf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:40308/0 (socket says 172.21.15.37:40308) 2024-02-11T16:59:29.131 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:29.125+0000 7fce84eca700 1 -- 172.21.15.37:0/2953250909 learned_addr learned my addr 172.21.15.37:0/2953250909 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-02-11T16:59:29.131 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:29.126+0000 7fce84eca700 1 -- 172.21.15.37:0/2953250909 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce800fc550 con 0x7fce800fbb00 2024-02-11T16:59:29.131 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:29.126+0000 7fce84eca700 1 --2- 172.21.15.37:0/2953250909 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce800fbb00 0x7fce800fbf20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fce7400b5f0 tx=0x7fce74006ee0).ready entity=mon.0 client_cookie=db15c398776504cc server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:29.131 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:29.126+0000 7fce7f7fe700 1 -- 172.21.15.37:0/2953250909 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce7400d600 con 0x7fce800fbb00 2024-02-11T16:59:29.131 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:29.126+0000 7fce7f7fe700 1 -- 172.21.15.37:0/2953250909 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fce7400dbd0 con 0x7fce800fbb00 2024-02-11T16:59:29.131 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:29.126+0000 7fce7f7fe700 1 -- 172.21.15.37:0/2953250909 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce74011c00 con 0x7fce800fbb00 2024-02-11T16:59:29.131 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:29.127+0000 7fce85ecc700 1 -- 172.21.15.37:0/2953250909 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce800fbb00 msgr2=0x7fce800fbf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:29.131 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:29.127+0000 7fce85ecc700 1 --2- 172.21.15.37:0/2953250909 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce800fbb00 0x7fce800fbf20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fce7400b5f0 tx=0x7fce74006ee0).stop 2024-02-11T16:59:29.131 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:29.127+0000 7fce85ecc700 1 -- 172.21.15.37:0/2953250909 shutdown_connections 2024-02-11T16:59:29.131 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:29.127+0000 7fce85ecc700 1 --2- 172.21.15.37:0/2953250909 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce800fbb00 0x7fce800fbf20 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:29.131 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:29.127+0000 7fce85ecc700 1 -- 172.21.15.37:0/2953250909 >> 172.21.15.37:0/2953250909 conn(0x7fce800f7380 msgr2=0x7fce800f97a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T16:59:29.131 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:29.127+0000 7fce85ecc700 1 -- 172.21.15.37:0/2953250909 shutdown_connections 2024-02-11T16:59:29.131 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:29.127+0000 7fce85ecc700 1 -- 172.21.15.37:0/2953250909 wait complete. 2024-02-11T16:59:29.131 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:29.127+0000 7fce85ecc700 1 Processor -- start 2024-02-11T16:59:29.132 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:29.127+0000 7fce85ecc700 1 -- start start 2024-02-11T16:59:29.132 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:29.128+0000 7fce85ecc700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce800fbb00 0x7fce8010ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:29.132 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:29.128+0000 7fce85ecc700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce8010cf80 con 0x7fce800fbb00 2024-02-11T16:59:29.132 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:29.128+0000 7fce84eca700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce800fbb00 0x7fce8010ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:29.132 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:29.128+0000 7fce84eca700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce800fbb00 0x7fce8010ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:40320/0 (socket says 172.21.15.37:40320) 2024-02-11T16:59:29.132 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:29.128+0000 7fce84eca700 1 -- 172.21.15.37:0/3381440320 learned_addr learned my addr 172.21.15.37:0/3381440320 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-02-11T16:59:29.133 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:29.128+0000 7fce84eca700 1 -- 172.21.15.37:0/3381440320 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce7400b200 con 0x7fce800fbb00 2024-02-11T16:59:29.133 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:29.128+0000 7fce84eca700 1 --2- 172.21.15.37:0/3381440320 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce800fbb00 0x7fce8010ca90 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7fce7400e040 tx=0x7fce74006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:29.134 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:29.128+0000 7fce7d7fa700 1 -- 172.21.15.37:0/3381440320 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce7401b410 con 0x7fce800fbb00 2024-02-11T16:59:29.134 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:29.128+0000 7fce7d7fa700 1 -- 172.21.15.37:0/3381440320 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fce7401b9e0 con 0x7fce800fbb00 2024-02-11T16:59:29.134 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:29.128+0000 7fce85ecc700 1 -- 172.21.15.37:0/3381440320 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce8010d180 con 0x7fce800fbb00 2024-02-11T16:59:29.134 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:29.128+0000 7fce7d7fa700 1 -- 172.21.15.37:0/3381440320 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce7401a770 con 0x7fce800fbb00 2024-02-11T16:59:29.134 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:29.129+0000 7fce85ecc700 1 -- 172.21.15.37:0/3381440320 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce8010d590 con 0x7fce800fbb00 2024-02-11T16:59:29.134 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:29.129+0000 7fce7d7fa700 1 -- 172.21.15.37:0/3381440320 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fce7400f070 con 0x7fce800fbb00 2024-02-11T16:59:29.134 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:29.130+0000 7fce7d7fa700 1 --2- 172.21.15.37:0/3381440320 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7fce6402fd50 0x7fce64032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:29.134 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:29.129+0000 7fce85ecc700 1 -- 172.21.15.37:0/3381440320 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce80045ad0 con 0x7fce800fbb00 2024-02-11T16:59:29.134 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:29.130+0000 7fce7d7fa700 1 -- 172.21.15.37:0/3381440320 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fce74041020 con 0x7fce800fbb00 2024-02-11T16:59:29.134 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:29.130+0000 7fce7ffff700 1 --2- 172.21.15.37:0/3381440320 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7fce6402fd50 0x7fce64032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:29.134 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:29.130+0000 7fce7ffff700 1 --2- 172.21.15.37:0/3381440320 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7fce6402fd50 0x7fce64032210 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fce6c0096a0 tx=0x7fce6c006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:29.136 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:29.134+0000 7fce7d7fa700 1 -- 172.21.15.37:0/3381440320 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fce7406c2b0 con 0x7fce800fbb00 2024-02-11T16:59:29.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:28 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:27.935805+0000 mgr.smithi037.eofnaq (mgr.14114) 1 : audit [DBG] from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-11T16:59:29.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:28 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:27.941744+0000 mgr.smithi037.eofnaq (mgr.14114) 2 : audit [DBG] from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-11T16:59:29.324 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:29.322+0000 7fce85ecc700 1 -- 172.21.15.37:0/3381440320 --> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7fce8010de70 con 0x7fce6402fd50 2024-02-11T16:59:29.330 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:29.328+0000 7fce7d7fa700 1 -- 172.21.15.37:0/3381440320 <== mgr.14114 v2:172.21.15.37:6800/2256954126 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fce8010de70 con 0x7fce6402fd50 2024-02-11T16:59:29.334 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:29.331+0000 7fce85ecc700 1 -- 172.21.15.37:0/3381440320 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7fce6402fd50 msgr2=0x7fce64032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:29.334 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:29.331+0000 7fce85ecc700 1 --2- 172.21.15.37:0/3381440320 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7fce6402fd50 0x7fce64032210 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fce6c0096a0 tx=0x7fce6c006b40).stop 2024-02-11T16:59:29.334 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:29.331+0000 7fce85ecc700 1 -- 172.21.15.37:0/3381440320 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce800fbb00 msgr2=0x7fce8010ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:29.334 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:29.331+0000 7fce85ecc700 1 --2- 172.21.15.37:0/3381440320 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce800fbb00 0x7fce8010ca90 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7fce7400e040 tx=0x7fce74006ee0).stop 2024-02-11T16:59:29.334 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:29.331+0000 7fce85ecc700 1 -- 172.21.15.37:0/3381440320 shutdown_connections 2024-02-11T16:59:29.334 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:29.331+0000 7fce85ecc700 1 --2- 172.21.15.37:0/3381440320 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7fce6402fd50 0x7fce64032210 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:29.334 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:29.331+0000 7fce85ecc700 1 --2- 172.21.15.37:0/3381440320 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce800fbb00 0x7fce8010ca90 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:29.334 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:29.331+0000 7fce85ecc700 1 -- 172.21.15.37:0/3381440320 >> 172.21.15.37:0/3381440320 conn(0x7fce800f7380 msgr2=0x7fce800f8e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T16:59:29.334 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:29.332+0000 7fce85ecc700 1 -- 172.21.15.37:0/3381440320 shutdown_connections 2024-02-11T16:59:29.334 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:29.332+0000 7fce85ecc700 1 -- 172.21.15.37:0/3381440320 wait complete. 2024-02-11T16:59:29.964 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi037 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpjftqjcr4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4ata0hlm:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm set-user root 2024-02-11T16:59:30.555 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:30.548+0000 7f27ea445700 1 Processor -- start 2024-02-11T16:59:30.555 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:30.548+0000 7f27ea445700 1 -- start start 2024-02-11T16:59:30.555 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:30.548+0000 7f27ea445700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f27e40fbca0 0x7f27e40fc0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:30.555 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:30.548+0000 7f27ea445700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27e40fc5b0 con 0x7f27e40fbca0 2024-02-11T16:59:30.555 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:30.549+0000 7f27e3fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f27e40fbca0 0x7f27e40fc0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:30.555 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:30.549+0000 7f27e3fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f27e40fbca0 0x7f27e40fc0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:40332/0 (socket says 172.21.15.37:40332) 2024-02-11T16:59:30.556 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:30.549+0000 7f27e3fff700 1 -- 172.21.15.37:0/4240658539 learned_addr learned my addr 172.21.15.37:0/4240658539 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-02-11T16:59:30.556 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:30.549+0000 7f27e3fff700 1 -- 172.21.15.37:0/4240658539 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27e40fc6f0 con 0x7f27e40fbca0 2024-02-11T16:59:30.556 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:30.549+0000 7f27e3fff700 1 --2- 172.21.15.37:0/4240658539 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f27e40fbca0 0x7f27e40fc0c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f27d400b5f0 tx=0x7f27d4006ee0).ready entity=mon.0 client_cookie=e63ef5b4c5f221fc server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:30.556 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:30.549+0000 7f27e2ffd700 1 -- 172.21.15.37:0/4240658539 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f27d400d600 con 0x7f27e40fbca0 2024-02-11T16:59:30.556 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:30.549+0000 7f27e2ffd700 1 -- 172.21.15.37:0/4240658539 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f27d400dbd0 con 0x7f27e40fbca0 2024-02-11T16:59:30.556 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:30.549+0000 7f27e2ffd700 1 -- 172.21.15.37:0/4240658539 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f27d4011c00 con 0x7f27e40fbca0 2024-02-11T16:59:30.556 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:30.550+0000 7f27ea445700 1 -- 172.21.15.37:0/4240658539 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f27e40fbca0 msgr2=0x7f27e40fc0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:30.556 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:30.550+0000 7f27ea445700 1 --2- 172.21.15.37:0/4240658539 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f27e40fbca0 0x7f27e40fc0c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f27d400b5f0 tx=0x7f27d4006ee0).stop 2024-02-11T16:59:30.556 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:30.550+0000 7f27ea445700 1 -- 172.21.15.37:0/4240658539 shutdown_connections 2024-02-11T16:59:30.556 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:30.550+0000 7f27ea445700 1 --2- 172.21.15.37:0/4240658539 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f27e40fbca0 0x7f27e40fc0c0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:30.556 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:30.550+0000 7f27ea445700 1 -- 172.21.15.37:0/4240658539 >> 172.21.15.37:0/4240658539 conn(0x7f27e40f74c0 msgr2=0x7f27e40f98e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T16:59:30.556 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:30.550+0000 7f27ea445700 1 -- 172.21.15.37:0/4240658539 shutdown_connections 2024-02-11T16:59:30.556 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:30.550+0000 7f27ea445700 1 -- 172.21.15.37:0/4240658539 wait complete. 2024-02-11T16:59:30.557 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:30.550+0000 7f27ea445700 1 Processor -- start 2024-02-11T16:59:30.557 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:30.551+0000 7f27ea445700 1 -- start start 2024-02-11T16:59:30.557 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:30.551+0000 7f27ea445700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f27e40fbca0 0x7f27e4072570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:30.557 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:30.551+0000 7f27ea445700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27e40709f0 con 0x7f27e40fbca0 2024-02-11T16:59:30.557 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:30.551+0000 7f27e3fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f27e40fbca0 0x7f27e4072570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:30.557 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:30.551+0000 7f27e3fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f27e40fbca0 0x7f27e4072570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:40348/0 (socket says 172.21.15.37:40348) 2024-02-11T16:59:30.557 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:30.551+0000 7f27e3fff700 1 -- 172.21.15.37:0/1268150786 learned_addr learned my addr 172.21.15.37:0/1268150786 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-02-11T16:59:30.557 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:30.551+0000 7f27e3fff700 1 -- 172.21.15.37:0/1268150786 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27d400b200 con 0x7f27e40fbca0 2024-02-11T16:59:30.557 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:30.551+0000 7f27e3fff700 1 --2- 172.21.15.37:0/1268150786 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f27e40fbca0 0x7f27e4072570 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f27d400e040 tx=0x7f27d4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:30.557 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:30.552+0000 7f27e0ff9700 1 -- 172.21.15.37:0/1268150786 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f27d401b410 con 0x7f27e40fbca0 2024-02-11T16:59:30.557 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:30.552+0000 7f27ea445700 1 -- 172.21.15.37:0/1268150786 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27e4070b90 con 0x7f27e40fbca0 2024-02-11T16:59:30.557 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:30.552+0000 7f27e0ff9700 1 -- 172.21.15.37:0/1268150786 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f27d401b9e0 con 0x7f27e40fbca0 2024-02-11T16:59:30.557 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:30.552+0000 7f27e0ff9700 1 -- 172.21.15.37:0/1268150786 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f27d401a820 con 0x7f27e40fbca0 2024-02-11T16:59:30.557 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:30.552+0000 7f27ea445700 1 -- 172.21.15.37:0/1268150786 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27e4070fa0 con 0x7f27e40fbca0 2024-02-11T16:59:30.558 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:30.553+0000 7f27e0ff9700 1 -- 172.21.15.37:0/1268150786 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f27d400f070 con 0x7f27e40fbca0 2024-02-11T16:59:30.558 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:30.553+0000 7f27ea445700 1 -- 172.21.15.37:0/1268150786 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27e4045ad0 con 0x7f27e40fbca0 2024-02-11T16:59:30.558 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:30.553+0000 7f27e0ff9700 1 --2- 172.21.15.37:0/1268150786 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7f27d002fd50 0x7f27d0032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:30.558 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:30.553+0000 7f27e0ff9700 1 -- 172.21.15.37:0/1268150786 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f27d40413d0 con 0x7f27e40fbca0 2024-02-11T16:59:30.558 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:30.553+0000 7f27e37fe700 1 --2- 172.21.15.37:0/1268150786 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7f27d002fd50 0x7f27d0032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:30.558 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:30.553+0000 7f27e37fe700 1 --2- 172.21.15.37:0/1268150786 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7f27d002fd50 0x7f27d0032210 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f27d80096a0 tx=0x7f27d8006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:30.559 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:30.557+0000 7f27e0ff9700 1 -- 172.21.15.37:0/1268150786 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f27d4022b70 con 0x7f27e40fbca0 2024-02-11T16:59:30.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:30 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:29.324567+0000 mgr.smithi037.eofnaq (mgr.14114) 3 : audit [DBG] from='client.14124 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-02-11T16:59:30.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:30 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:29.329125+0000 mon.smithi037 (mon.0) 50 : audit [INF] from='mgr.14114 172.21.15.37:0/1469767164' entity='mgr.smithi037.eofnaq' 2024-02-11T16:59:30.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:30 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:29.333007+0000 mon.smithi037 (mon.0) 51 : audit [DBG] from='mgr.14114 172.21.15.37:0/1469767164' entity='mgr.smithi037.eofnaq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-11T16:59:30.720 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:30.718+0000 7f27ea445700 1 -- 172.21.15.37:0/1268150786 --> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f27e40fdb30 con 0x7f27d002fd50 2024-02-11T16:59:30.720 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:30.718+0000 7f27e0ff9700 1 -- 172.21.15.37:0/1268150786 <== mgr.14114 v2:172.21.15.37:6800/2256954126 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f27e40fdb30 con 0x7f27d002fd50 2024-02-11T16:59:30.721 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: value unchanged 2024-02-11T16:59:30.723 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:30.720+0000 7f27ea445700 1 -- 172.21.15.37:0/1268150786 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7f27d002fd50 msgr2=0x7f27d0032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:30.723 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:30.720+0000 7f27ea445700 1 --2- 172.21.15.37:0/1268150786 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7f27d002fd50 0x7f27d0032210 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f27d80096a0 tx=0x7f27d8006b40).stop 2024-02-11T16:59:30.723 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:30.720+0000 7f27ea445700 1 -- 172.21.15.37:0/1268150786 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f27e40fbca0 msgr2=0x7f27e4072570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:30.723 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:30.720+0000 7f27ea445700 1 --2- 172.21.15.37:0/1268150786 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f27e40fbca0 0x7f27e4072570 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f27d400e040 tx=0x7f27d4006ee0).stop 2024-02-11T16:59:30.723 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:30.720+0000 7f27ea445700 1 -- 172.21.15.37:0/1268150786 shutdown_connections 2024-02-11T16:59:30.723 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:30.720+0000 7f27ea445700 1 --2- 172.21.15.37:0/1268150786 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7f27d002fd50 0x7f27d0032210 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:30.723 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:30.720+0000 7f27ea445700 1 --2- 172.21.15.37:0/1268150786 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f27e40fbca0 0x7f27e4072570 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:30.723 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:30.720+0000 7f27ea445700 1 -- 172.21.15.37:0/1268150786 >> 172.21.15.37:0/1268150786 conn(0x7f27e40f74c0 msgr2=0x7f27e40f8fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T16:59:30.723 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:30.720+0000 7f27ea445700 1 -- 172.21.15.37:0/1268150786 shutdown_connections 2024-02-11T16:59:30.723 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:30.720+0000 7f27ea445700 1 -- 172.21.15.37:0/1268150786 wait complete. 2024-02-11T16:59:31.330 INFO:teuthology.orchestra.run.smithi037.stderr:Generating ssh key... 2024-02-11T16:59:31.330 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi037 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpjftqjcr4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4ata0hlm:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm generate-key 2024-02-11T16:59:31.579 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:31 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:30.720223+0000 mgr.smithi037.eofnaq (mgr.14114) 4 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-02-11T16:59:31.882 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:31.877+0000 7fe0bf39e700 1 Processor -- start 2024-02-11T16:59:31.882 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:31.877+0000 7fe0bf39e700 1 -- start start 2024-02-11T16:59:31.882 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:31.877+0000 7fe0bf39e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe0b80fa9d0 0x7fe0b80fadf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:31.882 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:31.877+0000 7fe0bf39e700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0b80fb2e0 con 0x7fe0b80fa9d0 2024-02-11T16:59:31.882 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:31.877+0000 7fe0be39c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe0b80fa9d0 0x7fe0b80fadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:31.882 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:31.877+0000 7fe0be39c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe0b80fa9d0 0x7fe0b80fadf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:40360/0 (socket says 172.21.15.37:40360) 2024-02-11T16:59:31.882 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:31.877+0000 7fe0be39c700 1 -- 172.21.15.37:0/2436013374 learned_addr learned my addr 172.21.15.37:0/2436013374 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-02-11T16:59:31.882 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:31.877+0000 7fe0be39c700 1 -- 172.21.15.37:0/2436013374 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0b80fb420 con 0x7fe0b80fa9d0 2024-02-11T16:59:31.882 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:31.878+0000 7fe0be39c700 1 --2- 172.21.15.37:0/2436013374 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe0b80fa9d0 0x7fe0b80fadf0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fe0b400b5f0 tx=0x7fe0b4006ee0).ready entity=mon.0 client_cookie=e02b7d668dc16b35 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:31.882 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:31.878+0000 7fe0bd39a700 1 -- 172.21.15.37:0/2436013374 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe0b400d600 con 0x7fe0b80fa9d0 2024-02-11T16:59:31.883 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:31.878+0000 7fe0bd39a700 1 -- 172.21.15.37:0/2436013374 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fe0b400dbd0 con 0x7fe0b80fa9d0 2024-02-11T16:59:31.883 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:31.878+0000 7fe0bd39a700 1 -- 172.21.15.37:0/2436013374 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe0b4011c00 con 0x7fe0b80fa9d0 2024-02-11T16:59:31.883 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:31.878+0000 7fe0bf39e700 1 -- 172.21.15.37:0/2436013374 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe0b80fa9d0 msgr2=0x7fe0b80fadf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:31.883 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:31.878+0000 7fe0bf39e700 1 --2- 172.21.15.37:0/2436013374 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe0b80fa9d0 0x7fe0b80fadf0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fe0b400b5f0 tx=0x7fe0b4006ee0).stop 2024-02-11T16:59:31.883 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:31.878+0000 7fe0bf39e700 1 -- 172.21.15.37:0/2436013374 shutdown_connections 2024-02-11T16:59:31.883 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:31.878+0000 7fe0bf39e700 1 --2- 172.21.15.37:0/2436013374 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe0b80fa9d0 0x7fe0b80fadf0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:31.883 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:31.878+0000 7fe0bf39e700 1 -- 172.21.15.37:0/2436013374 >> 172.21.15.37:0/2436013374 conn(0x7fe0b80f6210 msgr2=0x7fe0b80f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T16:59:31.883 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:31.878+0000 7fe0bf39e700 1 -- 172.21.15.37:0/2436013374 shutdown_connections 2024-02-11T16:59:31.883 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:31.879+0000 7fe0bf39e700 1 -- 172.21.15.37:0/2436013374 wait complete. 2024-02-11T16:59:31.883 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:31.879+0000 7fe0bf39e700 1 Processor -- start 2024-02-11T16:59:31.883 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:31.879+0000 7fe0bf39e700 1 -- start start 2024-02-11T16:59:31.883 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:31.879+0000 7fe0bf39e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe0b80fa9d0 0x7fe0b810a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:31.883 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:31.879+0000 7fe0bf39e700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0b810acf0 con 0x7fe0b80fa9d0 2024-02-11T16:59:31.883 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:31.879+0000 7fe0be39c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe0b80fa9d0 0x7fe0b810a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:31.883 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:31.879+0000 7fe0be39c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe0b80fa9d0 0x7fe0b810a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:40366/0 (socket says 172.21.15.37:40366) 2024-02-11T16:59:31.884 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:31.879+0000 7fe0be39c700 1 -- 172.21.15.37:0/1715942624 learned_addr learned my addr 172.21.15.37:0/1715942624 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-02-11T16:59:31.885 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:31.880+0000 7fe0be39c700 1 -- 172.21.15.37:0/1715942624 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0b400b200 con 0x7fe0b80fa9d0 2024-02-11T16:59:31.885 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:31.880+0000 7fe0be39c700 1 --2- 172.21.15.37:0/1715942624 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe0b80fa9d0 0x7fe0b810a800 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fe0b400b1d0 tx=0x7fe0b4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:31.885 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:31.880+0000 7fe0a6ffd700 1 -- 172.21.15.37:0/1715942624 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe0b401b410 con 0x7fe0b80fa9d0 2024-02-11T16:59:31.885 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:31.880+0000 7fe0a6ffd700 1 -- 172.21.15.37:0/1715942624 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fe0b401b9e0 con 0x7fe0b80fa9d0 2024-02-11T16:59:31.885 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:31.880+0000 7fe0a6ffd700 1 -- 172.21.15.37:0/1715942624 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe0b401a840 con 0x7fe0b80fa9d0 2024-02-11T16:59:31.885 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:31.880+0000 7fe0bf39e700 1 -- 172.21.15.37:0/1715942624 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0b810aef0 con 0x7fe0b80fa9d0 2024-02-11T16:59:31.885 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:31.880+0000 7fe0bf39e700 1 -- 172.21.15.37:0/1715942624 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0b810b300 con 0x7fe0b80fa9d0 2024-02-11T16:59:31.885 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:31.881+0000 7fe0a6ffd700 1 -- 172.21.15.37:0/1715942624 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fe0b400f070 con 0x7fe0b80fa9d0 2024-02-11T16:59:31.885 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:31.881+0000 7fe0bf39e700 1 -- 172.21.15.37:0/1715942624 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0b8045ad0 con 0x7fe0b80fa9d0 2024-02-11T16:59:31.885 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:31.881+0000 7fe0a6ffd700 1 --2- 172.21.15.37:0/1715942624 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7fe0a002fdb0 0x7fe0a0032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:31.885 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:31.881+0000 7fe0a6ffd700 1 -- 172.21.15.37:0/1715942624 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fe0b40411f0 con 0x7fe0b80fa9d0 2024-02-11T16:59:31.885 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:31.881+0000 7fe0bdb9b700 1 --2- 172.21.15.37:0/1715942624 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7fe0a002fdb0 0x7fe0a0032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:31.885 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:31.882+0000 7fe0bdb9b700 1 --2- 172.21.15.37:0/1715942624 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7fe0a002fdb0 0x7fe0a0032270 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fe0a80096a0 tx=0x7fe0a8006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:31.888 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:31.886+0000 7fe0a6ffd700 1 -- 172.21.15.37:0/1715942624 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe0b4019cd0 con 0x7fe0b80fa9d0 2024-02-11T16:59:32.048 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:32.046+0000 7fe0bf39e700 1 -- 172.21.15.37:0/1715942624 --> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fe0b810bbe0 con 0x7fe0a002fdb0 2024-02-11T16:59:32.231 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:32.228+0000 7fe0a6ffd700 1 -- 172.21.15.37:0/1715942624 <== mgr.14114 v2:172.21.15.37:6800/2256954126 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe0b810bbe0 con 0x7fe0a002fdb0 2024-02-11T16:59:32.233 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:32.230+0000 7fe0bf39e700 1 -- 172.21.15.37:0/1715942624 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7fe0a002fdb0 msgr2=0x7fe0a0032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:32.233 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:32.230+0000 7fe0bf39e700 1 --2- 172.21.15.37:0/1715942624 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7fe0a002fdb0 0x7fe0a0032270 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fe0a80096a0 tx=0x7fe0a8006b40).stop 2024-02-11T16:59:32.233 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:32.230+0000 7fe0bf39e700 1 -- 172.21.15.37:0/1715942624 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe0b80fa9d0 msgr2=0x7fe0b810a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:32.233 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:32.230+0000 7fe0bf39e700 1 --2- 172.21.15.37:0/1715942624 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe0b80fa9d0 0x7fe0b810a800 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fe0b400b1d0 tx=0x7fe0b4006ee0).stop 2024-02-11T16:59:32.233 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:32.230+0000 7fe0bf39e700 1 -- 172.21.15.37:0/1715942624 shutdown_connections 2024-02-11T16:59:32.233 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:32.230+0000 7fe0bf39e700 1 --2- 172.21.15.37:0/1715942624 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7fe0a002fdb0 0x7fe0a0032270 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:32.233 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:32.230+0000 7fe0bf39e700 1 --2- 172.21.15.37:0/1715942624 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe0b80fa9d0 0x7fe0b810a800 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:32.233 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:32.230+0000 7fe0bf39e700 1 -- 172.21.15.37:0/1715942624 >> 172.21.15.37:0/1715942624 conn(0x7fe0b80f6210 msgr2=0x7fe0b8070090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T16:59:32.233 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:32.231+0000 7fe0bf39e700 1 -- 172.21.15.37:0/1715942624 shutdown_connections 2024-02-11T16:59:32.234 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:32.231+0000 7fe0bf39e700 1 -- 172.21.15.37:0/1715942624 wait complete. 2024-02-11T16:59:33.103 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi037 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpjftqjcr4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4ata0hlm:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm get-pub-key 2024-02-11T16:59:33.679 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:33.674+0000 7fa34fe41700 1 Processor -- start 2024-02-11T16:59:33.679 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:33.675+0000 7fa34fe41700 1 -- start start 2024-02-11T16:59:33.680 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:33.675+0000 7fa34fe41700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa348070430 0x7fa3480fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:33.680 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:33.675+0000 7fa34fe41700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3480fedd0 con 0x7fa348070430 2024-02-11T16:59:33.680 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:33.675+0000 7fa34dbdd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa348070430 0x7fa3480fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:33.680 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:33.675+0000 7fa34dbdd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa348070430 0x7fa3480fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:40370/0 (socket says 172.21.15.37:40370) 2024-02-11T16:59:33.680 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:33.675+0000 7fa34dbdd700 1 -- 172.21.15.37:0/2384208556 learned_addr learned my addr 172.21.15.37:0/2384208556 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-02-11T16:59:33.680 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:33.675+0000 7fa34dbdd700 1 -- 172.21.15.37:0/2384208556 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3480fef10 con 0x7fa348070430 2024-02-11T16:59:33.680 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:33.676+0000 7fa34dbdd700 1 --2- 172.21.15.37:0/2384208556 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa348070430 0x7fa3480fe8e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fa34400b5f0 tx=0x7fa344006ee0).ready entity=mon.0 client_cookie=b73d8f74f8371ff8 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:33.680 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:33.676+0000 7fa34cbdb700 1 -- 172.21.15.37:0/2384208556 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa34400d600 con 0x7fa348070430 2024-02-11T16:59:33.680 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:33.676+0000 7fa34cbdb700 1 -- 172.21.15.37:0/2384208556 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fa34400dbd0 con 0x7fa348070430 2024-02-11T16:59:33.680 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:33.676+0000 7fa34cbdb700 1 -- 172.21.15.37:0/2384208556 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa344011c00 con 0x7fa348070430 2024-02-11T16:59:33.680 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:33.676+0000 7fa34fe41700 1 -- 172.21.15.37:0/2384208556 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa348070430 msgr2=0x7fa3480fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:33.680 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:33.676+0000 7fa34fe41700 1 --2- 172.21.15.37:0/2384208556 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa348070430 0x7fa3480fe8e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fa34400b5f0 tx=0x7fa344006ee0).stop 2024-02-11T16:59:33.681 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:33.676+0000 7fa34fe41700 1 -- 172.21.15.37:0/2384208556 shutdown_connections 2024-02-11T16:59:33.681 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:33.676+0000 7fa34fe41700 1 --2- 172.21.15.37:0/2384208556 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa348070430 0x7fa3480fe8e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:33.681 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:33.676+0000 7fa34fe41700 1 -- 172.21.15.37:0/2384208556 >> 172.21.15.37:0/2384208556 conn(0x7fa3480f63c0 msgr2=0x7fa3480f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T16:59:33.681 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:33.677+0000 7fa34fe41700 1 -- 172.21.15.37:0/2384208556 shutdown_connections 2024-02-11T16:59:33.681 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:33.677+0000 7fa34fe41700 1 -- 172.21.15.37:0/2384208556 wait complete. 2024-02-11T16:59:33.681 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:33.677+0000 7fa34fe41700 1 Processor -- start 2024-02-11T16:59:33.681 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:33.677+0000 7fa34fe41700 1 -- start start 2024-02-11T16:59:33.683 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:33.678+0000 7fa34fe41700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa348070430 0x7fa34810aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:33.683 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:33.678+0000 7fa34fe41700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa34810af30 con 0x7fa348070430 2024-02-11T16:59:33.683 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:33.678+0000 7fa34dbdd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa348070430 0x7fa34810aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:33.684 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:33.678+0000 7fa34dbdd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa348070430 0x7fa34810aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:40384/0 (socket says 172.21.15.37:40384) 2024-02-11T16:59:33.684 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:33.678+0000 7fa34dbdd700 1 -- 172.21.15.37:0/4015190890 learned_addr learned my addr 172.21.15.37:0/4015190890 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-02-11T16:59:33.684 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:33.678+0000 7fa34dbdd700 1 -- 172.21.15.37:0/4015190890 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa34400b200 con 0x7fa348070430 2024-02-11T16:59:33.684 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:33.678+0000 7fa34dbdd700 1 --2- 172.21.15.37:0/4015190890 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa348070430 0x7fa34810aa40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fa34400e040 tx=0x7fa344006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:33.684 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:33.678+0000 7fa3367fc700 1 -- 172.21.15.37:0/4015190890 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa34400d9f0 con 0x7fa348070430 2024-02-11T16:59:33.684 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:33.678+0000 7fa3367fc700 1 -- 172.21.15.37:0/4015190890 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fa34401a9f0 con 0x7fa348070430 2024-02-11T16:59:33.684 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:33.678+0000 7fa34fe41700 1 -- 172.21.15.37:0/4015190890 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa34810b130 con 0x7fa348070430 2024-02-11T16:59:33.684 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:33.678+0000 7fa3367fc700 1 -- 172.21.15.37:0/4015190890 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa34401b8c0 con 0x7fa348070430 2024-02-11T16:59:33.684 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:33.678+0000 7fa34fe41700 1 -- 172.21.15.37:0/4015190890 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa34810b540 con 0x7fa348070430 2024-02-11T16:59:33.684 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:33.679+0000 7fa34fe41700 1 -- 172.21.15.37:0/4015190890 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa348045ad0 con 0x7fa348070430 2024-02-11T16:59:33.684 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:33.679+0000 7fa3367fc700 1 -- 172.21.15.37:0/4015190890 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fa34400f070 con 0x7fa348070430 2024-02-11T16:59:33.685 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:33.680+0000 7fa3367fc700 1 --2- 172.21.15.37:0/4015190890 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7fa33002fdb0 0x7fa330032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:33.685 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:33.680+0000 7fa3367fc700 1 -- 172.21.15.37:0/4015190890 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fa344041330 con 0x7fa348070430 2024-02-11T16:59:33.685 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:33.680+0000 7fa34d3dc700 1 --2- 172.21.15.37:0/4015190890 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7fa33002fdb0 0x7fa330032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:33.685 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:33.680+0000 7fa34d3dc700 1 --2- 172.21.15.37:0/4015190890 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7fa33002fdb0 0x7fa330032270 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fa3380096a0 tx=0x7fa338006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:33.685 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:33 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cephadm 2024-02-11T16:59:32.048677+0000 mgr.smithi037.eofnaq (mgr.14114) 6 : cephadm [INF] Generating ssh key... 2024-02-11T16:59:33.685 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:33 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:32.227755+0000 mon.smithi037 (mon.0) 52 : audit [INF] from='mgr.14114 172.21.15.37:0/1469767164' entity='mgr.smithi037.eofnaq' 2024-02-11T16:59:33.685 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:33 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:32.229492+0000 mon.smithi037 (mon.0) 53 : audit [INF] from='mgr.14114 172.21.15.37:0/1469767164' entity='mgr.smithi037.eofnaq' 2024-02-11T16:59:33.686 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:33.684+0000 7fa3367fc700 1 -- 172.21.15.37:0/4015190890 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa344022310 con 0x7fa348070430 2024-02-11T16:59:33.849 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:33.846+0000 7fa34fe41700 1 -- 172.21.15.37:0/4015190890 --> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fa34810be80 con 0x7fa33002fdb0 2024-02-11T16:59:33.850 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:33.847+0000 7fa3367fc700 1 -- 172.21.15.37:0/4015190890 <== mgr.14114 v2:172.21.15.37:6800/2256954126 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7fa34810be80 con 0x7fa33002fdb0 2024-02-11T16:59:33.850 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC9xY3F4EhclTI591iTKYgp9F5DirExl+ULrjgVq4dbUkf1WQV+3aXnHVqpeMJofYEn092/MypoL7W/pD8nJTMB/+hAd//v7oYwctXuyi+tER0uMXObwwupp1efJ4+NoObioCjcvFAaRjxEoOk2vRhqZ+O/d2KQ5/KljMVXj9RreE+D3i2aKiSwFniA0l5Tr4ZhK/02LmncgkbOMPf/PeVvxvP6XQ6dwjyIZW6NHvMGDGQ/U3s/LBc0ramuH5HGh2C/UqF+kyND+5ZSwwWOpd8oOY9Wlef9yGgciilQWJVN6BiFZIoxZi4wG5XX4ohlnngTHMXqE+DDbNAXkIWizN1WO8d3kOk4DJD2lQYubjsX//EmKfJz2ZAajktj3P+cNahgJZGY0TL8/4+rXZL9xUqdfUBrrZIgLBY4urLdNhcj6nooBTIqBFpArim6eaGXe11K1gYfaGgi0zin3yh8pOu/AqkVYVFSuzSpYg7474+yvioQ8IJIb1XJVQ61Fz+Z1XE= ceph-c65cd888-c8fe-11ee-95b9-87774f69a715 2024-02-11T16:59:33.852 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:33.849+0000 7fa34fe41700 1 -- 172.21.15.37:0/4015190890 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7fa33002fdb0 msgr2=0x7fa330032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:33.853 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:33.849+0000 7fa34fe41700 1 --2- 172.21.15.37:0/4015190890 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7fa33002fdb0 0x7fa330032270 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fa3380096a0 tx=0x7fa338006b40).stop 2024-02-11T16:59:33.853 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:33.849+0000 7fa34fe41700 1 -- 172.21.15.37:0/4015190890 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa348070430 msgr2=0x7fa34810aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:33.853 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:33.849+0000 7fa34fe41700 1 --2- 172.21.15.37:0/4015190890 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa348070430 0x7fa34810aa40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fa34400e040 tx=0x7fa344006ee0).stop 2024-02-11T16:59:33.853 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:33.849+0000 7fa34fe41700 1 -- 172.21.15.37:0/4015190890 shutdown_connections 2024-02-11T16:59:33.853 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:33.849+0000 7fa34fe41700 1 --2- 172.21.15.37:0/4015190890 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7fa33002fdb0 0x7fa330032270 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:33.853 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:33.849+0000 7fa34fe41700 1 --2- 172.21.15.37:0/4015190890 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa348070430 0x7fa34810aa40 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:33.853 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:33.849+0000 7fa34fe41700 1 -- 172.21.15.37:0/4015190890 >> 172.21.15.37:0/4015190890 conn(0x7fa3480f63c0 msgr2=0x7fa3480f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T16:59:33.853 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:33.849+0000 7fa34fe41700 1 -- 172.21.15.37:0/4015190890 shutdown_connections 2024-02-11T16:59:33.853 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:33.849+0000 7fa34fe41700 1 -- 172.21.15.37:0/4015190890 wait complete. 2024-02-11T16:59:34.468 INFO:teuthology.orchestra.run.smithi037.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-02-11T16:59:34.468 INFO:teuthology.orchestra.run.smithi037.stderr:Adding key to root@localhost authorized_keys... 2024-02-11T16:59:34.468 INFO:teuthology.orchestra.run.smithi037.stderr:Adding host smithi037... 2024-02-11T16:59:34.469 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi037 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpjftqjcr4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4ata0hlm:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch host add smithi037 172.21.15.37 2024-02-11T16:59:34.470 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:34 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:33.848817+0000 mgr.smithi037.eofnaq (mgr.14114) 7 : audit [DBG] from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-11T16:59:34.997 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:34.990+0000 7f28e594c700 1 Processor -- start 2024-02-11T16:59:34.997 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:34.990+0000 7f28e594c700 1 -- start start 2024-02-11T16:59:34.997 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:34.990+0000 7f28e594c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f28e00703f0 0x7f28e00fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:34.997 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:34.991+0000 7f28e594c700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28e00fee10 con 0x7f28e00703f0 2024-02-11T16:59:34.997 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:34.991+0000 7f28deffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f28e00703f0 0x7f28e00fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:34.997 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:34.991+0000 7f28deffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f28e00703f0 0x7f28e00fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:40398/0 (socket says 172.21.15.37:40398) 2024-02-11T16:59:34.997 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:34.991+0000 7f28deffd700 1 -- 172.21.15.37:0/241553231 learned_addr learned my addr 172.21.15.37:0/241553231 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-02-11T16:59:34.997 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:34.991+0000 7f28deffd700 1 -- 172.21.15.37:0/241553231 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28e00fef50 con 0x7f28e00703f0 2024-02-11T16:59:34.997 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:34.991+0000 7f28deffd700 1 --2- 172.21.15.37:0/241553231 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f28e00703f0 0x7f28e00fe920 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f28d000b5f0 tx=0x7f28d0006ee0).ready entity=mon.0 client_cookie=87946f1025bfa13d server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:34.997 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:34.991+0000 7f28ddffb700 1 -- 172.21.15.37:0/241553231 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f28d000d600 con 0x7f28e00703f0 2024-02-11T16:59:34.997 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:34.991+0000 7f28ddffb700 1 -- 172.21.15.37:0/241553231 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f28d000dbd0 con 0x7f28e00703f0 2024-02-11T16:59:34.997 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:34.991+0000 7f28ddffb700 1 -- 172.21.15.37:0/241553231 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f28d0011c00 con 0x7f28e00703f0 2024-02-11T16:59:34.998 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:34.992+0000 7f28e594c700 1 -- 172.21.15.37:0/241553231 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f28e00703f0 msgr2=0x7f28e00fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:34.998 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:34.992+0000 7f28e594c700 1 --2- 172.21.15.37:0/241553231 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f28e00703f0 0x7f28e00fe920 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f28d000b5f0 tx=0x7f28d0006ee0).stop 2024-02-11T16:59:34.998 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:34.992+0000 7f28e594c700 1 -- 172.21.15.37:0/241553231 shutdown_connections 2024-02-11T16:59:34.998 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:34.992+0000 7f28e594c700 1 --2- 172.21.15.37:0/241553231 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f28e00703f0 0x7f28e00fe920 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:34.998 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:34.992+0000 7f28e594c700 1 -- 172.21.15.37:0/241553231 >> 172.21.15.37:0/241553231 conn(0x7f28e00f63c0 msgr2=0x7f28e00f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T16:59:34.998 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:34.992+0000 7f28e594c700 1 -- 172.21.15.37:0/241553231 shutdown_connections 2024-02-11T16:59:34.998 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:34.992+0000 7f28e594c700 1 -- 172.21.15.37:0/241553231 wait complete. 2024-02-11T16:59:34.998 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:34.993+0000 7f28e594c700 1 Processor -- start 2024-02-11T16:59:34.998 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:34.993+0000 7f28e594c700 1 -- start start 2024-02-11T16:59:34.998 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:34.993+0000 7f28e594c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f28e00703f0 0x7f28e010a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:34.998 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:34.993+0000 7f28e594c700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28e010aeb0 con 0x7f28e00703f0 2024-02-11T16:59:34.998 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:34.993+0000 7f28deffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f28e00703f0 0x7f28e010a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:34.998 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:34.993+0000 7f28deffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f28e00703f0 0x7f28e010a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:40400/0 (socket says 172.21.15.37:40400) 2024-02-11T16:59:34.998 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:34.993+0000 7f28deffd700 1 -- 172.21.15.37:0/2381800388 learned_addr learned my addr 172.21.15.37:0/2381800388 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-02-11T16:59:34.998 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:34.993+0000 7f28deffd700 1 -- 172.21.15.37:0/2381800388 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28d000b200 con 0x7f28e00703f0 2024-02-11T16:59:34.999 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:34.994+0000 7f28deffd700 1 --2- 172.21.15.37:0/2381800388 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f28e00703f0 0x7f28e010a9c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f28d000b190 tx=0x7f28d0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:34.999 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:34.994+0000 7f28c77fe700 1 -- 172.21.15.37:0/2381800388 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f28d001b410 con 0x7f28e00703f0 2024-02-11T16:59:34.999 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:34.994+0000 7f28c77fe700 1 -- 172.21.15.37:0/2381800388 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f28d001b9e0 con 0x7f28e00703f0 2024-02-11T16:59:34.999 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:34.994+0000 7f28e594c700 1 -- 172.21.15.37:0/2381800388 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28e010b0b0 con 0x7f28e00703f0 2024-02-11T16:59:34.999 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:34.994+0000 7f28c77fe700 1 -- 172.21.15.37:0/2381800388 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f28d001a880 con 0x7f28e00703f0 2024-02-11T16:59:34.999 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:34.994+0000 7f28e594c700 1 -- 172.21.15.37:0/2381800388 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28e010b4c0 con 0x7f28e00703f0 2024-02-11T16:59:35.000 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:34.995+0000 7f28c77fe700 1 -- 172.21.15.37:0/2381800388 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f28d000f070 con 0x7f28e00703f0 2024-02-11T16:59:35.000 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:34.995+0000 7f28e594c700 1 -- 172.21.15.37:0/2381800388 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28e0045ad0 con 0x7f28e00703f0 2024-02-11T16:59:35.000 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:34.995+0000 7f28c77fe700 1 --2- 172.21.15.37:0/2381800388 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7f28cc02fd50 0x7f28cc032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:35.000 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:34.995+0000 7f28c77fe700 1 -- 172.21.15.37:0/2381800388 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f28d0041290 con 0x7f28e00703f0 2024-02-11T16:59:35.000 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:34.995+0000 7f28de7fc700 1 --2- 172.21.15.37:0/2381800388 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7f28cc02fd50 0x7f28cc032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:35.000 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:34.996+0000 7f28de7fc700 1 --2- 172.21.15.37:0/2381800388 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7f28cc02fd50 0x7f28cc032210 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f28d40096a0 tx=0x7f28d4006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:35.001 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:34.999+0000 7f28c77fe700 1 -- 172.21.15.37:0/2381800388 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f28d0019e60 con 0x7f28e00703f0 2024-02-11T16:59:35.182 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:35.180+0000 7f28e594c700 1 -- 172.21.15.37:0/2381800388 --> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi037", "addr": "172.21.15.37", "target": ["mon-mgr", ""]}) v1 -- 0x7f28e010bda0 con 0x7f28cc02fd50 2024-02-11T16:59:35.781 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:35.778+0000 7f28c77fe700 1 -- 172.21.15.37:0/2381800388 <== mgr.14114 v2:172.21.15.37:6800/2256954126 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7f28e010bda0 con 0x7f28cc02fd50 2024-02-11T16:59:35.781 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: Added host 'smithi037' 2024-02-11T16:59:35.782 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:35.780+0000 7f28e594c700 1 -- 172.21.15.37:0/2381800388 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7f28cc02fd50 msgr2=0x7f28cc032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:35.782 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:35.780+0000 7f28e594c700 1 --2- 172.21.15.37:0/2381800388 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7f28cc02fd50 0x7f28cc032210 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f28d40096a0 tx=0x7f28d4006b40).stop 2024-02-11T16:59:35.783 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:35.780+0000 7f28e594c700 1 -- 172.21.15.37:0/2381800388 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f28e00703f0 msgr2=0x7f28e010a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:35.783 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:35.780+0000 7f28e594c700 1 --2- 172.21.15.37:0/2381800388 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f28e00703f0 0x7f28e010a9c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f28d000b190 tx=0x7f28d0006ee0).stop 2024-02-11T16:59:35.784 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:35.780+0000 7f28e594c700 1 -- 172.21.15.37:0/2381800388 shutdown_connections 2024-02-11T16:59:35.784 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:35.780+0000 7f28e594c700 1 --2- 172.21.15.37:0/2381800388 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7f28cc02fd50 0x7f28cc032210 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:35.784 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:35.780+0000 7f28e594c700 1 --2- 172.21.15.37:0/2381800388 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f28e00703f0 0x7f28e010a9c0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:35.784 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:35.780+0000 7f28e594c700 1 -- 172.21.15.37:0/2381800388 >> 172.21.15.37:0/2381800388 conn(0x7f28e00f63c0 msgr2=0x7f28e00f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T16:59:35.784 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:35.780+0000 7f28e594c700 1 -- 172.21.15.37:0/2381800388 shutdown_connections 2024-02-11T16:59:35.784 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:35.781+0000 7f28e594c700 1 -- 172.21.15.37:0/2381800388 wait complete. 2024-02-11T16:59:36.407 INFO:teuthology.orchestra.run.smithi037.stderr:Deploying mon service with default placement... 2024-02-11T16:59:36.407 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi037 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpjftqjcr4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4ata0hlm:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mon 2024-02-11T16:59:36.934 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:36.928+0000 7f806c55f700 1 Processor -- start 2024-02-11T16:59:36.934 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:36.928+0000 7f806c55f700 1 -- start start 2024-02-11T16:59:36.934 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:36.928+0000 7f806c55f700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f80640703d0 0x7f8064100d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:36.934 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:36.928+0000 7f806c55f700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80640fc730 con 0x7f80640703d0 2024-02-11T16:59:36.934 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:36.929+0000 7f806a2fb700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f80640703d0 0x7f8064100d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:36.934 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:36.929+0000 7f806a2fb700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f80640703d0 0x7f8064100d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:40410/0 (socket says 172.21.15.37:40410) 2024-02-11T16:59:36.934 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:36.929+0000 7f806a2fb700 1 -- 172.21.15.37:0/555432378 learned_addr learned my addr 172.21.15.37:0/555432378 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-02-11T16:59:36.934 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:36.929+0000 7f806a2fb700 1 -- 172.21.15.37:0/555432378 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8064101230 con 0x7f80640703d0 2024-02-11T16:59:36.934 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:36.929+0000 7f806a2fb700 1 --2- 172.21.15.37:0/555432378 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f80640703d0 0x7f8064100d40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f806000b5f0 tx=0x7f8060006ee0).ready entity=mon.0 client_cookie=d4b3ef308f6c6030 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:36.935 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:36.929+0000 7f80692f9700 1 -- 172.21.15.37:0/555432378 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f806000d600 con 0x7f80640703d0 2024-02-11T16:59:36.935 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:36.929+0000 7f80692f9700 1 -- 172.21.15.37:0/555432378 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f806000dbd0 con 0x7f80640703d0 2024-02-11T16:59:36.935 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:36.929+0000 7f80692f9700 1 -- 172.21.15.37:0/555432378 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8060011c00 con 0x7f80640703d0 2024-02-11T16:59:36.935 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:36.930+0000 7f806c55f700 1 -- 172.21.15.37:0/555432378 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f80640703d0 msgr2=0x7f8064100d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:36.935 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:36.930+0000 7f806c55f700 1 --2- 172.21.15.37:0/555432378 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f80640703d0 0x7f8064100d40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f806000b5f0 tx=0x7f8060006ee0).stop 2024-02-11T16:59:36.935 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:36.930+0000 7f806c55f700 1 -- 172.21.15.37:0/555432378 shutdown_connections 2024-02-11T16:59:36.935 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:36.930+0000 7f806c55f700 1 --2- 172.21.15.37:0/555432378 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f80640703d0 0x7f8064100d40 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:36.935 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:36.930+0000 7f806c55f700 1 -- 172.21.15.37:0/555432378 >> 172.21.15.37:0/555432378 conn(0x7f80640f63c0 msgr2=0x7f80640f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T16:59:36.935 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:36.930+0000 7f806c55f700 1 -- 172.21.15.37:0/555432378 shutdown_connections 2024-02-11T16:59:36.935 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:36.930+0000 7f806c55f700 1 -- 172.21.15.37:0/555432378 wait complete. 2024-02-11T16:59:36.935 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:36.930+0000 7f806c55f700 1 Processor -- start 2024-02-11T16:59:36.935 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:36.930+0000 7f806c55f700 1 -- start start 2024-02-11T16:59:36.935 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:36.931+0000 7f806c55f700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f80640703d0 0x7f806410cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:36.935 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:36.931+0000 7f806c55f700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f806410d0a0 con 0x7f80640703d0 2024-02-11T16:59:36.936 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:36.931+0000 7f806a2fb700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f80640703d0 0x7f806410cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:36.936 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:36.931+0000 7f806a2fb700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f80640703d0 0x7f806410cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:40412/0 (socket says 172.21.15.37:40412) 2024-02-11T16:59:36.936 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:36.931+0000 7f806a2fb700 1 -- 172.21.15.37:0/2487251811 learned_addr learned my addr 172.21.15.37:0/2487251811 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-02-11T16:59:36.936 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:36.931+0000 7f806a2fb700 1 -- 172.21.15.37:0/2487251811 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f806000b200 con 0x7f80640703d0 2024-02-11T16:59:36.936 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:36.931+0000 7f806a2fb700 1 --2- 172.21.15.37:0/2487251811 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f80640703d0 0x7f806410cbb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f806000b190 tx=0x7f8060006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:36.936 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:36.931+0000 7f8052ffd700 1 -- 172.21.15.37:0/2487251811 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f806001b410 con 0x7f80640703d0 2024-02-11T16:59:36.936 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:36.931+0000 7f8052ffd700 1 -- 172.21.15.37:0/2487251811 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f806001b9e0 con 0x7f80640703d0 2024-02-11T16:59:36.936 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:36.931+0000 7f806c55f700 1 -- 172.21.15.37:0/2487251811 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f806410d2a0 con 0x7f80640703d0 2024-02-11T16:59:36.936 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:36.931+0000 7f8052ffd700 1 -- 172.21.15.37:0/2487251811 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f806001a880 con 0x7f80640703d0 2024-02-11T16:59:36.936 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:36.931+0000 7f806c55f700 1 -- 172.21.15.37:0/2487251811 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f806410d6b0 con 0x7f80640703d0 2024-02-11T16:59:36.936 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:36.932+0000 7f8052ffd700 1 -- 172.21.15.37:0/2487251811 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f806000f070 con 0x7f80640703d0 2024-02-11T16:59:36.936 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:36.932+0000 7f8052ffd700 1 --2- 172.21.15.37:0/2487251811 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7f804c02fdb0 0x7f804c032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:36.936 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:36.932+0000 7f806c55f700 1 -- 172.21.15.37:0/2487251811 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8064045ad0 con 0x7f80640703d0 2024-02-11T16:59:36.936 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:36.932+0000 7f8052ffd700 1 -- 172.21.15.37:0/2487251811 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f80600411d0 con 0x7f80640703d0 2024-02-11T16:59:36.937 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:36.932+0000 7f8069afa700 1 --2- 172.21.15.37:0/2487251811 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7f804c02fdb0 0x7f804c032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:36.937 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:36.934+0000 7f8069afa700 1 --2- 172.21.15.37:0/2487251811 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7f804c02fdb0 0x7f804c032270 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f80540096a0 tx=0x7f8054006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:36.938 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:36.937+0000 7f8052ffd700 1 -- 172.21.15.37:0/2487251811 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8060016070 con 0x7f80640703d0 2024-02-11T16:59:37.123 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:37.122+0000 7f806c55f700 1 -- 172.21.15.37:0/2487251811 --> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f806410df90 con 0x7f804c02fdb0 2024-02-11T16:59:37.128 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:37.126+0000 7f8052ffd700 1 -- 172.21.15.37:0/2487251811 <== mgr.14114 v2:172.21.15.37:6800/2256954126 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f806410df90 con 0x7f804c02fdb0 2024-02-11T16:59:37.128 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: Scheduled mon update... 2024-02-11T16:59:37.130 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:37.128+0000 7f806c55f700 1 -- 172.21.15.37:0/2487251811 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7f804c02fdb0 msgr2=0x7f804c032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:37.130 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:37.128+0000 7f806c55f700 1 --2- 172.21.15.37:0/2487251811 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7f804c02fdb0 0x7f804c032270 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f80540096a0 tx=0x7f8054006b40).stop 2024-02-11T16:59:37.130 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:37.128+0000 7f806c55f700 1 -- 172.21.15.37:0/2487251811 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f80640703d0 msgr2=0x7f806410cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:37.130 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:37.128+0000 7f806c55f700 1 --2- 172.21.15.37:0/2487251811 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f80640703d0 0x7f806410cbb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f806000b190 tx=0x7f8060006ee0).stop 2024-02-11T16:59:37.131 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:37.128+0000 7f806c55f700 1 -- 172.21.15.37:0/2487251811 shutdown_connections 2024-02-11T16:59:37.131 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:37.128+0000 7f806c55f700 1 --2- 172.21.15.37:0/2487251811 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7f804c02fdb0 0x7f804c032270 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:37.131 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:37.128+0000 7f806c55f700 1 --2- 172.21.15.37:0/2487251811 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f80640703d0 0x7f806410cbb0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:37.131 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:37.128+0000 7f806c55f700 1 -- 172.21.15.37:0/2487251811 >> 172.21.15.37:0/2487251811 conn(0x7f80640f63c0 msgr2=0x7f80640f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T16:59:37.131 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:37.128+0000 7f806c55f700 1 -- 172.21.15.37:0/2487251811 shutdown_connections 2024-02-11T16:59:37.131 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:37.128+0000 7f806c55f700 1 -- 172.21.15.37:0/2487251811 wait complete. 2024-02-11T16:59:37.150 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:36 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:35.182497+0000 mgr.smithi037.eofnaq (mgr.14114) 8 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi037", "addr": "172.21.15.37", "target": ["mon-mgr", ""]}]: dispatch 2024-02-11T16:59:37.150 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:36 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:35.779061+0000 mon.smithi037 (mon.0) 54 : audit [INF] from='mgr.14114 172.21.15.37:0/1469767164' entity='mgr.smithi037.eofnaq' 2024-02-11T16:59:37.150 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:36 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cephadm 2024-02-11T16:59:35.779369+0000 mgr.smithi037.eofnaq (mgr.14114) 9 : cephadm [INF] Added host smithi037 2024-02-11T16:59:37.150 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:36 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:35.779615+0000 mon.smithi037 (mon.0) 55 : audit [DBG] from='mgr.14114 172.21.15.37:0/1469767164' entity='mgr.smithi037.eofnaq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-11T16:59:37.740 INFO:teuthology.orchestra.run.smithi037.stderr:Deploying mgr service with default placement... 2024-02-11T16:59:37.740 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi037 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpjftqjcr4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4ata0hlm:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mgr 2024-02-11T16:59:38.331 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:38.328+0000 7f7726365700 1 Processor -- start 2024-02-11T16:59:38.331 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:38.328+0000 7f7726365700 1 -- start start 2024-02-11T16:59:38.331 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:38.328+0000 7f7726365700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f77200703b0 0x7f7720100bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:38.331 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:38.328+0000 7f7726365700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77200fc5a0 con 0x7f77200703b0 2024-02-11T16:59:38.331 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:38.328+0000 7f7725363700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f77200703b0 0x7f7720100bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:38.331 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:38.328+0000 7f7725363700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f77200703b0 0x7f7720100bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:40428/0 (socket says 172.21.15.37:40428) 2024-02-11T16:59:38.331 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:38.328+0000 7f7725363700 1 -- 172.21.15.37:0/2318169366 learned_addr learned my addr 172.21.15.37:0/2318169366 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-02-11T16:59:38.331 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:38.328+0000 7f7725363700 1 -- 172.21.15.37:0/2318169366 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77201010a0 con 0x7f77200703b0 2024-02-11T16:59:38.331 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:38.329+0000 7f7725363700 1 --2- 172.21.15.37:0/2318169366 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f77200703b0 0x7f7720100bb0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f771c00b5f0 tx=0x7f771c006ee0).ready entity=mon.0 client_cookie=ac6883ba51a14173 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:38.331 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:38.329+0000 7f770ffff700 1 -- 172.21.15.37:0/2318169366 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f771c00d600 con 0x7f77200703b0 2024-02-11T16:59:38.331 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:38.329+0000 7f770ffff700 1 -- 172.21.15.37:0/2318169366 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f771c00dbd0 con 0x7f77200703b0 2024-02-11T16:59:38.332 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:38.329+0000 7f770ffff700 1 -- 172.21.15.37:0/2318169366 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f771c011c00 con 0x7f77200703b0 2024-02-11T16:59:38.332 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:38.329+0000 7f7726365700 1 -- 172.21.15.37:0/2318169366 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f77200703b0 msgr2=0x7f7720100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:38.332 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:38.329+0000 7f7726365700 1 --2- 172.21.15.37:0/2318169366 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f77200703b0 0x7f7720100bb0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f771c00b5f0 tx=0x7f771c006ee0).stop 2024-02-11T16:59:38.332 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:38.329+0000 7f7726365700 1 -- 172.21.15.37:0/2318169366 shutdown_connections 2024-02-11T16:59:38.332 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:38.329+0000 7f7726365700 1 --2- 172.21.15.37:0/2318169366 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f77200703b0 0x7f7720100bb0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:38.333 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:38.329+0000 7f7726365700 1 -- 172.21.15.37:0/2318169366 >> 172.21.15.37:0/2318169366 conn(0x7f77200f6210 msgr2=0x7f77200f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T16:59:38.334 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:38.329+0000 7f7726365700 1 -- 172.21.15.37:0/2318169366 shutdown_connections 2024-02-11T16:59:38.334 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:38.330+0000 7f7726365700 1 -- 172.21.15.37:0/2318169366 wait complete. 2024-02-11T16:59:38.334 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:38.330+0000 7f7726365700 1 Processor -- start 2024-02-11T16:59:38.334 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:38.330+0000 7f7726365700 1 -- start start 2024-02-11T16:59:38.334 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:38.330+0000 7f7726365700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f77200703b0 0x7f772010ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:38.334 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:38.330+0000 7f7726365700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f772010cf20 con 0x7f77200703b0 2024-02-11T16:59:38.334 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:38.330+0000 7f7725363700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f77200703b0 0x7f772010ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:38.334 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:38.330+0000 7f7725363700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f77200703b0 0x7f772010ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:40436/0 (socket says 172.21.15.37:40436) 2024-02-11T16:59:38.334 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:38.330+0000 7f7725363700 1 -- 172.21.15.37:0/3027626922 learned_addr learned my addr 172.21.15.37:0/3027626922 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-02-11T16:59:38.334 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:38.331+0000 7f7725363700 1 -- 172.21.15.37:0/3027626922 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f771c00b200 con 0x7f77200703b0 2024-02-11T16:59:38.334 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:38.331+0000 7f7725363700 1 --2- 172.21.15.37:0/3027626922 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f77200703b0 0x7f772010ca30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f771c00b190 tx=0x7f771c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:38.334 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:38.331+0000 7f770dffb700 1 -- 172.21.15.37:0/3027626922 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f771c01b410 con 0x7f77200703b0 2024-02-11T16:59:38.334 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:38.331+0000 7f770dffb700 1 -- 172.21.15.37:0/3027626922 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f771c01b9e0 con 0x7f77200703b0 2024-02-11T16:59:38.335 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:38.331+0000 7f7726365700 1 -- 172.21.15.37:0/3027626922 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f772010d120 con 0x7f77200703b0 2024-02-11T16:59:38.335 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:38.331+0000 7f770dffb700 1 -- 172.21.15.37:0/3027626922 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f771c01a880 con 0x7f77200703b0 2024-02-11T16:59:38.335 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:38.331+0000 7f7726365700 1 -- 172.21.15.37:0/3027626922 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f772010d530 con 0x7f77200703b0 2024-02-11T16:59:38.335 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:38.332+0000 7f770dffb700 1 -- 172.21.15.37:0/3027626922 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f771c00f070 con 0x7f77200703b0 2024-02-11T16:59:38.335 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:38.332+0000 7f770dffb700 1 --2- 172.21.15.37:0/3027626922 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7f770802fd50 0x7f7708032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:38.335 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:38.332+0000 7f7726365700 1 -- 172.21.15.37:0/3027626922 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7720045ad0 con 0x7f77200703b0 2024-02-11T16:59:38.335 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:38.332+0000 7f770dffb700 1 -- 172.21.15.37:0/3027626922 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f771c0413a0 con 0x7f77200703b0 2024-02-11T16:59:38.335 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:38.332+0000 7f7724b62700 1 --2- 172.21.15.37:0/3027626922 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7f770802fd50 0x7f7708032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:38.335 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:38.333+0000 7f7724b62700 1 --2- 172.21.15.37:0/3027626922 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7f770802fd50 0x7f7708032210 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f77100096a0 tx=0x7f7710006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:38.339 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:38.337+0000 7f770dffb700 1 -- 172.21.15.37:0/3027626922 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f771c022b00 con 0x7f77200703b0 2024-02-11T16:59:38.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:38 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:37.123426+0000 mgr.smithi037.eofnaq (mgr.14114) 10 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-02-11T16:59:38.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:38 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cephadm 2024-02-11T16:59:37.124172+0000 mgr.smithi037.eofnaq (mgr.14114) 11 : cephadm [INF] Saving service mon spec with placement count:5 2024-02-11T16:59:38.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:38 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:37.127164+0000 mon.smithi037 (mon.0) 56 : audit [INF] from='mgr.14114 172.21.15.37:0/1469767164' entity='mgr.smithi037.eofnaq' 2024-02-11T16:59:38.523 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:38.522+0000 7f7726365700 1 -- 172.21.15.37:0/3027626922 --> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f772010de70 con 0x7f770802fd50 2024-02-11T16:59:38.527 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:38.526+0000 7f770dffb700 1 -- 172.21.15.37:0/3027626922 <== mgr.14114 v2:172.21.15.37:6800/2256954126 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f772010de70 con 0x7f770802fd50 2024-02-11T16:59:38.527 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-02-11T16:59:38.530 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:38.528+0000 7f7726365700 1 -- 172.21.15.37:0/3027626922 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7f770802fd50 msgr2=0x7f7708032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:38.530 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:38.528+0000 7f7726365700 1 --2- 172.21.15.37:0/3027626922 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7f770802fd50 0x7f7708032210 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f77100096a0 tx=0x7f7710006b40).stop 2024-02-11T16:59:38.530 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:38.528+0000 7f7726365700 1 -- 172.21.15.37:0/3027626922 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f77200703b0 msgr2=0x7f772010ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:38.530 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:38.528+0000 7f7726365700 1 --2- 172.21.15.37:0/3027626922 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f77200703b0 0x7f772010ca30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f771c00b190 tx=0x7f771c006ee0).stop 2024-02-11T16:59:38.530 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:38.528+0000 7f7726365700 1 -- 172.21.15.37:0/3027626922 shutdown_connections 2024-02-11T16:59:38.530 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:38.528+0000 7f7726365700 1 --2- 172.21.15.37:0/3027626922 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7f770802fd50 0x7f7708032210 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:38.530 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:38.528+0000 7f7726365700 1 --2- 172.21.15.37:0/3027626922 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f77200703b0 0x7f772010ca30 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:38.531 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:38.528+0000 7f7726365700 1 -- 172.21.15.37:0/3027626922 >> 172.21.15.37:0/3027626922 conn(0x7f77200f6210 msgr2=0x7f77200f7d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T16:59:38.531 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:38.528+0000 7f7726365700 1 -- 172.21.15.37:0/3027626922 shutdown_connections 2024-02-11T16:59:38.531 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:38.528+0000 7f7726365700 1 -- 172.21.15.37:0/3027626922 wait complete. 2024-02-11T16:59:39.087 INFO:teuthology.orchestra.run.smithi037.stderr:Deploying crash service with default placement... 2024-02-11T16:59:39.087 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi037 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpjftqjcr4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4ata0hlm:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply crash 2024-02-11T16:59:39.650 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:39.638+0000 7f29ad0b9700 1 Processor -- start 2024-02-11T16:59:39.650 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:39.638+0000 7f29ad0b9700 1 -- start start 2024-02-11T16:59:39.650 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:39.638+0000 7f29ad0b9700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f29a8070390 0x7f29a8100bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:39.651 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:39.638+0000 7f29ad0b9700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29a80fc5c0 con 0x7f29a8070390 2024-02-11T16:59:39.651 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:39.638+0000 7f29a7fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f29a8070390 0x7f29a8100bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:39.651 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:39.638+0000 7f29a7fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f29a8070390 0x7f29a8100bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:60690/0 (socket says 172.21.15.37:60690) 2024-02-11T16:59:39.651 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:39.638+0000 7f29a7fff700 1 -- 172.21.15.37:0/2087273878 learned_addr learned my addr 172.21.15.37:0/2087273878 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-02-11T16:59:39.651 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:39.639+0000 7f29a7fff700 1 -- 172.21.15.37:0/2087273878 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29a81010c0 con 0x7f29a8070390 2024-02-11T16:59:39.651 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:39.639+0000 7f29a7fff700 1 --2- 172.21.15.37:0/2087273878 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f29a8070390 0x7f29a8100bd0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f299800b5f0 tx=0x7f2998006ee0).ready entity=mon.0 client_cookie=9b030021d7f1da60 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:39.651 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:39.639+0000 7f29a6ffd700 1 -- 172.21.15.37:0/2087273878 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f299800d600 con 0x7f29a8070390 2024-02-11T16:59:39.651 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:39.639+0000 7f29a6ffd700 1 -- 172.21.15.37:0/2087273878 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f299800dbd0 con 0x7f29a8070390 2024-02-11T16:59:39.651 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:39.639+0000 7f29a6ffd700 1 -- 172.21.15.37:0/2087273878 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2998011c00 con 0x7f29a8070390 2024-02-11T16:59:39.651 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:39.639+0000 7f29ad0b9700 1 -- 172.21.15.37:0/2087273878 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f29a8070390 msgr2=0x7f29a8100bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:39.651 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:39.639+0000 7f29ad0b9700 1 --2- 172.21.15.37:0/2087273878 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f29a8070390 0x7f29a8100bd0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f299800b5f0 tx=0x7f2998006ee0).stop 2024-02-11T16:59:39.651 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:39.639+0000 7f29ad0b9700 1 -- 172.21.15.37:0/2087273878 shutdown_connections 2024-02-11T16:59:39.652 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:39.639+0000 7f29ad0b9700 1 --2- 172.21.15.37:0/2087273878 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f29a8070390 0x7f29a8100bd0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:39.652 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:39.639+0000 7f29ad0b9700 1 -- 172.21.15.37:0/2087273878 >> 172.21.15.37:0/2087273878 conn(0x7f29a80f6210 msgr2=0x7f29a80f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T16:59:39.652 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:39.640+0000 7f29ad0b9700 1 -- 172.21.15.37:0/2087273878 shutdown_connections 2024-02-11T16:59:39.652 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:39.640+0000 7f29ad0b9700 1 -- 172.21.15.37:0/2087273878 wait complete. 2024-02-11T16:59:39.652 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:39.640+0000 7f29ad0b9700 1 Processor -- start 2024-02-11T16:59:39.652 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:39.640+0000 7f29ad0b9700 1 -- start start 2024-02-11T16:59:39.652 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:39.640+0000 7f29ad0b9700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f29a8070390 0x7f29a810ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:39.652 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:39.640+0000 7f29ad0b9700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29a810cf10 con 0x7f29a8070390 2024-02-11T16:59:39.652 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:39.640+0000 7f29a7fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f29a8070390 0x7f29a810ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:39.652 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:39.640+0000 7f29a7fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f29a8070390 0x7f29a810ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:60702/0 (socket says 172.21.15.37:60702) 2024-02-11T16:59:39.652 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:39.640+0000 7f29a7fff700 1 -- 172.21.15.37:0/2375854420 learned_addr learned my addr 172.21.15.37:0/2375854420 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-02-11T16:59:39.652 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:39.641+0000 7f29a7fff700 1 -- 172.21.15.37:0/2375854420 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f299800b200 con 0x7f29a8070390 2024-02-11T16:59:39.653 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:39.641+0000 7f29a7fff700 1 --2- 172.21.15.37:0/2375854420 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f29a8070390 0x7f29a810ca20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f299800b190 tx=0x7f2998006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:39.653 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:39.641+0000 7f29a4ff9700 1 -- 172.21.15.37:0/2375854420 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f299801b410 con 0x7f29a8070390 2024-02-11T16:59:39.653 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:39.641+0000 7f29a4ff9700 1 -- 172.21.15.37:0/2375854420 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f299801b9e0 con 0x7f29a8070390 2024-02-11T16:59:39.653 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:39.641+0000 7f29a4ff9700 1 -- 172.21.15.37:0/2375854420 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f299801a880 con 0x7f29a8070390 2024-02-11T16:59:39.653 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:39.641+0000 7f29ad0b9700 1 -- 172.21.15.37:0/2375854420 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29a810d110 con 0x7f29a8070390 2024-02-11T16:59:39.653 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:39.641+0000 7f29ad0b9700 1 -- 172.21.15.37:0/2375854420 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29a810d520 con 0x7f29a8070390 2024-02-11T16:59:39.653 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:39.642+0000 7f29a4ff9700 1 -- 172.21.15.37:0/2375854420 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f299800f070 con 0x7f29a8070390 2024-02-11T16:59:39.653 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:39.642+0000 7f29a4ff9700 1 --2- 172.21.15.37:0/2375854420 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7f298802fd50 0x7f2988032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:39.653 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:39.642+0000 7f29a4ff9700 1 -- 172.21.15.37:0/2375854420 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f2998041360 con 0x7f29a8070390 2024-02-11T16:59:39.653 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:39.642+0000 7f29ad0b9700 1 -- 172.21.15.37:0/2375854420 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29a8045ad0 con 0x7f29a8070390 2024-02-11T16:59:39.653 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:39.642+0000 7f29a77fe700 1 --2- 172.21.15.37:0/2375854420 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7f298802fd50 0x7f2988032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:39.654 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:39.642+0000 7f29a77fe700 1 --2- 172.21.15.37:0/2375854420 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7f298802fd50 0x7f2988032210 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f29900096a0 tx=0x7f2990006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:39.654 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:39.646+0000 7f29a4ff9700 1 -- 172.21.15.37:0/2375854420 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2998016070 con 0x7f29a8070390 2024-02-11T16:59:39.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:39 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:38.523345+0000 mgr.smithi037.eofnaq (mgr.14114) 12 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-02-11T16:59:39.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:39 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cephadm 2024-02-11T16:59:38.523862+0000 mgr.smithi037.eofnaq (mgr.14114) 13 : cephadm [INF] Saving service mgr spec with placement count:2 2024-02-11T16:59:39.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:39 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:38.526800+0000 mon.smithi037 (mon.0) 57 : audit [INF] from='mgr.14114 172.21.15.37:0/1469767164' entity='mgr.smithi037.eofnaq' 2024-02-11T16:59:39.833 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:39.832+0000 7f29ad0b9700 1 -- 172.21.15.37:0/2375854420 --> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f29a81069c0 con 0x7f298802fd50 2024-02-11T16:59:39.837 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:39.836+0000 7f29a4ff9700 1 -- 172.21.15.37:0/2375854420 <== mgr.14114 v2:172.21.15.37:6800/2256954126 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f29a81069c0 con 0x7f298802fd50 2024-02-11T16:59:39.837 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: Scheduled crash update... 2024-02-11T16:59:39.840 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:39.838+0000 7f29ad0b9700 1 -- 172.21.15.37:0/2375854420 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7f298802fd50 msgr2=0x7f2988032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:39.840 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:39.838+0000 7f29ad0b9700 1 --2- 172.21.15.37:0/2375854420 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7f298802fd50 0x7f2988032210 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f29900096a0 tx=0x7f2990006b40).stop 2024-02-11T16:59:39.840 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:39.838+0000 7f29ad0b9700 1 -- 172.21.15.37:0/2375854420 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f29a8070390 msgr2=0x7f29a810ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:39.840 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:39.838+0000 7f29ad0b9700 1 --2- 172.21.15.37:0/2375854420 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f29a8070390 0x7f29a810ca20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f299800b190 tx=0x7f2998006ee0).stop 2024-02-11T16:59:39.840 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:39.838+0000 7f29ad0b9700 1 -- 172.21.15.37:0/2375854420 shutdown_connections 2024-02-11T16:59:39.840 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:39.838+0000 7f29ad0b9700 1 --2- 172.21.15.37:0/2375854420 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7f298802fd50 0x7f2988032210 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:39.840 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:39.838+0000 7f29ad0b9700 1 --2- 172.21.15.37:0/2375854420 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f29a8070390 0x7f29a810ca20 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:39.840 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:39.838+0000 7f29ad0b9700 1 -- 172.21.15.37:0/2375854420 >> 172.21.15.37:0/2375854420 conn(0x7f29a80f6210 msgr2=0x7f29a80f7cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T16:59:39.840 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:39.838+0000 7f29ad0b9700 1 -- 172.21.15.37:0/2375854420 shutdown_connections 2024-02-11T16:59:39.840 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:39.838+0000 7f29ad0b9700 1 -- 172.21.15.37:0/2375854420 wait complete. 2024-02-11T16:59:40.377 INFO:teuthology.orchestra.run.smithi037.stderr:Enabling mgr prometheus module... 2024-02-11T16:59:40.378 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi037 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpjftqjcr4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4ata0hlm:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable prometheus 2024-02-11T16:59:40.924 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:40.915+0000 7fc5a1df8700 1 Processor -- start 2024-02-11T16:59:40.924 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:40.915+0000 7fc5a1df8700 1 -- start start 2024-02-11T16:59:40.924 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:40.915+0000 7fc5a1df8700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc59c070430 0x7fc59c0fe730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:40.924 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:40.915+0000 7fc5a1df8700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc59c0fec20 con 0x7fc59c070430 2024-02-11T16:59:40.924 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:40.915+0000 7fc5a0df6700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc59c070430 0x7fc59c0fe730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:40.924 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:40.915+0000 7fc5a0df6700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc59c070430 0x7fc59c0fe730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:60714/0 (socket says 172.21.15.37:60714) 2024-02-11T16:59:40.924 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:40.915+0000 7fc5a0df6700 1 -- 172.21.15.37:0/1354424066 learned_addr learned my addr 172.21.15.37:0/1354424066 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-02-11T16:59:40.924 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:40.916+0000 7fc5a0df6700 1 -- 172.21.15.37:0/1354424066 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc59c0fed60 con 0x7fc59c070430 2024-02-11T16:59:40.924 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:40.916+0000 7fc5a0df6700 1 --2- 172.21.15.37:0/1354424066 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc59c070430 0x7fc59c0fe730 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fc59000b5f0 tx=0x7fc590006ee0).ready entity=mon.0 client_cookie=aa908690c2a5936a server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:40.924 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:40.916+0000 7fc59b7fe700 1 -- 172.21.15.37:0/1354424066 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc59000d600 con 0x7fc59c070430 2024-02-11T16:59:40.925 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:40.916+0000 7fc59b7fe700 1 -- 172.21.15.37:0/1354424066 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fc59000dbd0 con 0x7fc59c070430 2024-02-11T16:59:40.925 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:40.916+0000 7fc59b7fe700 1 -- 172.21.15.37:0/1354424066 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc590011c00 con 0x7fc59c070430 2024-02-11T16:59:40.925 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:40.916+0000 7fc5a1df8700 1 -- 172.21.15.37:0/1354424066 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc59c070430 msgr2=0x7fc59c0fe730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:40.925 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:40.916+0000 7fc5a1df8700 1 --2- 172.21.15.37:0/1354424066 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc59c070430 0x7fc59c0fe730 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fc59000b5f0 tx=0x7fc590006ee0).stop 2024-02-11T16:59:40.925 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:40.916+0000 7fc5a1df8700 1 -- 172.21.15.37:0/1354424066 shutdown_connections 2024-02-11T16:59:40.925 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:40.916+0000 7fc5a1df8700 1 --2- 172.21.15.37:0/1354424066 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc59c070430 0x7fc59c0fe730 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:40.925 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:40.916+0000 7fc5a1df8700 1 -- 172.21.15.37:0/1354424066 >> 172.21.15.37:0/1354424066 conn(0x7fc59c0f6210 msgr2=0x7fc59c0f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T16:59:40.925 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:40.916+0000 7fc5a1df8700 1 -- 172.21.15.37:0/1354424066 shutdown_connections 2024-02-11T16:59:40.925 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:40.916+0000 7fc5a1df8700 1 -- 172.21.15.37:0/1354424066 wait complete. 2024-02-11T16:59:40.925 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:40.917+0000 7fc5a1df8700 1 Processor -- start 2024-02-11T16:59:40.925 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:40.917+0000 7fc5a1df8700 1 -- start start 2024-02-11T16:59:40.925 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:40.917+0000 7fc5a1df8700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc59c070430 0x7fc59c0fc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:40.925 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:40.917+0000 7fc5a1df8700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc59c0fa630 con 0x7fc59c070430 2024-02-11T16:59:40.926 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:40.917+0000 7fc5a0df6700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc59c070430 0x7fc59c0fc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:40.926 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:40.917+0000 7fc5a0df6700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc59c070430 0x7fc59c0fc1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:60718/0 (socket says 172.21.15.37:60718) 2024-02-11T16:59:40.926 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:40.918+0000 7fc5a0df6700 1 -- 172.21.15.37:0/2625478575 learned_addr learned my addr 172.21.15.37:0/2625478575 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-02-11T16:59:40.926 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:40.918+0000 7fc5a0df6700 1 -- 172.21.15.37:0/2625478575 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc59000b200 con 0x7fc59c070430 2024-02-11T16:59:40.926 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:40.918+0000 7fc5a0df6700 1 --2- 172.21.15.37:0/2625478575 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc59c070430 0x7fc59c0fc1b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fc59000b190 tx=0x7fc590006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:40.926 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:40.918+0000 7fc5997fa700 1 -- 172.21.15.37:0/2625478575 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc59001b410 con 0x7fc59c070430 2024-02-11T16:59:40.926 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:40.918+0000 7fc5a1df8700 1 -- 172.21.15.37:0/2625478575 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc59c0fa7d0 con 0x7fc59c070430 2024-02-11T16:59:40.926 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:40.918+0000 7fc5997fa700 1 -- 172.21.15.37:0/2625478575 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fc59001b9e0 con 0x7fc59c070430 2024-02-11T16:59:40.926 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:40.918+0000 7fc5997fa700 1 -- 172.21.15.37:0/2625478575 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc59001a880 con 0x7fc59c070430 2024-02-11T16:59:40.926 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:40.918+0000 7fc5a1df8700 1 -- 172.21.15.37:0/2625478575 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc59c0fabe0 con 0x7fc59c070430 2024-02-11T16:59:40.926 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:40.919+0000 7fc5997fa700 1 -- 172.21.15.37:0/2625478575 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fc59000f070 con 0x7fc59c070430 2024-02-11T16:59:40.926 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:40.919+0000 7fc5997fa700 1 --2- 172.21.15.37:0/2625478575 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7fc58002fd50 0x7fc580032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:40.926 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:40.919+0000 7fc5997fa700 1 -- 172.21.15.37:0/2625478575 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fc590041250 con 0x7fc59c070430 2024-02-11T16:59:40.926 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:40.919+0000 7fc59bfff700 1 --2- 172.21.15.37:0/2625478575 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7fc58002fd50 0x7fc580032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:40.927 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:40.919+0000 7fc5a1df8700 1 -- 172.21.15.37:0/2625478575 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc59c059100 con 0x7fc59c070430 2024-02-11T16:59:40.927 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:40.920+0000 7fc59bfff700 1 --2- 172.21.15.37:0/2625478575 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7fc58002fd50 0x7fc580032210 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fc5880096a0 tx=0x7fc588006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:40.927 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:40.924+0000 7fc5997fa700 1 -- 172.21.15.37:0/2625478575 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc590010360 con 0x7fc59c070430 2024-02-11T16:59:41.120 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:41.118+0000 7fc5a1df8700 1 -- 172.21.15.37:0/2625478575 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "prometheus"} v 0) v1 -- 0x7fc59c045ad0 con 0x7fc59c070430 2024-02-11T16:59:41.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:40 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:39.833835+0000 mgr.smithi037.eofnaq (mgr.14114) 14 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-02-11T16:59:41.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:40 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cephadm 2024-02-11T16:59:39.834472+0000 mgr.smithi037.eofnaq (mgr.14114) 15 : cephadm [INF] Saving service crash spec with placement * 2024-02-11T16:59:41.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:40 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:39.836781+0000 mon.smithi037 (mon.0) 58 : audit [INF] from='mgr.14114 172.21.15.37:0/1469767164' entity='mgr.smithi037.eofnaq' 2024-02-11T16:59:41.841 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:41.839+0000 7fc5997fa700 1 -- 172.21.15.37:0/2625478575 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7fc590025980 con 0x7fc59c070430 2024-02-11T16:59:41.842 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:41.841+0000 7fc5997fa700 1 -- 172.21.15.37:0/2625478575 <== mon.0 v2:172.21.15.37:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "prometheus"}]=0 v9) v1 ==== 89+0+0 (secure 0 0 0) 0x7fc590019d10 con 0x7fc59c070430 2024-02-11T16:59:41.845 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:41.843+0000 7fc5a1df8700 1 -- 172.21.15.37:0/2625478575 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7fc58002fd50 msgr2=0x7fc580032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:41.845 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:41.843+0000 7fc5a1df8700 1 --2- 172.21.15.37:0/2625478575 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7fc58002fd50 0x7fc580032210 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fc5880096a0 tx=0x7fc588006b40).stop 2024-02-11T16:59:41.845 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:41.843+0000 7fc5a1df8700 1 -- 172.21.15.37:0/2625478575 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc59c070430 msgr2=0x7fc59c0fc1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:41.845 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:41.843+0000 7fc5a1df8700 1 --2- 172.21.15.37:0/2625478575 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc59c070430 0x7fc59c0fc1b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fc59000b190 tx=0x7fc590006ee0).stop 2024-02-11T16:59:41.846 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:41.845+0000 7fc5a1df8700 1 -- 172.21.15.37:0/2625478575 shutdown_connections 2024-02-11T16:59:41.847 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:41.845+0000 7fc5a1df8700 1 --2- 172.21.15.37:0/2625478575 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7fc58002fd50 0x7fc580032210 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:41.847 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:41.845+0000 7fc5a1df8700 1 --2- 172.21.15.37:0/2625478575 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc59c070430 0x7fc59c0fc1b0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:41.847 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:41.845+0000 7fc5a1df8700 1 -- 172.21.15.37:0/2625478575 >> 172.21.15.37:0/2625478575 conn(0x7fc59c0f6210 msgr2=0x7fc59c0f7cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T16:59:41.847 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:41.845+0000 7fc5a1df8700 1 -- 172.21.15.37:0/2625478575 shutdown_connections 2024-02-11T16:59:41.847 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:41.845+0000 7fc5a1df8700 1 -- 172.21.15.37:0/2625478575 wait complete. 2024-02-11T16:59:42.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:41 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:41.119833+0000 mon.smithi037 (mon.0) 59 : audit [INF] from='client.? 172.21.15.37:0/2625478575' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-11T16:59:42.467 INFO:teuthology.orchestra.run.smithi037.stderr:Deploying prometheus service with default placement... 2024-02-11T16:59:42.467 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi037 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpjftqjcr4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4ata0hlm:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply prometheus 2024-02-11T16:59:42.996 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:42.988+0000 7f9754964700 1 Processor -- start 2024-02-11T16:59:42.997 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:42.988+0000 7f9754964700 1 -- start start 2024-02-11T16:59:42.997 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:42.988+0000 7f9754964700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9750072490 0x7f9750070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:42.997 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:42.988+0000 7f9754964700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97500728b0 con 0x7f9750072490 2024-02-11T16:59:42.997 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:42.988+0000 7f974e59c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9750072490 0x7f9750070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:42.997 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:42.988+0000 7f974e59c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9750072490 0x7f9750070a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:60746/0 (socket says 172.21.15.37:60746) 2024-02-11T16:59:42.997 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:42.988+0000 7f974e59c700 1 -- 172.21.15.37:0/3408840712 learned_addr learned my addr 172.21.15.37:0/3408840712 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-02-11T16:59:42.997 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:42.989+0000 7f974e59c700 1 -- 172.21.15.37:0/3408840712 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9750070f20 con 0x7f9750072490 2024-02-11T16:59:42.997 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:42.989+0000 7f974e59c700 1 --2- 172.21.15.37:0/3408840712 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9750072490 0x7f9750070a30 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f974000b5f0 tx=0x7f9740006ee0).ready entity=mon.0 client_cookie=81a1f52a5947aca7 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:42.997 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:42.989+0000 7f974d59a700 1 -- 172.21.15.37:0/3408840712 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f974000d600 con 0x7f9750072490 2024-02-11T16:59:42.997 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:42.989+0000 7f974d59a700 1 -- 172.21.15.37:0/3408840712 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f974000dbd0 con 0x7f9750072490 2024-02-11T16:59:42.997 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:42.989+0000 7f974d59a700 1 -- 172.21.15.37:0/3408840712 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9740011c00 con 0x7f9750072490 2024-02-11T16:59:42.997 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:42.990+0000 7f9754964700 1 -- 172.21.15.37:0/3408840712 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9750072490 msgr2=0x7f9750070a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:42.997 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:42.990+0000 7f9754964700 1 --2- 172.21.15.37:0/3408840712 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9750072490 0x7f9750070a30 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f974000b5f0 tx=0x7f9740006ee0).stop 2024-02-11T16:59:42.998 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:42.990+0000 7f9754964700 1 -- 172.21.15.37:0/3408840712 shutdown_connections 2024-02-11T16:59:42.998 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:42.990+0000 7f9754964700 1 --2- 172.21.15.37:0/3408840712 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9750072490 0x7f9750070a30 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:42.998 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:42.990+0000 7f9754964700 1 -- 172.21.15.37:0/3408840712 >> 172.21.15.37:0/3408840712 conn(0x7f97500f74c0 msgr2=0x7f97500f98e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T16:59:42.998 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:42.990+0000 7f9754964700 1 -- 172.21.15.37:0/3408840712 shutdown_connections 2024-02-11T16:59:42.998 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:42.990+0000 7f9754964700 1 -- 172.21.15.37:0/3408840712 wait complete. 2024-02-11T16:59:42.998 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:42.990+0000 7f9754964700 1 Processor -- start 2024-02-11T16:59:42.998 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:42.990+0000 7f9754964700 1 -- start start 2024-02-11T16:59:42.998 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:42.991+0000 7f9754964700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9750072490 0x7f975010cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:42.998 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:42.991+0000 7f9754964700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f975010d1d0 con 0x7f9750072490 2024-02-11T16:59:42.998 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:42.991+0000 7f974e59c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9750072490 0x7f975010cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:42.998 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:42.991+0000 7f974e59c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9750072490 0x7f975010cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:60760/0 (socket says 172.21.15.37:60760) 2024-02-11T16:59:42.998 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:42.991+0000 7f974e59c700 1 -- 172.21.15.37:0/1030545588 learned_addr learned my addr 172.21.15.37:0/1030545588 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-02-11T16:59:42.998 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:42.991+0000 7f974e59c700 1 -- 172.21.15.37:0/1030545588 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f974000b200 con 0x7f9750072490 2024-02-11T16:59:42.998 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:42.991+0000 7f974e59c700 1 --2- 172.21.15.37:0/1030545588 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9750072490 0x7f975010cce0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f9740011820 tx=0x7f9740006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:42.999 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:42.991+0000 7f973effd700 1 -- 172.21.15.37:0/1030545588 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f974001b410 con 0x7f9750072490 2024-02-11T16:59:42.999 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:42.991+0000 7f9754964700 1 -- 172.21.15.37:0/1030545588 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f975010d3d0 con 0x7f9750072490 2024-02-11T16:59:42.999 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:42.991+0000 7f973effd700 1 -- 172.21.15.37:0/1030545588 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f974001b9e0 con 0x7f9750072490 2024-02-11T16:59:42.999 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:42.991+0000 7f973effd700 1 -- 172.21.15.37:0/1030545588 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f974001a8b0 con 0x7f9750072490 2024-02-11T16:59:42.999 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:42.992+0000 7f9754964700 1 -- 172.21.15.37:0/1030545588 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f975010d7e0 con 0x7f9750072490 2024-02-11T16:59:42.999 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:42.992+0000 7f973effd700 1 -- 172.21.15.37:0/1030545588 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7f974000f070 con 0x7f9750072490 2024-02-11T16:59:42.999 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:42.993+0000 7f973effd700 1 --2- 172.21.15.37:0/1030545588 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7f973402fe00 0x7f97340322c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:42.999 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:42.993+0000 7f973effd700 1 -- 172.21.15.37:0/1030545588 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f9740041350 con 0x7f9750072490 2024-02-11T16:59:42.999 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:42.993+0000 7f974dd9b700 1 -- 172.21.15.37:0/1030545588 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7f973402fe00 msgr2=0x7f97340322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:6800/2256954126 2024-02-11T16:59:42.999 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:42.993+0000 7f974dd9b700 1 --2- 172.21.15.37:0/1030545588 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7f973402fe00 0x7f97340322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-11T16:59:42.999 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:42.993+0000 7f9754964700 1 -- 172.21.15.37:0/1030545588 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9750045ad0 con 0x7f9750072490 2024-02-11T16:59:43.000 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:42.998+0000 7f973effd700 1 -- 172.21.15.37:0/1030545588 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9740016070 con 0x7f9750072490 2024-02-11T16:59:43.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:42 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:41.841831+0000 mon.smithi037 (mon.0) 60 : audit [INF] from='client.? 172.21.15.37:0/2625478575' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-11T16:59:43.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:42 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T16:59:41.841896+0000 mon.smithi037 (mon.0) 61 : cluster [DBG] mgrmap e9: smithi037.eofnaq(active, since 18s) 2024-02-11T16:59:43.195 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:43.194+0000 7f974dd9b700 1 -- 172.21.15.37:0/1030545588 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7f973402fe00 msgr2=0x7f97340322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:6800/2256954126 2024-02-11T16:59:43.195 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:43.194+0000 7f974dd9b700 1 --2- 172.21.15.37:0/1030545588 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7f973402fe00 0x7f97340322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-11T16:59:43.233 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:43.232+0000 7f9754964700 1 -- 172.21.15.37:0/1030545588 --> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f975010ed10 con 0x7f973402fe00 2024-02-11T16:59:43.596 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:43.595+0000 7f974dd9b700 1 -- 172.21.15.37:0/1030545588 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7f973402fe00 msgr2=0x7f97340322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:6800/2256954126 2024-02-11T16:59:43.596 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:43.595+0000 7f974dd9b700 1 --2- 172.21.15.37:0/1030545588 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7f973402fe00 0x7f97340322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-02-11T16:59:44.397 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:44.396+0000 7f974dd9b700 1 -- 172.21.15.37:0/1030545588 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7f973402fe00 msgr2=0x7f97340322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:6800/2256954126 2024-02-11T16:59:44.398 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:44.396+0000 7f974dd9b700 1 --2- 172.21.15.37:0/1030545588 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7f973402fe00 0x7f97340322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-02-11T16:59:45.998 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:45.997+0000 7f974dd9b700 1 -- 172.21.15.37:0/1030545588 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7f973402fe00 msgr2=0x7f97340322c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:6800/2256954126 2024-02-11T16:59:45.999 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:45.997+0000 7f974dd9b700 1 --2- 172.21.15.37:0/1030545588 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7f973402fe00 0x7f97340322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2024-02-11T16:59:47.124 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:47.122+0000 7f973effd700 1 -- 172.21.15.37:0/1030545588 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mgrmap(e 10) v1 ==== 35984+0+0 (secure 0 0 0) 0x7f97400257e0 con 0x7f9750072490 2024-02-11T16:59:47.124 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:47.122+0000 7f973effd700 1 -- 172.21.15.37:0/1030545588 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7f973402fe00 msgr2=0x7f97340322c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-11T16:59:47.124 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:47.122+0000 7f973effd700 1 --2- 172.21.15.37:0/1030545588 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7f973402fe00 0x7f97340322c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-11T16:59:47.377 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:47 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T16:59:47.068459+0000 mon.smithi037 (mon.0) 62 : cluster [INF] Active manager daemon smithi037.eofnaq restarted 2024-02-11T16:59:47.378 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:47 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T16:59:47.068900+0000 mon.smithi037 (mon.0) 63 : cluster [INF] Activating manager daemon smithi037.eofnaq 2024-02-11T16:59:47.378 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:47 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T16:59:47.073172+0000 mon.smithi037 (mon.0) 64 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2024-02-11T16:59:48.128 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:48.126+0000 7f973effd700 1 -- 172.21.15.37:0/1030545588 <== mon.0 v2:172.21.15.37:3300/0 8 ==== mgrmap(e 11) v1 ==== 36054+0+0 (secure 0 0 0) 0x7f9740042ee0 con 0x7f9750072490 2024-02-11T16:59:48.129 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:48.126+0000 7f973effd700 1 --2- 172.21.15.37:0/1030545588 >> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] conn(0x7f9734033c20 0x7f9734036010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:48.129 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:48.127+0000 7f973effd700 1 -- 172.21.15.37:0/1030545588 --> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f975010ed10 con 0x7f9734033c20 2024-02-11T16:59:48.129 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:48.127+0000 7f974dd9b700 1 --2- 172.21.15.37:0/1030545588 >> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] conn(0x7f9734033c20 0x7f9734036010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:48.129 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:48.128+0000 7f974dd9b700 1 --2- 172.21.15.37:0/1030545588 >> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] conn(0x7f9734033c20 0x7f9734036010 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f9744006fd0 tx=0x7f9744006d30).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:48.134 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:48.133+0000 7f973effd700 1 -- 172.21.15.37:0/1030545588 <== mgr.14142 v2:172.21.15.37:6800/162023810 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f975010ed10 con 0x7f9734033c20 2024-02-11T16:59:48.135 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: Scheduled prometheus update... 2024-02-11T16:59:48.137 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:48.136+0000 7f9754964700 1 -- 172.21.15.37:0/1030545588 >> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] conn(0x7f9734033c20 msgr2=0x7f9734036010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:48.138 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:48.136+0000 7f9754964700 1 --2- 172.21.15.37:0/1030545588 >> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] conn(0x7f9734033c20 0x7f9734036010 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f9744006fd0 tx=0x7f9744006d30).stop 2024-02-11T16:59:48.138 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:48.136+0000 7f9754964700 1 -- 172.21.15.37:0/1030545588 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9750072490 msgr2=0x7f975010cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:48.138 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:48.136+0000 7f9754964700 1 --2- 172.21.15.37:0/1030545588 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9750072490 0x7f975010cce0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f9740011820 tx=0x7f9740006ee0).stop 2024-02-11T16:59:48.138 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:48.136+0000 7f9754964700 1 -- 172.21.15.37:0/1030545588 shutdown_connections 2024-02-11T16:59:48.138 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:48.136+0000 7f9754964700 1 --2- 172.21.15.37:0/1030545588 >> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] conn(0x7f9734033c20 0x7f9734036010 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:48.138 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:48.136+0000 7f9754964700 1 --2- 172.21.15.37:0/1030545588 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9750072490 0x7f975010cce0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:48.138 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:48.136+0000 7f9754964700 1 --2- 172.21.15.37:0/1030545588 >> [v2:172.21.15.37:6800/2256954126,v1:172.21.15.37:6801/2256954126] conn(0x7f973402fe00 0x7f97340322c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-11T16:59:48.138 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:48.136+0000 7f9754964700 1 -- 172.21.15.37:0/1030545588 >> 172.21.15.37:0/1030545588 conn(0x7f97500f74c0 msgr2=0x7f97500f9040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T16:59:48.138 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:48.136+0000 7f9754964700 1 -- 172.21.15.37:0/1030545588 shutdown_connections 2024-02-11T16:59:48.138 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:48.136+0000 7f9754964700 1 -- 172.21.15.37:0/1030545588 wait complete. 2024-02-11T16:59:48.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:48 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T16:59:47.123863+0000 mon.smithi037 (mon.0) 65 : cluster [DBG] mgrmap e10: smithi037.eofnaq(active, starting, since 0.055038s) 2024-02-11T16:59:48.434 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:48 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:47.127282+0000 mon.smithi037 (mon.0) 66 : audit [DBG] from='mgr.14142 172.21.15.37:0/3416274609' entity='mgr.smithi037.eofnaq' cmd=[{"prefix": "mon metadata", "id": "smithi037"}]: dispatch 2024-02-11T16:59:48.434 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:48 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:47.127825+0000 mon.smithi037 (mon.0) 67 : audit [DBG] from='mgr.14142 172.21.15.37:0/3416274609' entity='mgr.smithi037.eofnaq' cmd=[{"prefix": "mgr metadata", "who": "smithi037.eofnaq", "id": "smithi037.eofnaq"}]: dispatch 2024-02-11T16:59:48.434 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:48 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:47.128710+0000 mon.smithi037 (mon.0) 68 : audit [DBG] from='mgr.14142 172.21.15.37:0/3416274609' entity='mgr.smithi037.eofnaq' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-11T16:59:48.434 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:48 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:47.128847+0000 mon.smithi037 (mon.0) 69 : audit [DBG] from='mgr.14142 172.21.15.37:0/3416274609' entity='mgr.smithi037.eofnaq' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-11T16:59:48.434 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:48 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:47.128977+0000 mon.smithi037 (mon.0) 70 : audit [DBG] from='mgr.14142 172.21.15.37:0/3416274609' entity='mgr.smithi037.eofnaq' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-11T16:59:48.434 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:48 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T16:59:47.142155+0000 mon.smithi037 (mon.0) 71 : cluster [INF] Manager daemon smithi037.eofnaq is now available 2024-02-11T16:59:48.434 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:48 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:47.245627+0000 mon.smithi037 (mon.0) 72 : audit [DBG] from='mgr.14142 172.21.15.37:0/3416274609' entity='mgr.smithi037.eofnaq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-11T16:59:48.434 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:48 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:47.247289+0000 mon.smithi037 (mon.0) 73 : audit [DBG] from='mgr.14142 172.21.15.37:0/3416274609' entity='mgr.smithi037.eofnaq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-11T16:59:48.434 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:48 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:47.262233+0000 mon.smithi037 (mon.0) 74 : audit [INF] from='mgr.14142 172.21.15.37:0/3416274609' entity='mgr.smithi037.eofnaq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi037.eofnaq/mirror_snapshot_schedule"}]: dispatch 2024-02-11T16:59:48.434 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:48 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:47.264930+0000 mon.smithi037 (mon.0) 75 : audit [INF] from='mgr.14142 172.21.15.37:0/3416274609' entity='mgr.smithi037.eofnaq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi037.eofnaq/trash_purge_schedule"}]: dispatch 2024-02-11T16:59:48.742 INFO:teuthology.orchestra.run.smithi037.stderr:Deploying grafana service with default placement... 2024-02-11T16:59:48.742 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi037 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpjftqjcr4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4ata0hlm:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply grafana 2024-02-11T16:59:49.331 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:49.325+0000 7fe22073b700 1 Processor -- start 2024-02-11T16:59:49.331 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:49.325+0000 7fe22073b700 1 -- start start 2024-02-11T16:59:49.332 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:49.325+0000 7fe22073b700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe218070430 0x7fe2180fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:49.332 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:49.325+0000 7fe22073b700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2180fedd0 con 0x7fe218070430 2024-02-11T16:59:49.332 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:49.326+0000 7fe21e4d7700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe218070430 0x7fe2180fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:49.332 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:49.326+0000 7fe21e4d7700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe218070430 0x7fe2180fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:41208/0 (socket says 172.21.15.37:41208) 2024-02-11T16:59:49.332 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:49.326+0000 7fe21e4d7700 1 -- 172.21.15.37:0/824883630 learned_addr learned my addr 172.21.15.37:0/824883630 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-02-11T16:59:49.332 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:49.326+0000 7fe21e4d7700 1 -- 172.21.15.37:0/824883630 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2180fef10 con 0x7fe218070430 2024-02-11T16:59:49.332 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:49.326+0000 7fe21e4d7700 1 --2- 172.21.15.37:0/824883630 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe218070430 0x7fe2180fe8e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fe20800b5f0 tx=0x7fe208006ee0).ready entity=mon.0 client_cookie=eac4c3038caeec76 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:49.332 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:49.326+0000 7fe21d4d5700 1 -- 172.21.15.37:0/824883630 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe20800d600 con 0x7fe218070430 2024-02-11T16:59:49.332 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:49.326+0000 7fe21d4d5700 1 -- 172.21.15.37:0/824883630 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fe20800dbd0 con 0x7fe218070430 2024-02-11T16:59:49.332 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:49.326+0000 7fe21d4d5700 1 -- 172.21.15.37:0/824883630 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe208011c00 con 0x7fe218070430 2024-02-11T16:59:49.332 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:49.327+0000 7fe22073b700 1 -- 172.21.15.37:0/824883630 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe218070430 msgr2=0x7fe2180fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:49.333 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:49.327+0000 7fe22073b700 1 --2- 172.21.15.37:0/824883630 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe218070430 0x7fe2180fe8e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fe20800b5f0 tx=0x7fe208006ee0).stop 2024-02-11T16:59:49.333 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:49.327+0000 7fe22073b700 1 -- 172.21.15.37:0/824883630 shutdown_connections 2024-02-11T16:59:49.333 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:49.327+0000 7fe22073b700 1 --2- 172.21.15.37:0/824883630 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe218070430 0x7fe2180fe8e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:49.333 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:49.327+0000 7fe22073b700 1 -- 172.21.15.37:0/824883630 >> 172.21.15.37:0/824883630 conn(0x7fe2180f63c0 msgr2=0x7fe2180f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T16:59:49.333 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:49.327+0000 7fe22073b700 1 -- 172.21.15.37:0/824883630 shutdown_connections 2024-02-11T16:59:49.333 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:49.327+0000 7fe22073b700 1 -- 172.21.15.37:0/824883630 wait complete. 2024-02-11T16:59:49.333 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:49.328+0000 7fe22073b700 1 Processor -- start 2024-02-11T16:59:49.333 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:49.328+0000 7fe22073b700 1 -- start start 2024-02-11T16:59:49.333 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:49.328+0000 7fe22073b700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe218070430 0x7fe21810aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:49.333 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:49.328+0000 7fe22073b700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe21810af30 con 0x7fe218070430 2024-02-11T16:59:49.333 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:49.328+0000 7fe21e4d7700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe218070430 0x7fe21810aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:49.334 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:49.328+0000 7fe21e4d7700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe218070430 0x7fe21810aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:41214/0 (socket says 172.21.15.37:41214) 2024-02-11T16:59:49.334 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:49.328+0000 7fe21e4d7700 1 -- 172.21.15.37:0/1616502456 learned_addr learned my addr 172.21.15.37:0/1616502456 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-02-11T16:59:49.334 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:49.328+0000 7fe21e4d7700 1 -- 172.21.15.37:0/1616502456 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe20800b200 con 0x7fe218070430 2024-02-11T16:59:49.334 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:49.328+0000 7fe21e4d7700 1 --2- 172.21.15.37:0/1616502456 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe218070430 0x7fe21810aa40 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fe20800e040 tx=0x7fe208006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:49.334 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:49.329+0000 7fe206ffd700 1 -- 172.21.15.37:0/1616502456 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe20800d9f0 con 0x7fe218070430 2024-02-11T16:59:49.334 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:49.329+0000 7fe206ffd700 1 -- 172.21.15.37:0/1616502456 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fe20801a9f0 con 0x7fe218070430 2024-02-11T16:59:49.334 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:49.329+0000 7fe22073b700 1 -- 172.21.15.37:0/1616502456 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe21810b130 con 0x7fe218070430 2024-02-11T16:59:49.334 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:49.329+0000 7fe206ffd700 1 -- 172.21.15.37:0/1616502456 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe20801b8c0 con 0x7fe218070430 2024-02-11T16:59:49.334 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:49.329+0000 7fe22073b700 1 -- 172.21.15.37:0/1616502456 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe21810b540 con 0x7fe218070430 2024-02-11T16:59:49.335 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:49.329+0000 7fe206ffd700 1 -- 172.21.15.37:0/1616502456 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fe20800f070 con 0x7fe218070430 2024-02-11T16:59:49.335 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:49.330+0000 7fe22073b700 1 -- 172.21.15.37:0/1616502456 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe218045ad0 con 0x7fe218070430 2024-02-11T16:59:49.335 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:49.330+0000 7fe206ffd700 1 --2- 172.21.15.37:0/1616502456 >> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] conn(0x7fe20002feb0 0x7fe200032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:49.335 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:49.330+0000 7fe206ffd700 1 -- 172.21.15.37:0/1616502456 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fe208041430 con 0x7fe218070430 2024-02-11T16:59:49.335 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:49.330+0000 7fe21dcd6700 1 --2- 172.21.15.37:0/1616502456 >> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] conn(0x7fe20002feb0 0x7fe200032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:49.335 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:49.330+0000 7fe21dcd6700 1 --2- 172.21.15.37:0/1616502456 >> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] conn(0x7fe20002feb0 0x7fe200032370 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fe2140096a0 tx=0x7fe214006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:49.335 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:49.334+0000 7fe206ffd700 1 -- 172.21.15.37:0/1616502456 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe208022d50 con 0x7fe218070430 2024-02-11T16:59:49.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:49 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T16:59:48.129494+0000 mon.smithi037 (mon.0) 76 : cluster [DBG] mgrmap e11: smithi037.eofnaq(active, since 1.06067s) 2024-02-11T16:59:49.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:49 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cephadm 2024-02-11T16:59:48.130526+0000 mgr.smithi037.eofnaq (mgr.14142) 2 : cephadm [INF] Saving service prometheus spec with placement count:1 2024-02-11T16:59:49.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:49 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:48.134003+0000 mon.smithi037 (mon.0) 77 : audit [INF] from='mgr.14142 172.21.15.37:0/3416274609' entity='mgr.smithi037.eofnaq' 2024-02-11T16:59:49.521 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:49.519+0000 7fe22073b700 1 -- 172.21.15.37:0/1616502456 --> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7fe21810be20 con 0x7fe20002feb0 2024-02-11T16:59:49.527 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:49.526+0000 7fe206ffd700 1 -- 172.21.15.37:0/1616502456 <== mgr.14142 v2:172.21.15.37:6800/162023810 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7fe21810be20 con 0x7fe20002feb0 2024-02-11T16:59:49.527 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: Scheduled grafana update... 2024-02-11T16:59:49.530 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:49.528+0000 7fe22073b700 1 -- 172.21.15.37:0/1616502456 >> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] conn(0x7fe20002feb0 msgr2=0x7fe200032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:49.530 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:49.528+0000 7fe22073b700 1 --2- 172.21.15.37:0/1616502456 >> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] conn(0x7fe20002feb0 0x7fe200032370 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fe2140096a0 tx=0x7fe214006b40).stop 2024-02-11T16:59:49.530 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:49.528+0000 7fe22073b700 1 -- 172.21.15.37:0/1616502456 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe218070430 msgr2=0x7fe21810aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:49.530 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:49.528+0000 7fe22073b700 1 --2- 172.21.15.37:0/1616502456 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe218070430 0x7fe21810aa40 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fe20800e040 tx=0x7fe208006ee0).stop 2024-02-11T16:59:49.530 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:49.528+0000 7fe22073b700 1 -- 172.21.15.37:0/1616502456 shutdown_connections 2024-02-11T16:59:49.530 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:49.528+0000 7fe22073b700 1 --2- 172.21.15.37:0/1616502456 >> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] conn(0x7fe20002feb0 0x7fe200032370 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:49.530 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:49.528+0000 7fe22073b700 1 --2- 172.21.15.37:0/1616502456 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe218070430 0x7fe21810aa40 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:49.530 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:49.528+0000 7fe22073b700 1 -- 172.21.15.37:0/1616502456 >> 172.21.15.37:0/1616502456 conn(0x7fe2180f63c0 msgr2=0x7fe2180f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T16:59:49.530 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:49.528+0000 7fe22073b700 1 -- 172.21.15.37:0/1616502456 shutdown_connections 2024-02-11T16:59:49.530 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:49.528+0000 7fe22073b700 1 -- 172.21.15.37:0/1616502456 wait complete. 2024-02-11T16:59:50.207 INFO:teuthology.orchestra.run.smithi037.stderr:Deploying node-exporter service with default placement... 2024-02-11T16:59:50.208 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi037 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpjftqjcr4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4ata0hlm:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply node-exporter 2024-02-11T16:59:50.432 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:50 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T16:59:49.135485+0000 mon.smithi037 (mon.0) 78 : cluster [DBG] mgrmap e12: smithi037.eofnaq(active, since 2s) 2024-02-11T16:59:50.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:50 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:49.521182+0000 mgr.smithi037.eofnaq (mgr.14142) 3 : audit [DBG] from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-02-11T16:59:50.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:50 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cephadm 2024-02-11T16:59:49.521719+0000 mgr.smithi037.eofnaq (mgr.14142) 4 : cephadm [INF] Saving service grafana spec with placement count:1 2024-02-11T16:59:50.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:50 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:49.526684+0000 mon.smithi037 (mon.0) 79 : audit [INF] from='mgr.14142 172.21.15.37:0/3416274609' entity='mgr.smithi037.eofnaq' 2024-02-11T16:59:50.758 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:50.754+0000 7fe577357700 1 Processor -- start 2024-02-11T16:59:50.758 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:50.755+0000 7fe577357700 1 -- start start 2024-02-11T16:59:50.758 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:50.755+0000 7fe577357700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe5700fab80 0x7fe5700fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:50.758 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:50.755+0000 7fe577357700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5700fb490 con 0x7fe5700fab80 2024-02-11T16:59:50.758 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:50.755+0000 7fe5750f3700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe5700fab80 0x7fe5700fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:50.758 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:50.755+0000 7fe5750f3700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe5700fab80 0x7fe5700fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:41222/0 (socket says 172.21.15.37:41222) 2024-02-11T16:59:50.758 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:50.755+0000 7fe5750f3700 1 -- 172.21.15.37:0/3968840839 learned_addr learned my addr 172.21.15.37:0/3968840839 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-02-11T16:59:50.758 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:50.755+0000 7fe5750f3700 1 -- 172.21.15.37:0/3968840839 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5700fb5d0 con 0x7fe5700fab80 2024-02-11T16:59:50.758 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:50.755+0000 7fe5750f3700 1 --2- 172.21.15.37:0/3968840839 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe5700fab80 0x7fe5700fafa0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fe56c00b5f0 tx=0x7fe56c006ee0).ready entity=mon.0 client_cookie=e11a8989a610d748 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:50.758 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:50.755+0000 7fe563fff700 1 -- 172.21.15.37:0/3968840839 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe56c00d600 con 0x7fe5700fab80 2024-02-11T16:59:50.759 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:50.756+0000 7fe563fff700 1 -- 172.21.15.37:0/3968840839 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fe56c00dbd0 con 0x7fe5700fab80 2024-02-11T16:59:50.759 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:50.756+0000 7fe563fff700 1 -- 172.21.15.37:0/3968840839 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe56c011c00 con 0x7fe5700fab80 2024-02-11T16:59:50.759 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:50.756+0000 7fe577357700 1 -- 172.21.15.37:0/3968840839 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe5700fab80 msgr2=0x7fe5700fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:50.759 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:50.756+0000 7fe577357700 1 --2- 172.21.15.37:0/3968840839 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe5700fab80 0x7fe5700fafa0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fe56c00b5f0 tx=0x7fe56c006ee0).stop 2024-02-11T16:59:50.759 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:50.756+0000 7fe577357700 1 -- 172.21.15.37:0/3968840839 shutdown_connections 2024-02-11T16:59:50.759 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:50.756+0000 7fe577357700 1 --2- 172.21.15.37:0/3968840839 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe5700fab80 0x7fe5700fafa0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:50.759 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:50.756+0000 7fe577357700 1 -- 172.21.15.37:0/3968840839 >> 172.21.15.37:0/3968840839 conn(0x7fe5700f63c0 msgr2=0x7fe5700f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T16:59:50.759 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:50.756+0000 7fe577357700 1 -- 172.21.15.37:0/3968840839 shutdown_connections 2024-02-11T16:59:50.759 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:50.756+0000 7fe577357700 1 -- 172.21.15.37:0/3968840839 wait complete. 2024-02-11T16:59:50.759 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:50.757+0000 7fe577357700 1 Processor -- start 2024-02-11T16:59:50.761 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:50.757+0000 7fe577357700 1 -- start start 2024-02-11T16:59:50.761 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:50.757+0000 7fe577357700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe5700fab80 0x7fe57010a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:50.762 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:50.757+0000 7fe577357700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe57010ac70 con 0x7fe5700fab80 2024-02-11T16:59:50.762 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:50.757+0000 7fe5750f3700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe5700fab80 0x7fe57010a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:50.762 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:50.757+0000 7fe5750f3700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe5700fab80 0x7fe57010a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:41238/0 (socket says 172.21.15.37:41238) 2024-02-11T16:59:50.762 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:50.757+0000 7fe5750f3700 1 -- 172.21.15.37:0/3396766055 learned_addr learned my addr 172.21.15.37:0/3396766055 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-02-11T16:59:50.762 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:50.758+0000 7fe5750f3700 1 -- 172.21.15.37:0/3396766055 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe56c00b200 con 0x7fe5700fab80 2024-02-11T16:59:50.762 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:50.758+0000 7fe5750f3700 1 --2- 172.21.15.37:0/3396766055 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe5700fab80 0x7fe57010a780 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fe56c00bf40 tx=0x7fe56c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:50.762 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:50.758+0000 7fe561ffb700 1 -- 172.21.15.37:0/3396766055 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe56c01a6e0 con 0x7fe5700fab80 2024-02-11T16:59:50.762 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:50.758+0000 7fe561ffb700 1 -- 172.21.15.37:0/3396766055 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fe56c01acb0 con 0x7fe5700fab80 2024-02-11T16:59:50.762 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:50.758+0000 7fe577357700 1 -- 172.21.15.37:0/3396766055 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe57010ae70 con 0x7fe5700fab80 2024-02-11T16:59:50.762 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:50.758+0000 7fe561ffb700 1 -- 172.21.15.37:0/3396766055 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe56c01bb80 con 0x7fe5700fab80 2024-02-11T16:59:50.762 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:50.758+0000 7fe577357700 1 -- 172.21.15.37:0/3396766055 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe57010b280 con 0x7fe5700fab80 2024-02-11T16:59:50.762 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:50.759+0000 7fe561ffb700 1 -- 172.21.15.37:0/3396766055 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fe56c00f070 con 0x7fe5700fab80 2024-02-11T16:59:50.762 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:50.759+0000 7fe561ffb700 1 --2- 172.21.15.37:0/3396766055 >> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] conn(0x7fe55802fe60 0x7fe558032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:50.762 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:50.759+0000 7fe561ffb700 1 -- 172.21.15.37:0/3396766055 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fe56c041130 con 0x7fe5700fab80 2024-02-11T16:59:50.763 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:50.759+0000 7fe5748f2700 1 --2- 172.21.15.37:0/3396766055 >> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] conn(0x7fe55802fe60 0x7fe558032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:50.763 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:50.760+0000 7fe577357700 1 -- 172.21.15.37:0/3396766055 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe570059100 con 0x7fe5700fab80 2024-02-11T16:59:50.763 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:50.760+0000 7fe5748f2700 1 --2- 172.21.15.37:0/3396766055 >> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] conn(0x7fe55802fe60 0x7fe558032320 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fe5640096a0 tx=0x7fe564006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:50.765 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:50.764+0000 7fe561ffb700 1 -- 172.21.15.37:0/3396766055 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe56c016070 con 0x7fe5700fab80 2024-02-11T16:59:50.952 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:50.950+0000 7fe577357700 1 -- 172.21.15.37:0/3396766055 --> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7fe570104920 con 0x7fe55802fe60 2024-02-11T16:59:50.956 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:50.954+0000 7fe561ffb700 1 -- 172.21.15.37:0/3396766055 <== mgr.14142 v2:172.21.15.37:6800/162023810 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7fe570104920 con 0x7fe55802fe60 2024-02-11T16:59:50.956 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2024-02-11T16:59:50.958 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:50.956+0000 7fe577357700 1 -- 172.21.15.37:0/3396766055 >> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] conn(0x7fe55802fe60 msgr2=0x7fe558032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:50.958 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:50.956+0000 7fe577357700 1 --2- 172.21.15.37:0/3396766055 >> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] conn(0x7fe55802fe60 0x7fe558032320 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fe5640096a0 tx=0x7fe564006b40).stop 2024-02-11T16:59:50.958 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:50.956+0000 7fe577357700 1 -- 172.21.15.37:0/3396766055 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe5700fab80 msgr2=0x7fe57010a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:50.958 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:50.956+0000 7fe577357700 1 --2- 172.21.15.37:0/3396766055 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe5700fab80 0x7fe57010a780 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fe56c00bf40 tx=0x7fe56c006ee0).stop 2024-02-11T16:59:50.958 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:50.956+0000 7fe577357700 1 -- 172.21.15.37:0/3396766055 shutdown_connections 2024-02-11T16:59:50.958 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:50.956+0000 7fe577357700 1 --2- 172.21.15.37:0/3396766055 >> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] conn(0x7fe55802fe60 0x7fe558032320 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:50.958 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:50.956+0000 7fe577357700 1 --2- 172.21.15.37:0/3396766055 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe5700fab80 0x7fe57010a780 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:50.958 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:50.956+0000 7fe577357700 1 -- 172.21.15.37:0/3396766055 >> 172.21.15.37:0/3396766055 conn(0x7fe5700f63c0 msgr2=0x7fe5700f6fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T16:59:50.958 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:50.956+0000 7fe577357700 1 -- 172.21.15.37:0/3396766055 shutdown_connections 2024-02-11T16:59:50.958 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:50.956+0000 7fe577357700 1 -- 172.21.15.37:0/3396766055 wait complete. 2024-02-11T16:59:51.573 INFO:teuthology.orchestra.run.smithi037.stderr:Deploying alertmanager service with default placement... 2024-02-11T16:59:51.574 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi037 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpjftqjcr4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4ata0hlm:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply alertmanager 2024-02-11T16:59:52.150 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:52.148+0000 7fdbcd4f2700 1 Processor -- start 2024-02-11T16:59:52.151 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:52.148+0000 7fdbcd4f2700 1 -- start start 2024-02-11T16:59:52.151 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:52.148+0000 7fdbcd4f2700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdbc8070370 0x7fdbc80fe9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:52.151 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:52.148+0000 7fdbcd4f2700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc8070790 con 0x7fdbc8070370 2024-02-11T16:59:52.151 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:52.149+0000 7fdbc6ffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdbc8070370 0x7fdbc80fe9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:52.151 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:52.149+0000 7fdbc6ffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdbc8070370 0x7fdbc80fe9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:41244/0 (socket says 172.21.15.37:41244) 2024-02-11T16:59:52.151 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:52.149+0000 7fdbc6ffd700 1 -- 172.21.15.37:0/2066035097 learned_addr learned my addr 172.21.15.37:0/2066035097 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-02-11T16:59:52.154 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:52.149+0000 7fdbc6ffd700 1 -- 172.21.15.37:0/2066035097 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbc80fee90 con 0x7fdbc8070370 2024-02-11T16:59:52.155 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:52.149+0000 7fdbc6ffd700 1 --2- 172.21.15.37:0/2066035097 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdbc8070370 0x7fdbc80fe9a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7fdbb000b5f0 tx=0x7fdbb0006ee0).ready entity=mon.0 client_cookie=25eef3d13dd7179f server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:52.155 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:52.149+0000 7fdbc5ffb700 1 -- 172.21.15.37:0/2066035097 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdbb000d600 con 0x7fdbc8070370 2024-02-11T16:59:52.155 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:52.149+0000 7fdbc5ffb700 1 -- 172.21.15.37:0/2066035097 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fdbb000dbd0 con 0x7fdbc8070370 2024-02-11T16:59:52.155 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:52.149+0000 7fdbc5ffb700 1 -- 172.21.15.37:0/2066035097 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdbb0011c00 con 0x7fdbc8070370 2024-02-11T16:59:52.155 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:52.150+0000 7fdbcd4f2700 1 -- 172.21.15.37:0/2066035097 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdbc8070370 msgr2=0x7fdbc80fe9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:52.155 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:52.150+0000 7fdbcd4f2700 1 --2- 172.21.15.37:0/2066035097 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdbc8070370 0x7fdbc80fe9a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7fdbb000b5f0 tx=0x7fdbb0006ee0).stop 2024-02-11T16:59:52.155 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:52.150+0000 7fdbcd4f2700 1 -- 172.21.15.37:0/2066035097 shutdown_connections 2024-02-11T16:59:52.155 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:52.150+0000 7fdbcd4f2700 1 --2- 172.21.15.37:0/2066035097 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdbc8070370 0x7fdbc80fe9a0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:52.155 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:52.150+0000 7fdbcd4f2700 1 -- 172.21.15.37:0/2066035097 >> 172.21.15.37:0/2066035097 conn(0x7fdbc80f63c0 msgr2=0x7fdbc80f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T16:59:52.155 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:52.150+0000 7fdbcd4f2700 1 -- 172.21.15.37:0/2066035097 shutdown_connections 2024-02-11T16:59:52.155 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:52.150+0000 7fdbcd4f2700 1 -- 172.21.15.37:0/2066035097 wait complete. 2024-02-11T16:59:52.155 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:52.151+0000 7fdbcd4f2700 1 Processor -- start 2024-02-11T16:59:52.155 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:52.151+0000 7fdbcd4f2700 1 -- start start 2024-02-11T16:59:52.156 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:52.151+0000 7fdbcd4f2700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdbc8070370 0x7fdbc810a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:52.156 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:52.151+0000 7fdbcd4f2700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc810aea0 con 0x7fdbc8070370 2024-02-11T16:59:52.156 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:52.151+0000 7fdbc6ffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdbc8070370 0x7fdbc810a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:52.156 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:52.151+0000 7fdbc6ffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdbc8070370 0x7fdbc810a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:41248/0 (socket says 172.21.15.37:41248) 2024-02-11T16:59:52.156 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:52.151+0000 7fdbc6ffd700 1 -- 172.21.15.37:0/1618295262 learned_addr learned my addr 172.21.15.37:0/1618295262 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-02-11T16:59:52.156 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:52.152+0000 7fdbc6ffd700 1 -- 172.21.15.37:0/1618295262 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbb000b200 con 0x7fdbc8070370 2024-02-11T16:59:52.156 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:52.152+0000 7fdbc6ffd700 1 --2- 172.21.15.37:0/1618295262 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdbc8070370 0x7fdbc810a9b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7fdbb000b190 tx=0x7fdbb0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:52.156 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:52.152+0000 7fdbbf7fe700 1 -- 172.21.15.37:0/1618295262 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdbb001b410 con 0x7fdbc8070370 2024-02-11T16:59:52.156 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:52.152+0000 7fdbbf7fe700 1 -- 172.21.15.37:0/1618295262 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fdbb001b9e0 con 0x7fdbc8070370 2024-02-11T16:59:52.156 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:52.152+0000 7fdbcd4f2700 1 -- 172.21.15.37:0/1618295262 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbc810b0a0 con 0x7fdbc8070370 2024-02-11T16:59:52.156 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:52.152+0000 7fdbbf7fe700 1 -- 172.21.15.37:0/1618295262 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdbb001a8f0 con 0x7fdbc8070370 2024-02-11T16:59:52.156 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:52.152+0000 7fdbcd4f2700 1 -- 172.21.15.37:0/1618295262 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbc810b4b0 con 0x7fdbc8070370 2024-02-11T16:59:52.156 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:52.153+0000 7fdbbf7fe700 1 -- 172.21.15.37:0/1618295262 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fdbb000f070 con 0x7fdbc8070370 2024-02-11T16:59:52.157 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:52.153+0000 7fdbbf7fe700 1 --2- 172.21.15.37:0/1618295262 >> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] conn(0x7fdba802fe60 0x7fdba8032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:52.157 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:52.153+0000 7fdbbf7fe700 1 -- 172.21.15.37:0/1618295262 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fdbb0041450 con 0x7fdbc8070370 2024-02-11T16:59:52.157 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:52.153+0000 7fdbcd4f2700 1 -- 172.21.15.37:0/1618295262 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbc8045ad0 con 0x7fdbc8070370 2024-02-11T16:59:52.157 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:52.153+0000 7fdbc67fc700 1 --2- 172.21.15.37:0/1618295262 >> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] conn(0x7fdba802fe60 0x7fdba8032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:52.157 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:52.153+0000 7fdbc67fc700 1 --2- 172.21.15.37:0/1618295262 >> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] conn(0x7fdba802fe60 0x7fdba8032320 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fdbb80096a0 tx=0x7fdbb8006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:52.158 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:52.157+0000 7fdbbf7fe700 1 -- 172.21.15.37:0/1618295262 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdbb0016070 con 0x7fdbc8070370 2024-02-11T16:59:52.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:51 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:50.952160+0000 mgr.smithi037.eofnaq (mgr.14142) 5 : audit [DBG] from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-02-11T16:59:52.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:51 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cephadm 2024-02-11T16:59:50.952830+0000 mgr.smithi037.eofnaq (mgr.14142) 6 : cephadm [INF] Saving service node-exporter spec with placement * 2024-02-11T16:59:52.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:51 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:50.954948+0000 mon.smithi037 (mon.0) 80 : audit [INF] from='mgr.14142 172.21.15.37:0/3416274609' entity='mgr.smithi037.eofnaq' 2024-02-11T16:59:52.344 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:52.343+0000 7fdbcd4f2700 1 -- 172.21.15.37:0/1618295262 --> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7fdbc8104800 con 0x7fdba802fe60 2024-02-11T16:59:52.347 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:52.346+0000 7fdbbf7fe700 1 -- 172.21.15.37:0/1618295262 <== mgr.14142 v2:172.21.15.37:6800/162023810 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7fdbc8104800 con 0x7fdba802fe60 2024-02-11T16:59:52.347 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2024-02-11T16:59:52.350 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:52.348+0000 7fdbcd4f2700 1 -- 172.21.15.37:0/1618295262 >> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] conn(0x7fdba802fe60 msgr2=0x7fdba8032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:52.350 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:52.348+0000 7fdbcd4f2700 1 --2- 172.21.15.37:0/1618295262 >> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] conn(0x7fdba802fe60 0x7fdba8032320 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fdbb80096a0 tx=0x7fdbb8006b40).stop 2024-02-11T16:59:52.350 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:52.348+0000 7fdbcd4f2700 1 -- 172.21.15.37:0/1618295262 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdbc8070370 msgr2=0x7fdbc810a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:52.350 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:52.348+0000 7fdbcd4f2700 1 --2- 172.21.15.37:0/1618295262 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdbc8070370 0x7fdbc810a9b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7fdbb000b190 tx=0x7fdbb0006ee0).stop 2024-02-11T16:59:52.350 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:52.348+0000 7fdbcd4f2700 1 -- 172.21.15.37:0/1618295262 shutdown_connections 2024-02-11T16:59:52.350 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:52.348+0000 7fdbcd4f2700 1 --2- 172.21.15.37:0/1618295262 >> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] conn(0x7fdba802fe60 0x7fdba8032320 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:52.350 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:52.348+0000 7fdbcd4f2700 1 --2- 172.21.15.37:0/1618295262 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdbc8070370 0x7fdbc810a9b0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:52.350 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:52.348+0000 7fdbcd4f2700 1 -- 172.21.15.37:0/1618295262 >> 172.21.15.37:0/1618295262 conn(0x7fdbc80f63c0 msgr2=0x7fdbc80f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T16:59:52.350 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:52.348+0000 7fdbcd4f2700 1 -- 172.21.15.37:0/1618295262 shutdown_connections 2024-02-11T16:59:52.350 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:52.348+0000 7fdbcd4f2700 1 -- 172.21.15.37:0/1618295262 wait complete. 2024-02-11T16:59:52.914 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi037 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpjftqjcr4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4ata0hlm:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/cephadm/container_init True --force 2024-02-11T16:59:53.418 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:53.403+0000 7fc64dff2700 1 Processor -- start 2024-02-11T16:59:53.418 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:53.403+0000 7fc64dff2700 1 -- start start 2024-02-11T16:59:53.418 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:53.403+0000 7fc64dff2700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc648070430 0x7fc6480fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:53.418 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:53.403+0000 7fc64dff2700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6480fedd0 con 0x7fc648070430 2024-02-11T16:59:53.418 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:53.404+0000 7fc6477fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc648070430 0x7fc6480fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:53.418 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:53.404+0000 7fc6477fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc648070430 0x7fc6480fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:41250/0 (socket says 172.21.15.37:41250) 2024-02-11T16:59:53.418 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:53.404+0000 7fc6477fe700 1 -- 172.21.15.37:0/3838044698 learned_addr learned my addr 172.21.15.37:0/3838044698 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-02-11T16:59:53.418 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:53.404+0000 7fc6477fe700 1 -- 172.21.15.37:0/3838044698 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6480fef10 con 0x7fc648070430 2024-02-11T16:59:53.418 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:53.404+0000 7fc6477fe700 1 --2- 172.21.15.37:0/3838044698 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc648070430 0x7fc6480fe8e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fc63c00b5f0 tx=0x7fc63c006ee0).ready entity=mon.0 client_cookie=9955c137b4abeb04 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:53.418 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:53.404+0000 7fc6467fc700 1 -- 172.21.15.37:0/3838044698 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc63c00d600 con 0x7fc648070430 2024-02-11T16:59:53.418 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:53.404+0000 7fc6467fc700 1 -- 172.21.15.37:0/3838044698 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fc63c00dbd0 con 0x7fc648070430 2024-02-11T16:59:53.418 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:53.404+0000 7fc6467fc700 1 -- 172.21.15.37:0/3838044698 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc63c011c00 con 0x7fc648070430 2024-02-11T16:59:53.419 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:53.405+0000 7fc64dff2700 1 -- 172.21.15.37:0/3838044698 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc648070430 msgr2=0x7fc6480fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:53.419 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:53.405+0000 7fc64dff2700 1 --2- 172.21.15.37:0/3838044698 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc648070430 0x7fc6480fe8e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fc63c00b5f0 tx=0x7fc63c006ee0).stop 2024-02-11T16:59:53.419 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:53.405+0000 7fc64dff2700 1 -- 172.21.15.37:0/3838044698 shutdown_connections 2024-02-11T16:59:53.419 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:53.405+0000 7fc64dff2700 1 --2- 172.21.15.37:0/3838044698 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc648070430 0x7fc6480fe8e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:53.419 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:53.405+0000 7fc64dff2700 1 -- 172.21.15.37:0/3838044698 >> 172.21.15.37:0/3838044698 conn(0x7fc6480f63c0 msgr2=0x7fc6480f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T16:59:53.419 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:53.405+0000 7fc64dff2700 1 -- 172.21.15.37:0/3838044698 shutdown_connections 2024-02-11T16:59:53.419 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:53.405+0000 7fc64dff2700 1 -- 172.21.15.37:0/3838044698 wait complete. 2024-02-11T16:59:53.419 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:53.405+0000 7fc64dff2700 1 Processor -- start 2024-02-11T16:59:53.419 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:53.405+0000 7fc64dff2700 1 -- start start 2024-02-11T16:59:53.419 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:53.406+0000 7fc64dff2700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc648070430 0x7fc6480fc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:53.419 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:53.406+0000 7fc64dff2700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6480fa7e0 con 0x7fc648070430 2024-02-11T16:59:53.419 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:53.406+0000 7fc6477fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc648070430 0x7fc6480fc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:53.419 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:53.406+0000 7fc6477fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc648070430 0x7fc6480fc360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:41262/0 (socket says 172.21.15.37:41262) 2024-02-11T16:59:53.419 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:53.406+0000 7fc6477fe700 1 -- 172.21.15.37:0/3406738902 learned_addr learned my addr 172.21.15.37:0/3406738902 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-02-11T16:59:53.419 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:53.406+0000 7fc6477fe700 1 -- 172.21.15.37:0/3406738902 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc63c00b200 con 0x7fc648070430 2024-02-11T16:59:53.420 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:53.406+0000 7fc6477fe700 1 --2- 172.21.15.37:0/3406738902 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc648070430 0x7fc6480fc360 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fc63c00b190 tx=0x7fc63c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:53.420 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:53.406+0000 7fc62ffff700 1 -- 172.21.15.37:0/3406738902 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc63c01b410 con 0x7fc648070430 2024-02-11T16:59:53.420 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:53.406+0000 7fc62ffff700 1 -- 172.21.15.37:0/3406738902 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fc63c01b9e0 con 0x7fc648070430 2024-02-11T16:59:53.420 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:53.406+0000 7fc64dff2700 1 -- 172.21.15.37:0/3406738902 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6480fa980 con 0x7fc648070430 2024-02-11T16:59:53.420 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:53.406+0000 7fc62ffff700 1 -- 172.21.15.37:0/3406738902 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc63c01a880 con 0x7fc648070430 2024-02-11T16:59:53.420 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:53.406+0000 7fc64dff2700 1 -- 172.21.15.37:0/3406738902 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6480fad90 con 0x7fc648070430 2024-02-11T16:59:53.420 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:53.407+0000 7fc62ffff700 1 -- 172.21.15.37:0/3406738902 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fc63c00f070 con 0x7fc648070430 2024-02-11T16:59:53.420 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:53.407+0000 7fc64dff2700 1 -- 172.21.15.37:0/3406738902 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc648045ad0 con 0x7fc648070430 2024-02-11T16:59:53.420 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:53.407+0000 7fc62ffff700 1 --2- 172.21.15.37:0/3406738902 >> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] conn(0x7fc63402fe60 0x7fc634032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:53.420 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:53.407+0000 7fc62ffff700 1 -- 172.21.15.37:0/3406738902 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fc63c041400 con 0x7fc648070430 2024-02-11T16:59:53.420 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:53.407+0000 7fc646ffd700 1 --2- 172.21.15.37:0/3406738902 >> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] conn(0x7fc63402fe60 0x7fc634032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:53.420 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:53.408+0000 7fc646ffd700 1 --2- 172.21.15.37:0/3406738902 >> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] conn(0x7fc63402fe60 0x7fc634032320 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fc6380096a0 tx=0x7fc638006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:53.420 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:53.411+0000 7fc62ffff700 1 -- 172.21.15.37:0/3406738902 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc63c019cd0 con 0x7fc648070430 2024-02-11T16:59:53.570 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:53.569+0000 7fc64dff2700 1 -- 172.21.15.37:0/3406738902 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7fc6480ffca0 con 0x7fc648070430 2024-02-11T16:59:53.573 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:53.572+0000 7fc62ffff700 1 -- 172.21.15.37:0/3406738902 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v8)=0 v8) v1 ==== 142+0+0 (secure 0 0 0) 0x7fc63c016070 con 0x7fc648070430 2024-02-11T16:59:53.576 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:53.574+0000 7fc64dff2700 1 -- 172.21.15.37:0/3406738902 >> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] conn(0x7fc63402fe60 msgr2=0x7fc634032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:53.576 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:53.574+0000 7fc64dff2700 1 --2- 172.21.15.37:0/3406738902 >> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] conn(0x7fc63402fe60 0x7fc634032320 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fc6380096a0 tx=0x7fc638006b40).stop 2024-02-11T16:59:53.576 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:53.574+0000 7fc64dff2700 1 -- 172.21.15.37:0/3406738902 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc648070430 msgr2=0x7fc6480fc360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:53.576 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:53.574+0000 7fc64dff2700 1 --2- 172.21.15.37:0/3406738902 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc648070430 0x7fc6480fc360 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fc63c00b190 tx=0x7fc63c006ee0).stop 2024-02-11T16:59:53.576 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:53.574+0000 7fc64dff2700 1 -- 172.21.15.37:0/3406738902 shutdown_connections 2024-02-11T16:59:53.576 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:53.574+0000 7fc64dff2700 1 --2- 172.21.15.37:0/3406738902 >> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] conn(0x7fc63402fe60 0x7fc634032320 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:53.576 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:53.574+0000 7fc64dff2700 1 --2- 172.21.15.37:0/3406738902 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc648070430 0x7fc6480fc360 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:53.576 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:53.574+0000 7fc64dff2700 1 -- 172.21.15.37:0/3406738902 >> 172.21.15.37:0/3406738902 conn(0x7fc6480f63c0 msgr2=0x7fc6480f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T16:59:53.577 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:53.574+0000 7fc64dff2700 1 -- 172.21.15.37:0/3406738902 shutdown_connections 2024-02-11T16:59:53.577 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:53.574+0000 7fc64dff2700 1 -- 172.21.15.37:0/3406738902 wait complete. 2024-02-11T16:59:53.598 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:53 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:52.344328+0000 mgr.smithi037.eofnaq (mgr.14142) 7 : audit [DBG] from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-02-11T16:59:53.598 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:53 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cephadm 2024-02-11T16:59:52.344835+0000 mgr.smithi037.eofnaq (mgr.14142) 8 : cephadm [INF] Saving service alertmanager spec with placement count:1 2024-02-11T16:59:53.598 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:53 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:52.346989+0000 mon.smithi037 (mon.0) 81 : audit [INF] from='mgr.14142 172.21.15.37:0/3416274609' entity='mgr.smithi037.eofnaq' 2024-02-11T16:59:54.220 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi037 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpjftqjcr4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4ata0hlm:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/dashboard/ssl_server_port 8443 2024-02-11T16:59:54.933 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:54 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:53.572952+0000 mon.smithi037 (mon.0) 82 : audit [INF] from='client.? 172.21.15.37:0/3406738902' entity='client.admin' 2024-02-11T16:59:54.941 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:54.937+0000 7f62354a0700 1 Processor -- start 2024-02-11T16:59:54.941 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:54.937+0000 7f62354a0700 1 -- start start 2024-02-11T16:59:54.941 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:54.937+0000 7f62354a0700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6230070370 0x7f62300fe9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:54.941 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:54.937+0000 7f62354a0700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6230070790 con 0x7f6230070370 2024-02-11T16:59:54.941 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:54.938+0000 7f622effd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6230070370 0x7f62300fe9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:54.941 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:54.938+0000 7f622effd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6230070370 0x7f62300fe9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:41276/0 (socket says 172.21.15.37:41276) 2024-02-11T16:59:54.941 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:54.938+0000 7f622effd700 1 -- 172.21.15.37:0/2151674465 learned_addr learned my addr 172.21.15.37:0/2151674465 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-02-11T16:59:54.941 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:54.938+0000 7f622effd700 1 -- 172.21.15.37:0/2151674465 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62300fee90 con 0x7f6230070370 2024-02-11T16:59:54.941 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:54.938+0000 7f622effd700 1 --2- 172.21.15.37:0/2151674465 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6230070370 0x7f62300fe9a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f622000b5f0 tx=0x7f6220006ee0).ready entity=mon.0 client_cookie=df9206e5d9c1264d server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:54.941 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:54.938+0000 7f622dffb700 1 -- 172.21.15.37:0/2151674465 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f622000d600 con 0x7f6230070370 2024-02-11T16:59:54.942 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:54.938+0000 7f622dffb700 1 -- 172.21.15.37:0/2151674465 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f622000dbd0 con 0x7f6230070370 2024-02-11T16:59:54.942 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:54.938+0000 7f622dffb700 1 -- 172.21.15.37:0/2151674465 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6220011c00 con 0x7f6230070370 2024-02-11T16:59:54.942 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:54.939+0000 7f62354a0700 1 -- 172.21.15.37:0/2151674465 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6230070370 msgr2=0x7f62300fe9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:54.942 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:54.939+0000 7f62354a0700 1 --2- 172.21.15.37:0/2151674465 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6230070370 0x7f62300fe9a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f622000b5f0 tx=0x7f6220006ee0).stop 2024-02-11T16:59:54.942 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:54.939+0000 7f62354a0700 1 -- 172.21.15.37:0/2151674465 shutdown_connections 2024-02-11T16:59:54.942 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:54.939+0000 7f62354a0700 1 --2- 172.21.15.37:0/2151674465 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6230070370 0x7f62300fe9a0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:54.942 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:54.939+0000 7f62354a0700 1 -- 172.21.15.37:0/2151674465 >> 172.21.15.37:0/2151674465 conn(0x7f62300f63c0 msgr2=0x7f62300f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T16:59:54.942 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:54.939+0000 7f62354a0700 1 -- 172.21.15.37:0/2151674465 shutdown_connections 2024-02-11T16:59:54.942 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:54.939+0000 7f62354a0700 1 -- 172.21.15.37:0/2151674465 wait complete. 2024-02-11T16:59:54.942 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:54.940+0000 7f62354a0700 1 Processor -- start 2024-02-11T16:59:54.942 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:54.940+0000 7f62354a0700 1 -- start start 2024-02-11T16:59:54.944 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:54.940+0000 7f62354a0700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6230070370 0x7f623010aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:54.944 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:54.940+0000 7f62354a0700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f623010af00 con 0x7f6230070370 2024-02-11T16:59:54.945 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:54.940+0000 7f622effd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6230070370 0x7f623010aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:54.945 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:54.940+0000 7f622effd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6230070370 0x7f623010aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:41290/0 (socket says 172.21.15.37:41290) 2024-02-11T16:59:54.945 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:54.940+0000 7f622effd700 1 -- 172.21.15.37:0/1780473382 learned_addr learned my addr 172.21.15.37:0/1780473382 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-02-11T16:59:54.945 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:54.940+0000 7f622effd700 1 -- 172.21.15.37:0/1780473382 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f622000b200 con 0x7f6230070370 2024-02-11T16:59:54.945 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:54.940+0000 7f622effd700 1 --2- 172.21.15.37:0/1780473382 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6230070370 0x7f623010aa10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f622000b1d0 tx=0x7f6220006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:54.945 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:54.941+0000 7f62177fe700 1 -- 172.21.15.37:0/1780473382 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6220011a20 con 0x7f6230070370 2024-02-11T16:59:54.945 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:54.941+0000 7f62177fe700 1 -- 172.21.15.37:0/1780473382 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f622001b880 con 0x7f6230070370 2024-02-11T16:59:54.945 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:54.941+0000 7f62354a0700 1 -- 172.21.15.37:0/1780473382 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f623010b100 con 0x7f6230070370 2024-02-11T16:59:54.945 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:54.941+0000 7f62177fe700 1 -- 172.21.15.37:0/1780473382 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f622001a770 con 0x7f6230070370 2024-02-11T16:59:54.945 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:54.941+0000 7f62354a0700 1 -- 172.21.15.37:0/1780473382 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f623010b510 con 0x7f6230070370 2024-02-11T16:59:54.945 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:54.941+0000 7f62177fe700 1 -- 172.21.15.37:0/1780473382 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f622000f070 con 0x7f6230070370 2024-02-11T16:59:54.945 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:54.942+0000 7f62354a0700 1 -- 172.21.15.37:0/1780473382 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6230045ad0 con 0x7f6230070370 2024-02-11T16:59:54.945 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:54.942+0000 7f62177fe700 1 --2- 172.21.15.37:0/1780473382 >> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] conn(0x7f621c02feb0 0x7f621c032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:54.945 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:54.942+0000 7f62177fe700 1 -- 172.21.15.37:0/1780473382 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f62200412a0 con 0x7f6230070370 2024-02-11T16:59:54.946 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:54.942+0000 7f622e7fc700 1 --2- 172.21.15.37:0/1780473382 >> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] conn(0x7f621c02feb0 0x7f621c032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:54.946 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:54.942+0000 7f622e7fc700 1 --2- 172.21.15.37:0/1780473382 >> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] conn(0x7f621c02feb0 0x7f621c032370 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f62240096a0 tx=0x7f6224006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:54.947 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:54.946+0000 7f62177fe700 1 -- 172.21.15.37:0/1780473382 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6220022b40 con 0x7f6230070370 2024-02-11T16:59:55.108 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:55.106+0000 7f62354a0700 1 -- 172.21.15.37:0/1780473382 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f62300ffb00 con 0x7f6230070370 2024-02-11T16:59:55.110 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:55.109+0000 7f62177fe700 1 -- 172.21.15.37:0/1780473382 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v9)=0 v9) v1 ==== 130+0+0 (secure 0 0 0) 0x7f6220016070 con 0x7f6230070370 2024-02-11T16:59:55.113 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:55.111+0000 7f62354a0700 1 -- 172.21.15.37:0/1780473382 >> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] conn(0x7f621c02feb0 msgr2=0x7f621c032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:55.113 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:55.111+0000 7f62354a0700 1 --2- 172.21.15.37:0/1780473382 >> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] conn(0x7f621c02feb0 0x7f621c032370 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f62240096a0 tx=0x7f6224006b40).stop 2024-02-11T16:59:55.113 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:55.111+0000 7f62354a0700 1 -- 172.21.15.37:0/1780473382 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6230070370 msgr2=0x7f623010aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:55.113 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:55.111+0000 7f62354a0700 1 --2- 172.21.15.37:0/1780473382 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6230070370 0x7f623010aa10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f622000b1d0 tx=0x7f6220006ee0).stop 2024-02-11T16:59:55.113 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:55.111+0000 7f62354a0700 1 -- 172.21.15.37:0/1780473382 shutdown_connections 2024-02-11T16:59:55.113 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:55.111+0000 7f62354a0700 1 --2- 172.21.15.37:0/1780473382 >> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] conn(0x7f621c02feb0 0x7f621c032370 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:55.113 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:55.111+0000 7f62354a0700 1 --2- 172.21.15.37:0/1780473382 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6230070370 0x7f623010aa10 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:55.113 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:55.111+0000 7f62354a0700 1 -- 172.21.15.37:0/1780473382 >> 172.21.15.37:0/1780473382 conn(0x7f62300f63c0 msgr2=0x7f62300f7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T16:59:55.113 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:55.111+0000 7f62354a0700 1 -- 172.21.15.37:0/1780473382 shutdown_connections 2024-02-11T16:59:55.114 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:55.111+0000 7f62354a0700 1 -- 172.21.15.37:0/1780473382 wait complete. 2024-02-11T16:59:56.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:56 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:55.110209+0000 mon.smithi037 (mon.0) 83 : audit [INF] from='client.? 172.21.15.37:0/1780473382' entity='client.admin' 2024-02-11T16:59:56.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:56 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:56.068980+0000 mon.smithi037 (mon.0) 84 : audit [INF] from='mgr.14142 172.21.15.37:0/3416274609' entity='mgr.smithi037.eofnaq' 2024-02-11T16:59:56.528 INFO:teuthology.orchestra.run.smithi037.stderr:Enabling the dashboard module... 2024-02-11T16:59:56.528 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi037 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpjftqjcr4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4ata0hlm:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable dashboard 2024-02-11T16:59:57.022 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:57.019+0000 7fcf75b4b700 1 Processor -- start 2024-02-11T16:59:57.022 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:57.019+0000 7fcf75b4b700 1 -- start start 2024-02-11T16:59:57.022 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:57.019+0000 7fcf75b4b700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcf700724d0 0x7fcf70070a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:57.022 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:57.019+0000 7fcf75b4b700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf700728f0 con 0x7fcf700724d0 2024-02-11T16:59:57.022 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:57.019+0000 7fcf6f7fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcf700724d0 0x7fcf70070a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:57.022 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:57.019+0000 7fcf6f7fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcf700724d0 0x7fcf70070a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:41294/0 (socket says 172.21.15.37:41294) 2024-02-11T16:59:57.022 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:57.019+0000 7fcf6f7fe700 1 -- 172.21.15.37:0/3822194679 learned_addr learned my addr 172.21.15.37:0/3822194679 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-02-11T16:59:57.022 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:57.020+0000 7fcf6f7fe700 1 -- 172.21.15.37:0/3822194679 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf70070f60 con 0x7fcf700724d0 2024-02-11T16:59:57.022 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:57.020+0000 7fcf6f7fe700 1 --2- 172.21.15.37:0/3822194679 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcf700724d0 0x7fcf70070a70 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fcf6000b5f0 tx=0x7fcf60006ee0).ready entity=mon.0 client_cookie=1de10e854d7ce364 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:57.022 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:57.020+0000 7fcf6e7fc700 1 -- 172.21.15.37:0/3822194679 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf6000d600 con 0x7fcf700724d0 2024-02-11T16:59:57.023 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:57.020+0000 7fcf6e7fc700 1 -- 172.21.15.37:0/3822194679 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcf6000dc10 con 0x7fcf700724d0 2024-02-11T16:59:57.023 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:57.020+0000 7fcf6e7fc700 1 -- 172.21.15.37:0/3822194679 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf60011cd0 con 0x7fcf700724d0 2024-02-11T16:59:57.023 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:57.021+0000 7fcf75b4b700 1 -- 172.21.15.37:0/3822194679 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcf700724d0 msgr2=0x7fcf70070a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:57.025 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:57.021+0000 7fcf75b4b700 1 --2- 172.21.15.37:0/3822194679 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcf700724d0 0x7fcf70070a70 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fcf6000b5f0 tx=0x7fcf60006ee0).stop 2024-02-11T16:59:57.025 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:57.021+0000 7fcf75b4b700 1 -- 172.21.15.37:0/3822194679 shutdown_connections 2024-02-11T16:59:57.025 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:57.021+0000 7fcf75b4b700 1 --2- 172.21.15.37:0/3822194679 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcf700724d0 0x7fcf70070a70 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:57.025 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:57.021+0000 7fcf75b4b700 1 -- 172.21.15.37:0/3822194679 >> 172.21.15.37:0/3822194679 conn(0x7fcf700f74c0 msgr2=0x7fcf700f98e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T16:59:57.025 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:57.021+0000 7fcf75b4b700 1 -- 172.21.15.37:0/3822194679 shutdown_connections 2024-02-11T16:59:57.025 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:57.021+0000 7fcf75b4b700 1 -- 172.21.15.37:0/3822194679 wait complete. 2024-02-11T16:59:57.025 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:57.022+0000 7fcf75b4b700 1 Processor -- start 2024-02-11T16:59:57.025 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:57.022+0000 7fcf75b4b700 1 -- start start 2024-02-11T16:59:57.026 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:57.022+0000 7fcf75b4b700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcf700724d0 0x7fcf7010cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:57.026 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:57.022+0000 7fcf75b4b700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf7010d210 con 0x7fcf700724d0 2024-02-11T16:59:57.026 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:57.023+0000 7fcf6f7fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcf700724d0 0x7fcf7010cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:57.026 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:57.023+0000 7fcf6f7fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcf700724d0 0x7fcf7010cd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:41304/0 (socket says 172.21.15.37:41304) 2024-02-11T16:59:57.026 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:57.023+0000 7fcf6f7fe700 1 -- 172.21.15.37:0/3497605218 learned_addr learned my addr 172.21.15.37:0/3497605218 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-02-11T16:59:57.026 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:57.023+0000 7fcf6f7fe700 1 -- 172.21.15.37:0/3497605218 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf6000b200 con 0x7fcf700724d0 2024-02-11T16:59:57.026 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:57.023+0000 7fcf6f7fe700 1 --2- 172.21.15.37:0/3497605218 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcf700724d0 0x7fcf7010cd20 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fcf6000b1d0 tx=0x7fcf60006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:57.026 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:57.023+0000 7fcf57fff700 1 -- 172.21.15.37:0/3497605218 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf60011df0 con 0x7fcf700724d0 2024-02-11T16:59:57.026 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:57.023+0000 7fcf57fff700 1 -- 172.21.15.37:0/3497605218 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcf6001b8c0 con 0x7fcf700724d0 2024-02-11T16:59:57.026 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:57.023+0000 7fcf57fff700 1 -- 172.21.15.37:0/3497605218 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf6001a830 con 0x7fcf700724d0 2024-02-11T16:59:57.026 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:57.023+0000 7fcf75b4b700 1 -- 172.21.15.37:0/3497605218 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf7010d410 con 0x7fcf700724d0 2024-02-11T16:59:57.026 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:57.024+0000 7fcf75b4b700 1 -- 172.21.15.37:0/3497605218 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf7010d820 con 0x7fcf700724d0 2024-02-11T16:59:57.027 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:57.024+0000 7fcf57fff700 1 -- 172.21.15.37:0/3497605218 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fcf6000f070 con 0x7fcf700724d0 2024-02-11T16:59:57.027 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:57.024+0000 7fcf75b4b700 1 -- 172.21.15.37:0/3497605218 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf70045ad0 con 0x7fcf700724d0 2024-02-11T16:59:57.027 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:57.025+0000 7fcf57fff700 1 --2- 172.21.15.37:0/3497605218 >> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] conn(0x7fcf5c02feb0 0x7fcf5c032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:57.027 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:57.025+0000 7fcf57fff700 1 -- 172.21.15.37:0/3497605218 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fcf60040ef0 con 0x7fcf700724d0 2024-02-11T16:59:57.027 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:57.025+0000 7fcf6effd700 1 --2- 172.21.15.37:0/3497605218 >> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] conn(0x7fcf5c02feb0 0x7fcf5c032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:57.028 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:57.025+0000 7fcf6effd700 1 --2- 172.21.15.37:0/3497605218 >> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] conn(0x7fcf5c02feb0 0x7fcf5c032370 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fcf640096a0 tx=0x7fcf64006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:57.030 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:57.029+0000 7fcf57fff700 1 -- 172.21.15.37:0/3497605218 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcf60010950 con 0x7fcf700724d0 2024-02-11T16:59:57.239 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:57.238+0000 7fcf75b4b700 1 -- 172.21.15.37:0/3497605218 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7fcf70101e70 con 0x7fcf700724d0 2024-02-11T16:59:57.842 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:57.840+0000 7fcf57fff700 1 -- 172.21.15.37:0/3497605218 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fcf60024790 con 0x7fcf700724d0 2024-02-11T16:59:57.843 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:57.841+0000 7fcf57fff700 1 -- 172.21.15.37:0/3497605218 <== mon.0 v2:172.21.15.37:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v13) v1 ==== 88+0+0 (secure 0 0 0) 0x7fcf60005990 con 0x7fcf700724d0 2024-02-11T16:59:57.846 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:57.844+0000 7fcf75b4b700 1 -- 172.21.15.37:0/3497605218 >> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] conn(0x7fcf5c02feb0 msgr2=0x7fcf5c032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:57.846 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:57.844+0000 7fcf75b4b700 1 --2- 172.21.15.37:0/3497605218 >> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] conn(0x7fcf5c02feb0 0x7fcf5c032370 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fcf640096a0 tx=0x7fcf64006b40).stop 2024-02-11T16:59:57.846 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:57.844+0000 7fcf75b4b700 1 -- 172.21.15.37:0/3497605218 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcf700724d0 msgr2=0x7fcf7010cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:57.846 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:57.844+0000 7fcf75b4b700 1 --2- 172.21.15.37:0/3497605218 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcf700724d0 0x7fcf7010cd20 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fcf6000b1d0 tx=0x7fcf60006ee0).stop 2024-02-11T16:59:57.846 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:57.844+0000 7fcf75b4b700 1 -- 172.21.15.37:0/3497605218 shutdown_connections 2024-02-11T16:59:57.846 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:57.844+0000 7fcf75b4b700 1 --2- 172.21.15.37:0/3497605218 >> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] conn(0x7fcf5c02feb0 0x7fcf5c032370 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:57.846 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:57.844+0000 7fcf75b4b700 1 --2- 172.21.15.37:0/3497605218 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcf700724d0 0x7fcf7010cd20 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:57.846 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:57.844+0000 7fcf75b4b700 1 -- 172.21.15.37:0/3497605218 >> 172.21.15.37:0/3497605218 conn(0x7fcf700f74c0 msgr2=0x7fcf700f8f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T16:59:57.846 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:57.844+0000 7fcf75b4b700 1 -- 172.21.15.37:0/3497605218 shutdown_connections 2024-02-11T16:59:57.846 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:57.844+0000 7fcf75b4b700 1 -- 172.21.15.37:0/3497605218 wait complete. 2024-02-11T16:59:58.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:57 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:56.835614+0000 mon.smithi037 (mon.0) 85 : audit [INF] from='mgr.14142 172.21.15.37:0/3416274609' entity='mgr.smithi037.eofnaq' 2024-02-11T16:59:58.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:57 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:57.239304+0000 mon.smithi037 (mon.0) 86 : audit [INF] from='client.? 172.21.15.37:0/3497605218' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-02-11T16:59:58.599 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi037 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpjftqjcr4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4ata0hlm:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2024-02-11T16:59:58.896 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:58 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:57.842260+0000 mon.smithi037 (mon.0) 87 : audit [INF] from='client.? 172.21.15.37:0/3497605218' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-02-11T16:59:58.896 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:58 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T16:59:57.842312+0000 mon.smithi037 (mon.0) 88 : cluster [DBG] mgrmap e13: smithi037.eofnaq(active, since 10s) 2024-02-11T16:59:58.896 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:58 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:57.844763+0000 mon.smithi037 (mon.0) 89 : audit [INF] from='mgr.14142 172.21.15.37:0/3416274609' entity='mgr.smithi037.eofnaq' 2024-02-11T16:59:59.245 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:59.240+0000 7fbf52886700 1 Processor -- start 2024-02-11T16:59:59.245 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:59.240+0000 7fbf52886700 1 -- start start 2024-02-11T16:59:59.245 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:59.240+0000 7fbf52886700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbf4c0fbcb0 0x7fbf4c0fc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:59.245 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:59.240+0000 7fbf52886700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf4c0fc5c0 con 0x7fbf4c0fbcb0 2024-02-11T16:59:59.246 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:59.240+0000 7fbf4bfff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbf4c0fbcb0 0x7fbf4c0fc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:59.246 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:59.241+0000 7fbf4bfff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbf4c0fbcb0 0x7fbf4c0fc0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:51512/0 (socket says 172.21.15.37:51512) 2024-02-11T16:59:59.246 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:59.241+0000 7fbf4bfff700 1 -- 172.21.15.37:0/3160248076 learned_addr learned my addr 172.21.15.37:0/3160248076 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-02-11T16:59:59.246 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:59.241+0000 7fbf4bfff700 1 -- 172.21.15.37:0/3160248076 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf4c0fc700 con 0x7fbf4c0fbcb0 2024-02-11T16:59:59.246 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:59.241+0000 7fbf4bfff700 1 --2- 172.21.15.37:0/3160248076 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbf4c0fbcb0 0x7fbf4c0fc0d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7fbf3c00b5f0 tx=0x7fbf3c006ee0).ready entity=mon.0 client_cookie=5d0151090edaaf3f server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:59.246 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:59.241+0000 7fbf4affd700 1 -- 172.21.15.37:0/3160248076 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf3c00d600 con 0x7fbf4c0fbcb0 2024-02-11T16:59:59.246 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:59.241+0000 7fbf4affd700 1 -- 172.21.15.37:0/3160248076 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbf3c00dc10 con 0x7fbf4c0fbcb0 2024-02-11T16:59:59.246 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:59.241+0000 7fbf4affd700 1 -- 172.21.15.37:0/3160248076 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf3c011cd0 con 0x7fbf4c0fbcb0 2024-02-11T16:59:59.246 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:59.242+0000 7fbf52886700 1 -- 172.21.15.37:0/3160248076 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbf4c0fbcb0 msgr2=0x7fbf4c0fc0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:59.246 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:59.242+0000 7fbf52886700 1 --2- 172.21.15.37:0/3160248076 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbf4c0fbcb0 0x7fbf4c0fc0d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7fbf3c00b5f0 tx=0x7fbf3c006ee0).stop 2024-02-11T16:59:59.246 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:59.242+0000 7fbf52886700 1 -- 172.21.15.37:0/3160248076 shutdown_connections 2024-02-11T16:59:59.247 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:59.242+0000 7fbf52886700 1 --2- 172.21.15.37:0/3160248076 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbf4c0fbcb0 0x7fbf4c0fc0d0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:59.247 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:59.242+0000 7fbf52886700 1 -- 172.21.15.37:0/3160248076 >> 172.21.15.37:0/3160248076 conn(0x7fbf4c0f74b0 msgr2=0x7fbf4c0f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T16:59:59.247 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:59.242+0000 7fbf52886700 1 -- 172.21.15.37:0/3160248076 shutdown_connections 2024-02-11T16:59:59.247 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:59.242+0000 7fbf52886700 1 -- 172.21.15.37:0/3160248076 wait complete. 2024-02-11T16:59:59.247 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:59.242+0000 7fbf52886700 1 Processor -- start 2024-02-11T16:59:59.247 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:59.243+0000 7fbf52886700 1 -- start start 2024-02-11T16:59:59.247 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:59.243+0000 7fbf52886700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbf4c0fbcb0 0x7fbf4c0725e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:59.248 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:59.243+0000 7fbf52886700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf4c070a60 con 0x7fbf4c0fbcb0 2024-02-11T16:59:59.248 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:59.243+0000 7fbf4bfff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbf4c0fbcb0 0x7fbf4c0725e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T16:59:59.249 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:59.243+0000 7fbf4bfff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbf4c0fbcb0 0x7fbf4c0725e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:51524/0 (socket says 172.21.15.37:51524) 2024-02-11T16:59:59.249 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:59.243+0000 7fbf4bfff700 1 -- 172.21.15.37:0/1272445692 learned_addr learned my addr 172.21.15.37:0/1272445692 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-02-11T16:59:59.249 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:59.243+0000 7fbf4bfff700 1 -- 172.21.15.37:0/1272445692 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf3c00b200 con 0x7fbf4c0fbcb0 2024-02-11T16:59:59.249 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:59.243+0000 7fbf4bfff700 1 --2- 172.21.15.37:0/1272445692 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbf4c0fbcb0 0x7fbf4c0725e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fbf3c00dfb0 tx=0x7fbf3c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T16:59:59.249 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:59.244+0000 7fbf48ff9700 1 -- 172.21.15.37:0/1272445692 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf3c011df0 con 0x7fbf4c0fbcb0 2024-02-11T16:59:59.249 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:59.244+0000 7fbf48ff9700 1 -- 172.21.15.37:0/1272445692 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbf3c01b8c0 con 0x7fbf4c0fbcb0 2024-02-11T16:59:59.249 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:59.244+0000 7fbf52886700 1 -- 172.21.15.37:0/1272445692 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf4c070c00 con 0x7fbf4c0fbcb0 2024-02-11T16:59:59.249 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:59.244+0000 7fbf48ff9700 1 -- 172.21.15.37:0/1272445692 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf3c01a8c0 con 0x7fbf4c0fbcb0 2024-02-11T16:59:59.249 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:59.244+0000 7fbf52886700 1 -- 172.21.15.37:0/1272445692 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf4c071010 con 0x7fbf4c0fbcb0 2024-02-11T16:59:59.249 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:59.245+0000 7fbf48ff9700 1 -- 172.21.15.37:0/1272445692 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fbf3c00f070 con 0x7fbf4c0fbcb0 2024-02-11T16:59:59.250 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:59.245+0000 7fbf52886700 1 -- 172.21.15.37:0/1272445692 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf4c045ad0 con 0x7fbf4c0fbcb0 2024-02-11T16:59:59.250 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:59.245+0000 7fbf48ff9700 1 --2- 172.21.15.37:0/1272445692 >> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] conn(0x7fbf3802feb0 0x7fbf38032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T16:59:59.250 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:59.245+0000 7fbf48ff9700 1 -- 172.21.15.37:0/1272445692 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fbf3c0410c0 con 0x7fbf4c0fbcb0 2024-02-11T16:59:59.250 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:59.245+0000 7fbf4b7fe700 1 -- 172.21.15.37:0/1272445692 >> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] conn(0x7fbf3802feb0 msgr2=0x7fbf38032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:6800/162023810 2024-02-11T16:59:59.250 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:59.245+0000 7fbf4b7fe700 1 --2- 172.21.15.37:0/1272445692 >> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] conn(0x7fbf3802feb0 0x7fbf38032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-11T16:59:59.250 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:59.249+0000 7fbf48ff9700 1 -- 172.21.15.37:0/1272445692 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbf3c010360 con 0x7fbf4c0fbcb0 2024-02-11T16:59:59.447 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:59.446+0000 7fbf4b7fe700 1 -- 172.21.15.37:0/1272445692 >> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] conn(0x7fbf3802feb0 msgr2=0x7fbf38032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:6800/162023810 2024-02-11T16:59:59.447 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:59.446+0000 7fbf4b7fe700 1 --2- 172.21.15.37:0/1272445692 >> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] conn(0x7fbf3802feb0 0x7fbf38032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-11T16:59:59.478 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:59.476+0000 7fbf52886700 1 -- 172.21.15.37:0/1272445692 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fbf4c110a70 con 0x7fbf4c0fbcb0 2024-02-11T16:59:59.478 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:59.476+0000 7fbf48ff9700 1 -- 172.21.15.37:0/1272445692 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v13) v1 ==== 56+0+104 (secure 0 0 0) 0x7fbf3c016070 con 0x7fbf4c0fbcb0 2024-02-11T16:59:59.478 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: { 2024-02-11T16:59:59.478 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "epoch": 13, 2024-02-11T16:59:59.478 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "available": true, 2024-02-11T16:59:59.478 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "active_name": "smithi037.eofnaq", 2024-02-11T16:59:59.478 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "num_standby": 0 2024-02-11T16:59:59.478 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: } 2024-02-11T16:59:59.480 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:59.478+0000 7fbf52886700 1 -- 172.21.15.37:0/1272445692 >> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] conn(0x7fbf3802feb0 msgr2=0x7fbf38032370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-11T16:59:59.480 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:59.478+0000 7fbf52886700 1 --2- 172.21.15.37:0/1272445692 >> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] conn(0x7fbf3802feb0 0x7fbf38032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-11T16:59:59.480 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:59.478+0000 7fbf52886700 1 -- 172.21.15.37:0/1272445692 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbf4c0fbcb0 msgr2=0x7fbf4c0725e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T16:59:59.480 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:59.478+0000 7fbf52886700 1 --2- 172.21.15.37:0/1272445692 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbf4c0fbcb0 0x7fbf4c0725e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fbf3c00dfb0 tx=0x7fbf3c006ee0).stop 2024-02-11T16:59:59.481 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:59.478+0000 7fbf52886700 1 -- 172.21.15.37:0/1272445692 shutdown_connections 2024-02-11T16:59:59.481 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:59.478+0000 7fbf52886700 1 --2- 172.21.15.37:0/1272445692 >> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] conn(0x7fbf3802feb0 0x7fbf38032370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-11T16:59:59.481 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:59.478+0000 7fbf52886700 1 --2- 172.21.15.37:0/1272445692 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbf4c0fbcb0 0x7fbf4c0725e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T16:59:59.481 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:59.478+0000 7fbf52886700 1 -- 172.21.15.37:0/1272445692 >> 172.21.15.37:0/1272445692 conn(0x7fbf4c0f74b0 msgr2=0x7fbf4c0f8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T16:59:59.481 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:59.478+0000 7fbf52886700 1 -- 172.21.15.37:0/1272445692 shutdown_connections 2024-02-11T16:59:59.481 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T16:59:59.478+0000 7fbf52886700 1 -- 172.21.15.37:0/1272445692 wait complete. 2024-02-11T17:00:00.104 INFO:teuthology.orchestra.run.smithi037.stderr:Waiting for the mgr to restart... 2024-02-11T17:00:00.105 INFO:teuthology.orchestra.run.smithi037.stderr:Waiting for mgr epoch 13... 2024-02-11T17:00:00.105 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi037 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpjftqjcr4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4ata0hlm:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2024-02-11T17:00:00.105 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 16:59:59 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T16:59:59.476968+0000 mon.smithi037 (mon.0) 90 : audit [DBG] from='client.? 172.21.15.37:0/1272445692' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-11T17:00:00.652 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:00.650+0000 7f3f5a1fb700 1 Processor -- start 2024-02-11T17:00:00.652 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:00.650+0000 7f3f5a1fb700 1 -- start start 2024-02-11T17:00:00.652 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:00.650+0000 7f3f5a1fb700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3f540703d0 0x7f3f540fe940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-11T17:00:00.653 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:00.650+0000 7f3f5a1fb700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f540fee30 con 0x7f3f540703d0 2024-02-11T17:00:00.654 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:00.650+0000 7f3f537fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3f540703d0 0x7f3f540fe940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:00:00.654 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:00.650+0000 7f3f537fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3f540703d0 0x7f3f540fe940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:51538/0 (socket says 172.21.15.37:51538) 2024-02-11T17:00:00.654 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:00.650+0000 7f3f537fe700 1 -- 172.21.15.37:0/3767968579 learned_addr learned my addr 172.21.15.37:0/3767968579 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-02-11T17:00:00.654 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:00.651+0000 7f3f537fe700 1 -- 172.21.15.37:0/3767968579 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f540fef70 con 0x7f3f540703d0 2024-02-11T17:00:00.654 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:00.651+0000 7f3f537fe700 1 --2- 172.21.15.37:0/3767968579 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3f540703d0 0x7f3f540fe940 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f3f4400b5f0 tx=0x7f3f44006ee0).ready entity=mon.0 client_cookie=ff9979e56bc415b server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:00:00.654 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:00.651+0000 7f3f527fc700 1 -- 172.21.15.37:0/3767968579 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f4400d600 con 0x7f3f540703d0 2024-02-11T17:00:00.654 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:00.651+0000 7f3f527fc700 1 -- 172.21.15.37:0/3767968579 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3f4400dc10 con 0x7f3f540703d0 2024-02-11T17:00:00.654 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:00.651+0000 7f3f527fc700 1 -- 172.21.15.37:0/3767968579 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f44011cd0 con 0x7f3f540703d0 2024-02-11T17:00:00.656 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:00.652+0000 7f3f5a1fb700 1 -- 172.21.15.37:0/3767968579 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3f540703d0 msgr2=0x7f3f540fe940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:00:00.656 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:00.652+0000 7f3f5a1fb700 1 --2- 172.21.15.37:0/3767968579 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3f540703d0 0x7f3f540fe940 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f3f4400b5f0 tx=0x7f3f44006ee0).stop 2024-02-11T17:00:00.657 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:00.652+0000 7f3f5a1fb700 1 -- 172.21.15.37:0/3767968579 shutdown_connections 2024-02-11T17:00:00.657 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:00.652+0000 7f3f5a1fb700 1 --2- 172.21.15.37:0/3767968579 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3f540703d0 0x7f3f540fe940 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:00:00.657 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:00.652+0000 7f3f5a1fb700 1 -- 172.21.15.37:0/3767968579 >> 172.21.15.37:0/3767968579 conn(0x7f3f540f63c0 msgr2=0x7f3f540f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:00:00.657 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:00.652+0000 7f3f5a1fb700 1 -- 172.21.15.37:0/3767968579 shutdown_connections 2024-02-11T17:00:00.657 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:00.652+0000 7f3f5a1fb700 1 -- 172.21.15.37:0/3767968579 wait complete. 2024-02-11T17:00:00.657 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:00.652+0000 7f3f5a1fb700 1 Processor -- start 2024-02-11T17:00:00.657 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:00.653+0000 7f3f5a1fb700 1 -- start start 2024-02-11T17:00:00.657 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:00.653+0000 7f3f5a1fb700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3f540703d0 0x7f3f5410aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:00:00.657 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:00.653+0000 7f3f5a1fb700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f5410af30 con 0x7f3f540703d0 2024-02-11T17:00:00.657 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:00.653+0000 7f3f537fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3f540703d0 0x7f3f5410aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:00:00.657 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:00.653+0000 7f3f537fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3f540703d0 0x7f3f5410aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:51548/0 (socket says 172.21.15.37:51548) 2024-02-11T17:00:00.657 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:00.653+0000 7f3f537fe700 1 -- 172.21.15.37:0/2636141397 learned_addr learned my addr 172.21.15.37:0/2636141397 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-02-11T17:00:00.657 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:00.653+0000 7f3f537fe700 1 -- 172.21.15.37:0/2636141397 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f4400b200 con 0x7f3f540703d0 2024-02-11T17:00:00.657 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:00.654+0000 7f3f537fe700 1 --2- 172.21.15.37:0/2636141397 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3f540703d0 0x7f3f5410aa40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f3f44006a40 tx=0x7f3f44006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:00:00.658 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:00.654+0000 7f3f3bfff700 1 -- 172.21.15.37:0/2636141397 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f44011c90 con 0x7f3f540703d0 2024-02-11T17:00:00.658 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:00.654+0000 7f3f3bfff700 1 -- 172.21.15.37:0/2636141397 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3f4401b8c0 con 0x7f3f540703d0 2024-02-11T17:00:00.658 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:00.654+0000 7f3f5a1fb700 1 -- 172.21.15.37:0/2636141397 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f5410b130 con 0x7f3f540703d0 2024-02-11T17:00:00.658 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:00.654+0000 7f3f3bfff700 1 -- 172.21.15.37:0/2636141397 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f4401a8c0 con 0x7f3f540703d0 2024-02-11T17:00:00.659 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:00.654+0000 7f3f5a1fb700 1 -- 172.21.15.37:0/2636141397 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f5410b540 con 0x7f3f540703d0 2024-02-11T17:00:00.659 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:00.655+0000 7f3f3bfff700 1 -- 172.21.15.37:0/2636141397 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f3f4400f070 con 0x7f3f540703d0 2024-02-11T17:00:00.659 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:00.655+0000 7f3f3bfff700 1 --2- 172.21.15.37:0/2636141397 >> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] conn(0x7f3f4002feb0 0x7f3f40032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:00:00.659 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:00.655+0000 7f3f5a1fb700 1 -- 172.21.15.37:0/2636141397 --> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3f54059100 con 0x7f3f4002feb0 2024-02-11T17:00:00.659 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:00.655+0000 7f3f3bfff700 1 -- 172.21.15.37:0/2636141397 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f3f44040a50 con 0x7f3f540703d0 2024-02-11T17:00:00.659 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:00.655+0000 7f3f52ffd700 1 -- 172.21.15.37:0/2636141397 >> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] conn(0x7f3f4002feb0 msgr2=0x7f3f40032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:6800/162023810 2024-02-11T17:00:00.659 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:00.655+0000 7f3f52ffd700 1 --2- 172.21.15.37:0/2636141397 >> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] conn(0x7f3f4002feb0 0x7f3f40032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-11T17:00:00.857 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:00.856+0000 7f3f52ffd700 1 -- 172.21.15.37:0/2636141397 >> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] conn(0x7f3f4002feb0 msgr2=0x7f3f40032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:6800/162023810 2024-02-11T17:00:00.857 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:00.856+0000 7f3f52ffd700 1 --2- 172.21.15.37:0/2636141397 >> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] conn(0x7f3f4002feb0 0x7f3f40032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-11T17:00:01.258 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:01.257+0000 7f3f52ffd700 1 -- 172.21.15.37:0/2636141397 >> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] conn(0x7f3f4002feb0 msgr2=0x7f3f40032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:6800/162023810 2024-02-11T17:00:01.258 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:01.257+0000 7f3f52ffd700 1 --2- 172.21.15.37:0/2636141397 >> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] conn(0x7f3f4002feb0 0x7f3f40032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-02-11T17:00:02.059 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:02.058+0000 7f3f52ffd700 1 -- 172.21.15.37:0/2636141397 >> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] conn(0x7f3f4002feb0 msgr2=0x7f3f40032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.37:6800/162023810 2024-02-11T17:00:02.059 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:02.058+0000 7f3f52ffd700 1 --2- 172.21.15.37:0/2636141397 >> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] conn(0x7f3f4002feb0 0x7f3f40032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-02-11T17:00:03.124 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:03.122+0000 7f3f3bfff700 1 -- 172.21.15.37:0/2636141397 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mgrmap(e 14) v1 ==== 35997+0+0 (secure 0 0 0) 0x7f3f44024550 con 0x7f3f540703d0 2024-02-11T17:00:03.125 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:03.123+0000 7f3f3bfff700 1 -- 172.21.15.37:0/2636141397 >> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] conn(0x7f3f4002feb0 msgr2=0x7f3f40032370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-11T17:00:03.125 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:03.123+0000 7f3f3bfff700 1 --2- 172.21.15.37:0/2636141397 >> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] conn(0x7f3f4002feb0 0x7f3f40032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-11T17:00:04.125 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:04.123+0000 7f3f3bfff700 1 -- 172.21.15.37:0/2636141397 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mgrmap(e 15) v1 ==== 36067+0+0 (secure 0 0 0) 0x7f3f4401b5e0 con 0x7f3f540703d0 2024-02-11T17:00:04.125 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:04.124+0000 7f3f3bfff700 1 --2- 172.21.15.37:0/2636141397 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f3f40033990 0x7f3f40035d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:00:04.125 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:04.124+0000 7f3f3bfff700 1 -- 172.21.15.37:0/2636141397 --> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3f54059100 con 0x7f3f40033990 2024-02-11T17:00:04.125 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:04.124+0000 7f3f52ffd700 1 --2- 172.21.15.37:0/2636141397 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f3f40033990 0x7f3f40035d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:00:04.126 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:04.125+0000 7f3f52ffd700 1 --2- 172.21.15.37:0/2636141397 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f3f40033990 0x7f3f40035d80 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f3f480036d0 tx=0x7f3f48006ca0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:00:04.126 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:04.125+0000 7f3f3bfff700 1 -- 172.21.15.37:0/2636141397 <== mgr.14162 v2:172.21.15.37:6800/3714037575 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f3f54059100 con 0x7f3f40033990 2024-02-11T17:00:04.132 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:04.131+0000 7f3f5a1fb700 1 -- 172.21.15.37:0/2636141397 --> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f3f54045ad0 con 0x7f3f40033990 2024-02-11T17:00:04.132 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:04.131+0000 7f3f3bfff700 1 -- 172.21.15.37:0/2636141397 <== mgr.14162 v2:172.21.15.37:6800/3714037575 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f3f54045ad0 con 0x7f3f40033990 2024-02-11T17:00:04.133 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: { 2024-02-11T17:00:04.134 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "mgrmap_epoch": 15, 2024-02-11T17:00:04.134 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: "initialized": true 2024-02-11T17:00:04.134 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: } 2024-02-11T17:00:04.134 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:04.132+0000 7f3f5a1fb700 1 -- 172.21.15.37:0/2636141397 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f3f40033990 msgr2=0x7f3f40035d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:00:04.134 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:04.132+0000 7f3f5a1fb700 1 --2- 172.21.15.37:0/2636141397 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f3f40033990 0x7f3f40035d80 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f3f480036d0 tx=0x7f3f48006ca0).stop 2024-02-11T17:00:04.134 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:04.132+0000 7f3f5a1fb700 1 -- 172.21.15.37:0/2636141397 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3f540703d0 msgr2=0x7f3f5410aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:00:04.134 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:04.132+0000 7f3f5a1fb700 1 --2- 172.21.15.37:0/2636141397 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3f540703d0 0x7f3f5410aa40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f3f44006a40 tx=0x7f3f44006ee0).stop 2024-02-11T17:00:04.135 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:04.132+0000 7f3f5a1fb700 1 -- 172.21.15.37:0/2636141397 shutdown_connections 2024-02-11T17:00:04.135 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:04.132+0000 7f3f5a1fb700 1 --2- 172.21.15.37:0/2636141397 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f3f40033990 0x7f3f40035d80 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:00:04.135 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:04.132+0000 7f3f5a1fb700 1 --2- 172.21.15.37:0/2636141397 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3f540703d0 0x7f3f5410aa40 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:00:04.135 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:04.132+0000 7f3f5a1fb700 1 --2- 172.21.15.37:0/2636141397 >> [v2:172.21.15.37:6800/162023810,v1:172.21.15.37:6801/162023810] conn(0x7f3f4002feb0 0x7f3f40032370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-11T17:00:04.135 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:04.132+0000 7f3f5a1fb700 1 -- 172.21.15.37:0/2636141397 >> 172.21.15.37:0/2636141397 conn(0x7f3f540f63c0 msgr2=0x7f3f540f7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:00:04.135 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:04.132+0000 7f3f5a1fb700 1 -- 172.21.15.37:0/2636141397 shutdown_connections 2024-02-11T17:00:04.135 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:04.132+0000 7f3f5a1fb700 1 -- 172.21.15.37:0/2636141397 wait complete. 2024-02-11T17:00:04.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:00:04 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:00:03.071539+0000 mon.smithi037 (mon.0) 91 : cluster [INF] Active manager daemon smithi037.eofnaq restarted 2024-02-11T17:00:04.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:00:04 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:00:03.071962+0000 mon.smithi037 (mon.0) 92 : cluster [INF] Activating manager daemon smithi037.eofnaq 2024-02-11T17:00:04.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:00:04 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:00:03.073661+0000 mon.smithi037 (mon.0) 93 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-02-11T17:00:04.434 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:00:04 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:00:03.124406+0000 mon.smithi037 (mon.0) 94 : cluster [DBG] mgrmap e14: smithi037.eofnaq(active, starting, since 0.0525516s) 2024-02-11T17:00:04.434 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:00:04 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:00:03.125459+0000 mon.smithi037 (mon.0) 95 : audit [DBG] from='mgr.14162 172.21.15.37:0/2745465012' entity='mgr.smithi037.eofnaq' cmd=[{"prefix": "mon metadata", "id": "smithi037"}]: dispatch 2024-02-11T17:00:04.434 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:00:04 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:00:03.126045+0000 mon.smithi037 (mon.0) 96 : audit [DBG] from='mgr.14162 172.21.15.37:0/2745465012' entity='mgr.smithi037.eofnaq' cmd=[{"prefix": "mgr metadata", "who": "smithi037.eofnaq", "id": "smithi037.eofnaq"}]: dispatch 2024-02-11T17:00:04.434 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:00:04 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:00:03.126891+0000 mon.smithi037 (mon.0) 97 : audit [DBG] from='mgr.14162 172.21.15.37:0/2745465012' entity='mgr.smithi037.eofnaq' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-11T17:00:04.434 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:00:04 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:00:03.127055+0000 mon.smithi037 (mon.0) 98 : audit [DBG] from='mgr.14162 172.21.15.37:0/2745465012' entity='mgr.smithi037.eofnaq' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-11T17:00:04.434 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:00:04 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:00:03.127184+0000 mon.smithi037 (mon.0) 99 : audit [DBG] from='mgr.14162 172.21.15.37:0/2745465012' entity='mgr.smithi037.eofnaq' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-11T17:00:04.434 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:00:04 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:00:03.141148+0000 mon.smithi037 (mon.0) 100 : cluster [INF] Manager daemon smithi037.eofnaq is now available 2024-02-11T17:00:04.434 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:00:04 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:00:03.244169+0000 mon.smithi037 (mon.0) 101 : audit [DBG] from='mgr.14162 172.21.15.37:0/2745465012' entity='mgr.smithi037.eofnaq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-11T17:00:04.434 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:00:04 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:00:03.246544+0000 mon.smithi037 (mon.0) 102 : audit [DBG] from='mgr.14162 172.21.15.37:0/2745465012' entity='mgr.smithi037.eofnaq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-11T17:00:04.434 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:00:04 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:00:03.264589+0000 mon.smithi037 (mon.0) 103 : audit [INF] from='mgr.14162 172.21.15.37:0/2745465012' entity='mgr.smithi037.eofnaq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi037.eofnaq/mirror_snapshot_schedule"}]: dispatch 2024-02-11T17:00:04.434 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:00:04 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:00:03.272926+0000 mon.smithi037 (mon.0) 104 : audit [INF] from='mgr.14162 172.21.15.37:0/2745465012' entity='mgr.smithi037.eofnaq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi037.eofnaq/trash_purge_schedule"}]: dispatch 2024-02-11T17:00:05.184 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:00:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:00:04.126106+0000 mgr.smithi037.eofnaq (mgr.14162) 1 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-11T17:00:05.184 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:00:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:00:04.126546+0000 mon.smithi037 (mon.0) 105 : cluster [DBG] mgrmap e15: smithi037.eofnaq(active, since 1.05469s) 2024-02-11T17:00:05.184 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:00:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:00:04.132443+0000 mgr.smithi037.eofnaq (mgr.14162) 2 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-11T17:00:05.184 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:00:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:00:04.653768+0000 mon.smithi037 (mon.0) 106 : audit [INF] from='mgr.14162 172.21.15.37:0/2745465012' entity='mgr.smithi037.eofnaq' 2024-02-11T17:00:05.207 INFO:teuthology.orchestra.run.smithi037.stderr:mgr epoch 13 is available 2024-02-11T17:00:05.207 INFO:teuthology.orchestra.run.smithi037.stderr:Generating a dashboard self-signed certificate... 2024-02-11T17:00:05.207 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi037 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpjftqjcr4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4ata0hlm:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 dashboard create-self-signed-cert 2024-02-11T17:00:05.923 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:05.918+0000 7efe4acd2700 1 Processor -- start 2024-02-11T17:00:05.923 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:05.918+0000 7efe4acd2700 1 -- start start 2024-02-11T17:00:05.923 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:05.918+0000 7efe4acd2700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efe44070390 0x7efe44100d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-11T17:00:05.923 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:05.918+0000 7efe4acd2700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe440fc770 con 0x7efe44070390 2024-02-11T17:00:05.923 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:05.918+0000 7efe48a6e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efe44070390 0x7efe44100d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:00:05.923 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:05.919+0000 7efe48a6e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efe44070390 0x7efe44100d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:51598/0 (socket says 172.21.15.37:51598) 2024-02-11T17:00:05.923 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:05.919+0000 7efe48a6e700 1 -- 172.21.15.37:0/2739058247 learned_addr learned my addr 172.21.15.37:0/2739058247 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-02-11T17:00:05.923 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:05.919+0000 7efe48a6e700 1 -- 172.21.15.37:0/2739058247 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe44101270 con 0x7efe44070390 2024-02-11T17:00:05.923 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:05.919+0000 7efe48a6e700 1 --2- 172.21.15.37:0/2739058247 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efe44070390 0x7efe44100d80 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7efe3800b5f0 tx=0x7efe38006ee0).ready entity=mon.0 client_cookie=45ac1a8fb5586a0b server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:00:05.923 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:05.919+0000 7efe437fe700 1 -- 172.21.15.37:0/2739058247 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe3800d600 con 0x7efe44070390 2024-02-11T17:00:05.923 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:05.919+0000 7efe437fe700 1 -- 172.21.15.37:0/2739058247 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efe3800dc10 con 0x7efe44070390 2024-02-11T17:00:05.924 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:05.919+0000 7efe437fe700 1 -- 172.21.15.37:0/2739058247 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe38011cd0 con 0x7efe44070390 2024-02-11T17:00:05.924 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:05.920+0000 7efe4acd2700 1 -- 172.21.15.37:0/2739058247 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efe44070390 msgr2=0x7efe44100d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:00:05.924 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:05.920+0000 7efe4acd2700 1 --2- 172.21.15.37:0/2739058247 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efe44070390 0x7efe44100d80 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7efe3800b5f0 tx=0x7efe38006ee0).stop 2024-02-11T17:00:05.924 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:05.920+0000 7efe4acd2700 1 -- 172.21.15.37:0/2739058247 shutdown_connections 2024-02-11T17:00:05.924 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:05.920+0000 7efe4acd2700 1 --2- 172.21.15.37:0/2739058247 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efe44070390 0x7efe44100d80 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:00:05.924 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:05.920+0000 7efe4acd2700 1 -- 172.21.15.37:0/2739058247 >> 172.21.15.37:0/2739058247 conn(0x7efe440f63c0 msgr2=0x7efe440f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:00:05.924 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:05.920+0000 7efe4acd2700 1 -- 172.21.15.37:0/2739058247 shutdown_connections 2024-02-11T17:00:05.924 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:05.920+0000 7efe4acd2700 1 -- 172.21.15.37:0/2739058247 wait complete. 2024-02-11T17:00:05.924 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:05.920+0000 7efe4acd2700 1 Processor -- start 2024-02-11T17:00:05.924 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:05.920+0000 7efe4acd2700 1 -- start start 2024-02-11T17:00:05.925 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:05.921+0000 7efe4acd2700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efe44070390 0x7efe4410cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:00:05.925 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:05.921+0000 7efe4acd2700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe4410d110 con 0x7efe44070390 2024-02-11T17:00:05.925 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:05.921+0000 7efe48a6e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efe44070390 0x7efe4410cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:00:05.925 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:05.921+0000 7efe48a6e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efe44070390 0x7efe4410cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:51612/0 (socket says 172.21.15.37:51612) 2024-02-11T17:00:05.925 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:05.921+0000 7efe48a6e700 1 -- 172.21.15.37:0/1555180311 learned_addr learned my addr 172.21.15.37:0/1555180311 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-02-11T17:00:05.925 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:05.921+0000 7efe48a6e700 1 -- 172.21.15.37:0/1555180311 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe3800b200 con 0x7efe44070390 2024-02-11T17:00:05.925 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:05.921+0000 7efe48a6e700 1 --2- 172.21.15.37:0/1555180311 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efe44070390 0x7efe4410cc20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7efe3800b1d0 tx=0x7efe38006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:00:05.925 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:05.921+0000 7efe417fa700 1 -- 172.21.15.37:0/1555180311 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe38011df0 con 0x7efe44070390 2024-02-11T17:00:05.925 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:05.921+0000 7efe417fa700 1 -- 172.21.15.37:0/1555180311 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efe3801b8c0 con 0x7efe44070390 2024-02-11T17:00:05.926 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:05.921+0000 7efe4acd2700 1 -- 172.21.15.37:0/1555180311 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe4410d310 con 0x7efe44070390 2024-02-11T17:00:05.926 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:05.921+0000 7efe417fa700 1 -- 172.21.15.37:0/1555180311 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe3801a830 con 0x7efe44070390 2024-02-11T17:00:05.926 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:05.921+0000 7efe4acd2700 1 -- 172.21.15.37:0/1555180311 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe4410d720 con 0x7efe44070390 2024-02-11T17:00:05.926 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:05.922+0000 7efe417fa700 1 -- 172.21.15.37:0/1555180311 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 15) v1 ==== 36067+0+0 (secure 0 0 0) 0x7efe3800f070 con 0x7efe44070390 2024-02-11T17:00:05.926 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:05.922+0000 7efe417fa700 1 --2- 172.21.15.37:0/1555180311 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7efe2802fda0 0x7efe28032260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:00:05.926 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:05.922+0000 7efe417fa700 1 -- 172.21.15.37:0/1555180311 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efe38040f00 con 0x7efe44070390 2024-02-11T17:00:05.926 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:05.922+0000 7efe43fff700 1 --2- 172.21.15.37:0/1555180311 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7efe2802fda0 0x7efe28032260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:00:05.926 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:05.923+0000 7efe43fff700 1 --2- 172.21.15.37:0/1555180311 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7efe2802fda0 0x7efe28032260 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7efe300096a0 tx=0x7efe30006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:00:05.926 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:05.923+0000 7efe4acd2700 1 -- 172.21.15.37:0/1555180311 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe44045ad0 con 0x7efe44070390 2024-02-11T17:00:05.928 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:05.927+0000 7efe417fa700 1 -- 172.21.15.37:0/1555180311 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efe38016070 con 0x7efe44070390 2024-02-11T17:00:06.033 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:06.031+0000 7efe417fa700 1 -- 172.21.15.37:0/1555180311 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7efe38024790 con 0x7efe44070390 2024-02-11T17:00:06.101 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:06.099+0000 7efe4acd2700 1 -- 172.21.15.37:0/1555180311 --> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7efe44106b40 con 0x7efe2802fda0 2024-02-11T17:00:06.153 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:06.151+0000 7efe417fa700 1 -- 172.21.15.37:0/1555180311 <== mgr.14162 v2:172.21.15.37:6800/3714037575 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7efe44106b40 con 0x7efe2802fda0 2024-02-11T17:00:06.153 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: Self-signed certificate created 2024-02-11T17:00:06.155 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:06.153+0000 7efe4acd2700 1 -- 172.21.15.37:0/1555180311 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7efe2802fda0 msgr2=0x7efe28032260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:00:06.155 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:06.153+0000 7efe4acd2700 1 --2- 172.21.15.37:0/1555180311 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7efe2802fda0 0x7efe28032260 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7efe300096a0 tx=0x7efe30006b40).stop 2024-02-11T17:00:06.156 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:06.153+0000 7efe4acd2700 1 -- 172.21.15.37:0/1555180311 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efe44070390 msgr2=0x7efe4410cc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:00:06.156 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:06.153+0000 7efe4acd2700 1 --2- 172.21.15.37:0/1555180311 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efe44070390 0x7efe4410cc20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7efe3800b1d0 tx=0x7efe38006ee0).stop 2024-02-11T17:00:06.156 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:06.154+0000 7efe4acd2700 1 -- 172.21.15.37:0/1555180311 shutdown_connections 2024-02-11T17:00:06.156 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:06.154+0000 7efe4acd2700 1 --2- 172.21.15.37:0/1555180311 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7efe2802fda0 0x7efe28032260 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:00:06.156 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:06.154+0000 7efe4acd2700 1 --2- 172.21.15.37:0/1555180311 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efe44070390 0x7efe4410cc20 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:00:06.156 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:06.154+0000 7efe4acd2700 1 -- 172.21.15.37:0/1555180311 >> 172.21.15.37:0/1555180311 conn(0x7efe440f63c0 msgr2=0x7efe440f7e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:00:06.156 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:06.154+0000 7efe4acd2700 1 -- 172.21.15.37:0/1555180311 shutdown_connections 2024-02-11T17:00:06.156 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:06.154+0000 7efe4acd2700 1 -- 172.21.15.37:0/1555180311 wait complete. 2024-02-11T17:00:06.867 INFO:teuthology.orchestra.run.smithi037.stderr:Creating initial admin user... 2024-02-11T17:00:06.867 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi037 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpjftqjcr4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4ata0hlm:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp4m5fru_e:/tmp/dashboard.pw:z docker.io/ceph/ceph:v16.2.4 dashboard ac-user-create admin -i /tmp/dashboard.pw administrator --force-password --pwd-update-required 2024-02-11T17:00:07.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:00:07 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:00:06.033382+0000 mon.smithi037 (mon.0) 107 : cluster [DBG] mgrmap e16: smithi037.eofnaq(active, since 2s) 2024-02-11T17:00:07.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:00:07 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:00:06.100860+0000 mgr.smithi037.eofnaq (mgr.14162) 3 : audit [DBG] from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-02-11T17:00:07.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:00:07 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:00:06.150403+0000 mon.smithi037 (mon.0) 108 : audit [INF] from='mgr.14162 172.21.15.37:0/2745465012' entity='mgr.smithi037.eofnaq' 2024-02-11T17:00:07.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:00:07 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:00:06.152271+0000 mon.smithi037 (mon.0) 109 : audit [INF] from='mgr.14162 172.21.15.37:0/2745465012' entity='mgr.smithi037.eofnaq' 2024-02-11T17:00:07.489 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:07.486+0000 7f333c6c6700 1 Processor -- start 2024-02-11T17:00:07.489 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:07.486+0000 7f333c6c6700 1 -- start start 2024-02-11T17:00:07.489 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:07.486+0000 7f333c6c6700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f33340fbcb0 0x7f33340fc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-11T17:00:07.489 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:07.486+0000 7f333c6c6700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33340fc5c0 con 0x7f33340fbcb0 2024-02-11T17:00:07.490 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:07.487+0000 7f333a462700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f33340fbcb0 0x7f33340fc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:00:07.490 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:07.487+0000 7f333a462700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f33340fbcb0 0x7f33340fc0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:51618/0 (socket says 172.21.15.37:51618) 2024-02-11T17:00:07.490 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:07.487+0000 7f333a462700 1 -- 172.21.15.37:0/4060043404 learned_addr learned my addr 172.21.15.37:0/4060043404 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-02-11T17:00:07.490 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:07.487+0000 7f333a462700 1 -- 172.21.15.37:0/4060043404 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33340fc700 con 0x7f33340fbcb0 2024-02-11T17:00:07.490 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:07.487+0000 7f333a462700 1 --2- 172.21.15.37:0/4060043404 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f33340fbcb0 0x7f33340fc0d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f332400b5f0 tx=0x7f3324006ee0).ready entity=mon.0 client_cookie=d6b12fbabaa7e9c4 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:00:07.490 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:07.487+0000 7f3339460700 1 -- 172.21.15.37:0/4060043404 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f332400d600 con 0x7f33340fbcb0 2024-02-11T17:00:07.490 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:07.487+0000 7f3339460700 1 -- 172.21.15.37:0/4060043404 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f332400dc10 con 0x7f33340fbcb0 2024-02-11T17:00:07.490 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:07.487+0000 7f3339460700 1 -- 172.21.15.37:0/4060043404 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3324011cd0 con 0x7f33340fbcb0 2024-02-11T17:00:07.490 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:07.488+0000 7f333c6c6700 1 -- 172.21.15.37:0/4060043404 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f33340fbcb0 msgr2=0x7f33340fc0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:00:07.490 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:07.488+0000 7f333c6c6700 1 --2- 172.21.15.37:0/4060043404 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f33340fbcb0 0x7f33340fc0d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f332400b5f0 tx=0x7f3324006ee0).stop 2024-02-11T17:00:07.490 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:07.488+0000 7f333c6c6700 1 -- 172.21.15.37:0/4060043404 shutdown_connections 2024-02-11T17:00:07.490 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:07.488+0000 7f333c6c6700 1 --2- 172.21.15.37:0/4060043404 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f33340fbcb0 0x7f33340fc0d0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:00:07.490 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:07.488+0000 7f333c6c6700 1 -- 172.21.15.37:0/4060043404 >> 172.21.15.37:0/4060043404 conn(0x7f33340f74b0 msgr2=0x7f33340f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:00:07.490 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:07.488+0000 7f333c6c6700 1 -- 172.21.15.37:0/4060043404 shutdown_connections 2024-02-11T17:00:07.491 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:07.488+0000 7f333c6c6700 1 -- 172.21.15.37:0/4060043404 wait complete. 2024-02-11T17:00:07.493 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:07.489+0000 7f333c6c6700 1 Processor -- start 2024-02-11T17:00:07.493 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:07.489+0000 7f333c6c6700 1 -- start start 2024-02-11T17:00:07.493 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:07.489+0000 7f333c6c6700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f33340fbcb0 0x7f33340725e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:00:07.493 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:07.489+0000 7f333c6c6700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3334070a60 con 0x7f33340fbcb0 2024-02-11T17:00:07.493 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:07.489+0000 7f333a462700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f33340fbcb0 0x7f33340725e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:00:07.493 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:07.489+0000 7f333a462700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f33340fbcb0 0x7f33340725e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:51626/0 (socket says 172.21.15.37:51626) 2024-02-11T17:00:07.493 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:07.489+0000 7f333a462700 1 -- 172.21.15.37:0/773946408 learned_addr learned my addr 172.21.15.37:0/773946408 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-02-11T17:00:07.494 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:07.489+0000 7f333a462700 1 -- 172.21.15.37:0/773946408 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f332400b200 con 0x7f33340fbcb0 2024-02-11T17:00:07.494 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:07.489+0000 7f333a462700 1 --2- 172.21.15.37:0/773946408 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f33340fbcb0 0x7f33340725e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f332400e040 tx=0x7f3324006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:00:07.494 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:07.490+0000 7f3322ffd700 1 -- 172.21.15.37:0/773946408 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f332401b410 con 0x7f33340fbcb0 2024-02-11T17:00:07.494 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:07.490+0000 7f3322ffd700 1 -- 172.21.15.37:0/773946408 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f332401ba20 con 0x7f33340fbcb0 2024-02-11T17:00:07.494 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:07.490+0000 7f3322ffd700 1 -- 172.21.15.37:0/773946408 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f332401a910 con 0x7f33340fbcb0 2024-02-11T17:00:07.494 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:07.490+0000 7f333c6c6700 1 -- 172.21.15.37:0/773946408 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3334070c00 con 0x7f33340fbcb0 2024-02-11T17:00:07.494 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:07.490+0000 7f333c6c6700 1 -- 172.21.15.37:0/773946408 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3334071010 con 0x7f33340fbcb0 2024-02-11T17:00:07.494 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:07.490+0000 7f3322ffd700 1 -- 172.21.15.37:0/773946408 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f332400f070 con 0x7f33340fbcb0 2024-02-11T17:00:07.494 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:07.491+0000 7f3322ffd700 1 --2- 172.21.15.37:0/773946408 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f331c02ff00 0x7f331c0323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:00:07.494 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:07.491+0000 7f3322ffd700 1 -- 172.21.15.37:0/773946408 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3324041430 con 0x7f33340fbcb0 2024-02-11T17:00:07.494 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:07.491+0000 7f3339c61700 1 --2- 172.21.15.37:0/773946408 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f331c02ff00 0x7f331c0323c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:00:07.494 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:07.491+0000 7f3320ff9700 1 -- 172.21.15.37:0/773946408 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3318004fa0 con 0x7f33340fbcb0 2024-02-11T17:00:07.494 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:07.491+0000 7f3339c61700 1 --2- 172.21.15.37:0/773946408 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f331c02ff00 0x7f331c0323c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f33300096a0 tx=0x7f3330006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:00:07.496 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:07.495+0000 7f3322ffd700 1 -- 172.21.15.37:0/773946408 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3324016070 con 0x7f33340fbcb0 2024-02-11T17:00:07.670 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:07.668+0000 7f3320ff9700 1 -- 172.21.15.37:0/773946408 --> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f3318002400 con 0x7f331c02ff00 2024-02-11T17:00:07.915 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:07.914+0000 7f3322ffd700 1 -- 172.21.15.37:0/773946408 <== mgr.14162 v2:172.21.15.37:6800/3714037575 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f3318002400 con 0x7f331c02ff00 2024-02-11T17:00:07.916 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$PpZ33CzevM.DiAKOciv6PeHUTOfEciaK/zBibf5QuL.khemV7DKbW", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1707670807, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-02-11T17:00:07.918 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:07.916+0000 7f3320ff9700 1 -- 172.21.15.37:0/773946408 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f331c02ff00 msgr2=0x7f331c0323c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:00:07.918 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:07.916+0000 7f3320ff9700 1 --2- 172.21.15.37:0/773946408 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f331c02ff00 0x7f331c0323c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f33300096a0 tx=0x7f3330006b40).stop 2024-02-11T17:00:07.918 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:07.916+0000 7f3320ff9700 1 -- 172.21.15.37:0/773946408 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f33340fbcb0 msgr2=0x7f33340725e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:00:07.918 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:07.916+0000 7f3320ff9700 1 --2- 172.21.15.37:0/773946408 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f33340fbcb0 0x7f33340725e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f332400e040 tx=0x7f3324006ee0).stop 2024-02-11T17:00:07.918 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:07.916+0000 7f3320ff9700 1 -- 172.21.15.37:0/773946408 shutdown_connections 2024-02-11T17:00:07.919 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:07.916+0000 7f3320ff9700 1 --2- 172.21.15.37:0/773946408 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f331c02ff00 0x7f331c0323c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:00:07.919 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:07.916+0000 7f3320ff9700 1 --2- 172.21.15.37:0/773946408 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f33340fbcb0 0x7f33340725e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:00:07.919 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:07.916+0000 7f3320ff9700 1 -- 172.21.15.37:0/773946408 >> 172.21.15.37:0/773946408 conn(0x7f33340f74b0 msgr2=0x7f33340f8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:00:07.919 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:07.917+0000 7f3320ff9700 1 -- 172.21.15.37:0/773946408 shutdown_connections 2024-02-11T17:00:07.919 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:07.917+0000 7f3320ff9700 1 -- 172.21.15.37:0/773946408 wait complete. 2024-02-11T17:00:08.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:00:08 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:00:07.670092+0000 mgr.smithi037.eofnaq (mgr.14162) 4 : audit [DBG] from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-11T17:00:08.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:00:08 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:00:07.914910+0000 mon.smithi037 (mon.0) 110 : audit [INF] from='mgr.14162 172.21.15.37:0/2745465012' entity='mgr.smithi037.eofnaq' 2024-02-11T17:00:08.552 INFO:teuthology.orchestra.run.smithi037.stderr:Fetching dashboard port number... 2024-02-11T17:00:08.552 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi037 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpjftqjcr4:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp4ata0hlm:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config get mgr mgr/dashboard/ssl_server_port 2024-02-11T17:00:09.082 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:09.039+0000 7fb6addb1700 1 Processor -- start 2024-02-11T17:00:09.082 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:09.039+0000 7fb6addb1700 1 -- start start 2024-02-11T17:00:09.082 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:09.039+0000 7fb6addb1700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb6a8070430 0x7fb6a80fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-11T17:00:09.083 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:09.039+0000 7fb6addb1700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a80fedd0 con 0x7fb6a8070430 2024-02-11T17:00:09.083 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:09.040+0000 7fb6a77fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb6a8070430 0x7fb6a80fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:00:09.083 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:09.040+0000 7fb6a77fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb6a8070430 0x7fb6a80fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:36226/0 (socket says 172.21.15.37:36226) 2024-02-11T17:00:09.083 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:09.040+0000 7fb6a77fe700 1 -- 172.21.15.37:0/2840575304 learned_addr learned my addr 172.21.15.37:0/2840575304 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-02-11T17:00:09.083 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:09.040+0000 7fb6a77fe700 1 -- 172.21.15.37:0/2840575304 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6a80fef10 con 0x7fb6a8070430 2024-02-11T17:00:09.083 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:09.040+0000 7fb6a77fe700 1 --2- 172.21.15.37:0/2840575304 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb6a8070430 0x7fb6a80fe8e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7fb69800b5f0 tx=0x7fb698006ee0).ready entity=mon.0 client_cookie=7057ea2df640f6c7 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:00:09.083 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:09.040+0000 7fb6a67fc700 1 -- 172.21.15.37:0/2840575304 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb69800d600 con 0x7fb6a8070430 2024-02-11T17:00:09.083 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:09.041+0000 7fb6a67fc700 1 -- 172.21.15.37:0/2840575304 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb69800dc10 con 0x7fb6a8070430 2024-02-11T17:00:09.083 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:09.041+0000 7fb6a67fc700 1 -- 172.21.15.37:0/2840575304 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb698011cd0 con 0x7fb6a8070430 2024-02-11T17:00:09.083 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:09.041+0000 7fb6addb1700 1 -- 172.21.15.37:0/2840575304 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb6a8070430 msgr2=0x7fb6a80fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:00:09.083 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:09.041+0000 7fb6addb1700 1 --2- 172.21.15.37:0/2840575304 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb6a8070430 0x7fb6a80fe8e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7fb69800b5f0 tx=0x7fb698006ee0).stop 2024-02-11T17:00:09.083 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:09.041+0000 7fb6addb1700 1 -- 172.21.15.37:0/2840575304 shutdown_connections 2024-02-11T17:00:09.083 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:09.041+0000 7fb6addb1700 1 --2- 172.21.15.37:0/2840575304 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb6a8070430 0x7fb6a80fe8e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:00:09.083 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:09.041+0000 7fb6addb1700 1 -- 172.21.15.37:0/2840575304 >> 172.21.15.37:0/2840575304 conn(0x7fb6a80f63c0 msgr2=0x7fb6a80f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:00:09.084 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:09.041+0000 7fb6addb1700 1 -- 172.21.15.37:0/2840575304 shutdown_connections 2024-02-11T17:00:09.084 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:09.041+0000 7fb6addb1700 1 -- 172.21.15.37:0/2840575304 wait complete. 2024-02-11T17:00:09.084 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:09.042+0000 7fb6addb1700 1 Processor -- start 2024-02-11T17:00:09.084 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:09.042+0000 7fb6addb1700 1 -- start start 2024-02-11T17:00:09.084 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:09.042+0000 7fb6addb1700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb6a8070430 0x7fb6a810ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:00:09.084 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:09.042+0000 7fb6addb1700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a810b020 con 0x7fb6a8070430 2024-02-11T17:00:09.084 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:09.042+0000 7fb6a77fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb6a8070430 0x7fb6a810ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:00:09.084 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:09.042+0000 7fb6a77fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb6a8070430 0x7fb6a810ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:36242/0 (socket says 172.21.15.37:36242) 2024-02-11T17:00:09.084 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:09.042+0000 7fb6a77fe700 1 -- 172.21.15.37:0/3288567318 learned_addr learned my addr 172.21.15.37:0/3288567318 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-02-11T17:00:09.084 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:09.043+0000 7fb6a77fe700 1 -- 172.21.15.37:0/3288567318 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb69800b200 con 0x7fb6a8070430 2024-02-11T17:00:09.084 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:09.043+0000 7fb6a77fe700 1 --2- 172.21.15.37:0/3288567318 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb6a8070430 0x7fb6a810ab30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fb69800dfe0 tx=0x7fb698006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:00:09.084 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:09.043+0000 7fb69ffff700 1 -- 172.21.15.37:0/3288567318 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb698011db0 con 0x7fb6a8070430 2024-02-11T17:00:09.084 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:09.043+0000 7fb6addb1700 1 -- 172.21.15.37:0/3288567318 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6a810b220 con 0x7fb6a8070430 2024-02-11T17:00:09.084 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:09.043+0000 7fb69ffff700 1 -- 172.21.15.37:0/3288567318 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb69801b8c0 con 0x7fb6a8070430 2024-02-11T17:00:09.085 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:09.043+0000 7fb69ffff700 1 -- 172.21.15.37:0/3288567318 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb69801a840 con 0x7fb6a8070430 2024-02-11T17:00:09.085 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:09.043+0000 7fb6addb1700 1 -- 172.21.15.37:0/3288567318 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6a810b630 con 0x7fb6a8070430 2024-02-11T17:00:09.085 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:09.044+0000 7fb69ffff700 1 -- 172.21.15.37:0/3288567318 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fb69800f070 con 0x7fb6a8070430 2024-02-11T17:00:09.085 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:09.044+0000 7fb69ffff700 1 --2- 172.21.15.37:0/3288567318 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb68802feb0 0x7fb688032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:00:09.085 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:09.044+0000 7fb69ffff700 1 -- 172.21.15.37:0/3288567318 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb698041170 con 0x7fb6a8070430 2024-02-11T17:00:09.085 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:09.044+0000 7fb6a6ffd700 1 --2- 172.21.15.37:0/3288567318 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb68802feb0 0x7fb688032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:00:09.085 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:09.044+0000 7fb6addb1700 1 -- 172.21.15.37:0/3288567318 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6a8059100 con 0x7fb6a8070430 2024-02-11T17:00:09.085 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:09.044+0000 7fb6a6ffd700 1 --2- 172.21.15.37:0/3288567318 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb68802feb0 0x7fb688032370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fb6900096a0 tx=0x7fb690006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:00:09.085 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:09.048+0000 7fb69ffff700 1 -- 172.21.15.37:0/3288567318 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb698016070 con 0x7fb6a8070430 2024-02-11T17:00:09.138 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:09.136+0000 7fb69ffff700 1 -- 172.21.15.37:0/3288567318 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb69801b410 con 0x7fb6a8070430 2024-02-11T17:00:09.211 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:09.210+0000 7fb6addb1700 1 -- 172.21.15.37:0/3288567318 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7fb6a80ffc00 con 0x7fb6a8070430 2024-02-11T17:00:09.211 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:09.210+0000 7fb69ffff700 1 -- 172.21.15.37:0/3288567318 <== mon.0 v2:172.21.15.37:3300/0 8 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v10) v1 ==== 112+0+5 (secure 0 0 0) 0x7fb698019300 con 0x7fb6a8070430 2024-02-11T17:00:09.211 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 8443 2024-02-11T17:00:09.213 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:09.212+0000 7fb6addb1700 1 -- 172.21.15.37:0/3288567318 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb68802feb0 msgr2=0x7fb688032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:00:09.214 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:09.212+0000 7fb6addb1700 1 --2- 172.21.15.37:0/3288567318 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb68802feb0 0x7fb688032370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fb6900096a0 tx=0x7fb690006b40).stop 2024-02-11T17:00:09.214 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:09.212+0000 7fb6addb1700 1 -- 172.21.15.37:0/3288567318 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb6a8070430 msgr2=0x7fb6a810ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:00:09.214 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:09.212+0000 7fb6addb1700 1 --2- 172.21.15.37:0/3288567318 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb6a8070430 0x7fb6a810ab30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fb69800dfe0 tx=0x7fb698006ee0).stop 2024-02-11T17:00:09.214 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:09.212+0000 7fb6addb1700 1 -- 172.21.15.37:0/3288567318 shutdown_connections 2024-02-11T17:00:09.214 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:09.212+0000 7fb6addb1700 1 --2- 172.21.15.37:0/3288567318 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb68802feb0 0x7fb688032370 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:00:09.214 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:09.212+0000 7fb6addb1700 1 --2- 172.21.15.37:0/3288567318 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb6a8070430 0x7fb6a810ab30 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:00:09.214 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:09.212+0000 7fb6addb1700 1 -- 172.21.15.37:0/3288567318 >> 172.21.15.37:0/3288567318 conn(0x7fb6a80f63c0 msgr2=0x7fb6a80f7fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:00:09.214 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:09.212+0000 7fb6addb1700 1 -- 172.21.15.37:0/3288567318 shutdown_connections 2024-02-11T17:00:09.214 INFO:teuthology.orchestra.run.smithi037.stderr:/usr/bin/ceph: 2024-02-11T17:00:09.212+0000 7fb6addb1700 1 -- 172.21.15.37:0/3288567318 wait complete. 2024-02-11T17:00:09.895 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: systemctl is-enabled firewalld.service 2024-02-11T17:00:09.905 INFO:teuthology.orchestra.run.smithi037.stderr:systemctl: disabled 2024-02-11T17:00:09.905 INFO:teuthology.orchestra.run.smithi037.stderr:Running command: systemctl is-active firewalld.service 2024-02-11T17:00:09.914 INFO:teuthology.orchestra.run.smithi037.stderr:systemctl: inactive 2024-02-11T17:00:09.914 INFO:teuthology.orchestra.run.smithi037.stderr:firewalld.service is not enabled 2024-02-11T17:00:09.915 INFO:teuthology.orchestra.run.smithi037.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-02-11T17:00:09.917 INFO:teuthology.orchestra.run.smithi037.stderr:Ceph Dashboard is now available at: 2024-02-11T17:00:09.918 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-02-11T17:00:09.918 INFO:teuthology.orchestra.run.smithi037.stderr: URL: https://smithi037.front.sepia.ceph.com:8443/ 2024-02-11T17:00:09.918 INFO:teuthology.orchestra.run.smithi037.stderr: User: admin 2024-02-11T17:00:09.918 INFO:teuthology.orchestra.run.smithi037.stderr: Password: mfys9nr8mh 2024-02-11T17:00:09.918 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-02-11T17:00:09.919 INFO:teuthology.orchestra.run.smithi037.stderr:You can access the Ceph CLI with: 2024-02-11T17:00:09.919 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-02-11T17:00:09.919 INFO:teuthology.orchestra.run.smithi037.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-02-11T17:00:09.919 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-02-11T17:00:09.919 INFO:teuthology.orchestra.run.smithi037.stderr:Please consider enabling telemetry to help improve Ceph: 2024-02-11T17:00:09.919 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-02-11T17:00:09.919 INFO:teuthology.orchestra.run.smithi037.stderr: ceph telemetry on 2024-02-11T17:00:09.919 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-02-11T17:00:09.919 INFO:teuthology.orchestra.run.smithi037.stderr:For more information see: 2024-02-11T17:00:09.919 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-02-11T17:00:09.919 INFO:teuthology.orchestra.run.smithi037.stderr: https://docs.ceph.com/docs/pacific/mgr/telemetry/ 2024-02-11T17:00:09.919 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-02-11T17:00:09.919 INFO:teuthology.orchestra.run.smithi037.stderr:Bootstrap complete. 2024-02-11T17:00:09.920 INFO:teuthology.orchestra.run.smithi037.stderr:Releasing lock 140264508033512 on /run/cephadm/c65cd888-c8fe-11ee-95b9-87774f69a715.lock 2024-02-11T17:00:09.920 INFO:teuthology.orchestra.run.smithi037.stderr:Lock 140264508033512 released on /run/cephadm/c65cd888-c8fe-11ee-95b9-87774f69a715.lock 2024-02-11T17:00:09.956 INFO:tasks.cephadm:Fetching config... 2024-02-11T17:00:09.956 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-02-11T17:00:09.956 DEBUG:teuthology.orchestra.run.smithi037:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-02-11T17:00:10.016 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-02-11T17:00:10.017 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-02-11T17:00:10.017 DEBUG:teuthology.orchestra.run.smithi037:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-02-11T17:00:10.075 INFO:tasks.cephadm:Fetching mon keyring... 2024-02-11T17:00:10.075 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-02-11T17:00:10.076 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715/mon.smithi037/keyring of=/dev/stdout 2024-02-11T17:00:10.145 INFO:tasks.cephadm:Fetching pub ssh key... 2024-02-11T17:00:10.145 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-02-11T17:00:10.146 DEBUG:teuthology.orchestra.run.smithi037:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-02-11T17:00:10.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:00:10 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:00:09.138419+0000 mon.smithi037 (mon.0) 111 : cluster [DBG] mgrmap e17: smithi037.eofnaq(active, since 6s) 2024-02-11T17:00:10.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:00:10 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:00:09.210895+0000 mon.smithi037 (mon.0) 112 : audit [DBG] from='client.? 172.21.15.37:0/3288567318' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-02-11T17:00:10.201 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-02-11T17:00:10.201 DEBUG:teuthology.orchestra.run.smithi037:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC9xY3F4EhclTI591iTKYgp9F5DirExl+ULrjgVq4dbUkf1WQV+3aXnHVqpeMJofYEn092/MypoL7W/pD8nJTMB/+hAd//v7oYwctXuyi+tER0uMXObwwupp1efJ4+NoObioCjcvFAaRjxEoOk2vRhqZ+O/d2KQ5/KljMVXj9RreE+D3i2aKiSwFniA0l5Tr4ZhK/02LmncgkbOMPf/PeVvxvP6XQ6dwjyIZW6NHvMGDGQ/U3s/LBc0ramuH5HGh2C/UqF+kyND+5ZSwwWOpd8oOY9Wlef9yGgciilQWJVN6BiFZIoxZi4wG5XX4ohlnngTHMXqE+DDbNAXkIWizN1WO8d3kOk4DJD2lQYubjsX//EmKfJz2ZAajktj3P+cNahgJZGY0TL8/4+rXZL9xUqdfUBrrZIgLBY4urLdNhcj6nooBTIqBFpArim6eaGXe11K1gYfaGgi0zin3yh8pOu/AqkVYVFSuzSpYg7474+yvioQ8IJIb1XJVQ61Fz+Z1XE= ceph-c65cd888-c8fe-11ee-95b9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-11T17:00:10.280 INFO:teuthology.orchestra.run.smithi037.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC9xY3F4EhclTI591iTKYgp9F5DirExl+ULrjgVq4dbUkf1WQV+3aXnHVqpeMJofYEn092/MypoL7W/pD8nJTMB/+hAd//v7oYwctXuyi+tER0uMXObwwupp1efJ4+NoObioCjcvFAaRjxEoOk2vRhqZ+O/d2KQ5/KljMVXj9RreE+D3i2aKiSwFniA0l5Tr4ZhK/02LmncgkbOMPf/PeVvxvP6XQ6dwjyIZW6NHvMGDGQ/U3s/LBc0ramuH5HGh2C/UqF+kyND+5ZSwwWOpd8oOY9Wlef9yGgciilQWJVN6BiFZIoxZi4wG5XX4ohlnngTHMXqE+DDbNAXkIWizN1WO8d3kOk4DJD2lQYubjsX//EmKfJz2ZAajktj3P+cNahgJZGY0TL8/4+rXZL9xUqdfUBrrZIgLBY4urLdNhcj6nooBTIqBFpArim6eaGXe11K1gYfaGgi0zin3yh8pOu/AqkVYVFSuzSpYg7474+yvioQ8IJIb1XJVQ61Fz+Z1XE= ceph-c65cd888-c8fe-11ee-95b9-87774f69a715 2024-02-11T17:00:10.293 DEBUG:teuthology.orchestra.run.smithi178:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC9xY3F4EhclTI591iTKYgp9F5DirExl+ULrjgVq4dbUkf1WQV+3aXnHVqpeMJofYEn092/MypoL7W/pD8nJTMB/+hAd//v7oYwctXuyi+tER0uMXObwwupp1efJ4+NoObioCjcvFAaRjxEoOk2vRhqZ+O/d2KQ5/KljMVXj9RreE+D3i2aKiSwFniA0l5Tr4ZhK/02LmncgkbOMPf/PeVvxvP6XQ6dwjyIZW6NHvMGDGQ/U3s/LBc0ramuH5HGh2C/UqF+kyND+5ZSwwWOpd8oOY9Wlef9yGgciilQWJVN6BiFZIoxZi4wG5XX4ohlnngTHMXqE+DDbNAXkIWizN1WO8d3kOk4DJD2lQYubjsX//EmKfJz2ZAajktj3P+cNahgJZGY0TL8/4+rXZL9xUqdfUBrrZIgLBY4urLdNhcj6nooBTIqBFpArim6eaGXe11K1gYfaGgi0zin3yh8pOu/AqkVYVFSuzSpYg7474+yvioQ8IJIb1XJVQ61Fz+Z1XE= ceph-c65cd888-c8fe-11ee-95b9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-11T17:00:10.341 INFO:teuthology.orchestra.run.smithi178.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC9xY3F4EhclTI591iTKYgp9F5DirExl+ULrjgVq4dbUkf1WQV+3aXnHVqpeMJofYEn092/MypoL7W/pD8nJTMB/+hAd//v7oYwctXuyi+tER0uMXObwwupp1efJ4+NoObioCjcvFAaRjxEoOk2vRhqZ+O/d2KQ5/KljMVXj9RreE+D3i2aKiSwFniA0l5Tr4ZhK/02LmncgkbOMPf/PeVvxvP6XQ6dwjyIZW6NHvMGDGQ/U3s/LBc0ramuH5HGh2C/UqF+kyND+5ZSwwWOpd8oOY9Wlef9yGgciilQWJVN6BiFZIoxZi4wG5XX4ohlnngTHMXqE+DDbNAXkIWizN1WO8d3kOk4DJD2lQYubjsX//EmKfJz2ZAajktj3P+cNahgJZGY0TL8/4+rXZL9xUqdfUBrrZIgLBY4urLdNhcj6nooBTIqBFpArim6eaGXe11K1gYfaGgi0zin3yh8pOu/AqkVYVFSuzSpYg7474+yvioQ8IJIb1XJVQ61Fz+Z1XE= ceph-c65cd888-c8fe-11ee-95b9-87774f69a715 2024-02-11T17:00:10.352 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-02-11T17:00:12.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:12.901+0000 7f95c9233700 1 -- 172.21.15.37:0/3383234738 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95c40f5410 msgr2=0x7f95c40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:00:12.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:12.901+0000 7f95c9233700 1 --2- 172.21.15.37:0/3383234738 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95c40f5410 0x7f95c40f5830 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f95b8009fb0 tx=0x7f95b8009b30).stop 2024-02-11T17:00:12.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:12.901+0000 7f95c9233700 1 -- 172.21.15.37:0/3383234738 shutdown_connections 2024-02-11T17:00:12.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:12.901+0000 7f95c9233700 1 --2- 172.21.15.37:0/3383234738 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95c40f5410 0x7f95c40f5830 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:00:12.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:12.901+0000 7f95c9233700 1 -- 172.21.15.37:0/3383234738 >> 172.21.15.37:0/3383234738 conn(0x7f95c40f0b70 msgr2=0x7f95c40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:00:12.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:12.901+0000 7f95c9233700 1 -- 172.21.15.37:0/3383234738 shutdown_connections 2024-02-11T17:00:12.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:12.901+0000 7f95c9233700 1 -- 172.21.15.37:0/3383234738 wait complete. 2024-02-11T17:00:12.904 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:12.902+0000 7f95c9233700 1 Processor -- start 2024-02-11T17:00:12.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:12.902+0000 7f95c9233700 1 -- start start 2024-02-11T17:00:12.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:12.902+0000 7f95c9233700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95c40f5410 0x7f95c410b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:00:12.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:12.902+0000 7f95c9233700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95c4107240 con 0x7f95c40f5410 2024-02-11T17:00:12.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:12.902+0000 7f95c2d9d700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95c40f5410 0x7f95c410b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:00:12.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:12.902+0000 7f95c2d9d700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95c40f5410 0x7f95c410b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:36258/0 (socket says 172.21.15.37:36258) 2024-02-11T17:00:12.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:12.902+0000 7f95c2d9d700 1 -- 172.21.15.37:0/2101702669 learned_addr learned my addr 172.21.15.37:0/2101702669 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-02-11T17:00:12.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:12.902+0000 7f95c2d9d700 1 -- 172.21.15.37:0/2101702669 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95b8005040 con 0x7f95c40f5410 2024-02-11T17:00:12.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:12.902+0000 7f95c2d9d700 1 --2- 172.21.15.37:0/2101702669 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95c40f5410 0x7f95c410b850 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f95b8013510 tx=0x7f95b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:00:12.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:12.903+0000 7f95ab7fe700 1 -- 172.21.15.37:0/2101702669 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f95b8009320 con 0x7f95c40f5410 2024-02-11T17:00:12.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:12.903+0000 7f95ab7fe700 1 -- 172.21.15.37:0/2101702669 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f95b8013f00 con 0x7f95c40f5410 2024-02-11T17:00:12.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:12.903+0000 7f95c9233700 1 -- 172.21.15.37:0/2101702669 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95c4060220 con 0x7f95c40f5410 2024-02-11T17:00:12.905 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:12.903+0000 7f95ab7fe700 1 -- 172.21.15.37:0/2101702669 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f95b8014d90 con 0x7f95c40f5410 2024-02-11T17:00:12.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:12.903+0000 7f95c9233700 1 -- 172.21.15.37:0/2101702669 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95c410bd40 con 0x7f95c40f5410 2024-02-11T17:00:12.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:12.903+0000 7f95ab7fe700 1 -- 172.21.15.37:0/2101702669 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f95b80263a0 con 0x7f95c40f5410 2024-02-11T17:00:12.906 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:12.904+0000 7f95c9233700 1 -- 172.21.15.37:0/2101702669 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95c4045ad0 con 0x7f95c40f5410 2024-02-11T17:00:12.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:12.904+0000 7f95ab7fe700 1 --2- 172.21.15.37:0/2101702669 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f95ac02ff60 0x7f95ac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:00:12.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:12.904+0000 7f95ab7fe700 1 -- 172.21.15.37:0/2101702669 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f95b80413a0 con 0x7f95c40f5410 2024-02-11T17:00:12.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:12.904+0000 7f95c259c700 1 --2- 172.21.15.37:0/2101702669 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f95ac02ff60 0x7f95ac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:00:12.907 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:12.904+0000 7f95c259c700 1 --2- 172.21.15.37:0/2101702669 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f95ac02ff60 0x7f95ac032420 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f95b40096a0 tx=0x7f95b4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:00:12.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:12.908+0000 7f95ab7fe700 1 -- 172.21.15.37:0/2101702669 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f95b80126f0 con 0x7f95c40f5410 2024-02-11T17:00:13.074 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:13.073+0000 7f95c9233700 1 -- 172.21.15.37:0/2101702669 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f95c40fae60 con 0x7f95c40f5410 2024-02-11T17:00:13.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:13.076+0000 7f95ab7fe700 1 -- 172.21.15.37:0/2101702669 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v11)=0 v11) v1 ==== 125+0+0 (secure 0 0 0) 0x7f95b8012330 con 0x7f95c40f5410 2024-02-11T17:00:13.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:13.078+0000 7f95c9233700 1 -- 172.21.15.37:0/2101702669 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f95ac02ff60 msgr2=0x7f95ac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:00:13.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:13.078+0000 7f95c9233700 1 --2- 172.21.15.37:0/2101702669 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f95ac02ff60 0x7f95ac032420 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f95b40096a0 tx=0x7f95b4006b40).stop 2024-02-11T17:00:13.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:13.078+0000 7f95c9233700 1 -- 172.21.15.37:0/2101702669 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95c40f5410 msgr2=0x7f95c410b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:00:13.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:13.078+0000 7f95c9233700 1 --2- 172.21.15.37:0/2101702669 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95c40f5410 0x7f95c410b850 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f95b8013510 tx=0x7f95b800b040).stop 2024-02-11T17:00:13.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:13.078+0000 7f95c9233700 1 -- 172.21.15.37:0/2101702669 shutdown_connections 2024-02-11T17:00:13.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:13.078+0000 7f95c9233700 1 --2- 172.21.15.37:0/2101702669 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f95ac02ff60 0x7f95ac032420 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:00:13.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:13.078+0000 7f95c9233700 1 --2- 172.21.15.37:0/2101702669 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95c40f5410 0x7f95c410b850 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:00:13.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:13.078+0000 7f95c9233700 1 -- 172.21.15.37:0/2101702669 >> 172.21.15.37:0/2101702669 conn(0x7f95c40f0b70 msgr2=0x7f95c40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:00:13.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:13.078+0000 7f95c9233700 1 -- 172.21.15.37:0/2101702669 shutdown_connections 2024-02-11T17:00:13.080 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:13.079+0000 7f95c9233700 1 -- 172.21.15.37:0/2101702669 wait complete. 2024-02-11T17:00:13.730 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi178 2024-02-11T17:00:13.730 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-02-11T17:00:13.730 DEBUG:teuthology.orchestra.run.smithi178:> dd of=/etc/ceph/ceph.conf 2024-02-11T17:00:13.752 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2024-02-11T17:00:13.752 DEBUG:teuthology.orchestra.run.smithi178:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-11T17:00:13.813 INFO:tasks.cephadm:Adding host smithi178 to orchestrator... 2024-02-11T17:00:13.813 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph orch host add smithi178 2024-02-11T17:00:14.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:00:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:00:12.958688+0000 mon.smithi037 (mon.0) 113 : audit [INF] from='mgr.14162 172.21.15.37:0/2745465012' entity='mgr.smithi037.eofnaq' 2024-02-11T17:00:14.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:00:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:00:13.077360+0000 mon.smithi037 (mon.0) 114 : audit [INF] from='client.? 172.21.15.37:0/2101702669' entity='client.admin' 2024-02-11T17:00:14.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:00:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:00:13.407461+0000 mon.smithi037 (mon.0) 115 : audit [INF] from='mgr.14162 172.21.15.37:0/2745465012' entity='mgr.smithi037.eofnaq' 2024-02-11T17:00:14.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:00:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cephadm 2024-02-11T17:00:13.413913+0000 mgr.smithi037.eofnaq (mgr.14162) 5 : cephadm [INF] Deploying daemon alertmanager.smithi037 on smithi037 2024-02-11T17:00:24.434 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:00:24 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:00:23.128253+0000 mgr.smithi037.eofnaq (mgr.14162) 6 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:00:26.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:00:26 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:00:25.128627+0000 mgr.smithi037.eofnaq (mgr.14162) 7 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:00:28.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:00:28 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:00:27.129018+0000 mgr.smithi037.eofnaq (mgr.14162) 8 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:00:30.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:00:30 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:00:29.129370+0000 mgr.smithi037.eofnaq (mgr.14162) 9 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:00:32.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:00:32 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:00:31.129778+0000 mgr.smithi037.eofnaq (mgr.14162) 10 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:00:34.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:00:34 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:00:33.130048+0000 mgr.smithi037.eofnaq (mgr.14162) 11 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:00:35.624 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:35.621+0000 7fb323286700 1 -- 172.21.15.178:0/1095351837 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb31c0f5410 msgr2=0x7fb31c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:00:35.624 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:35.621+0000 7fb323286700 1 --2- 172.21.15.178:0/1095351837 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb31c0f5410 0x7fb31c0f5830 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fb30c009fb0 tx=0x7fb30c009b30).stop 2024-02-11T17:00:35.624 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:35.621+0000 7fb323286700 1 -- 172.21.15.178:0/1095351837 shutdown_connections 2024-02-11T17:00:35.624 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:35.621+0000 7fb323286700 1 --2- 172.21.15.178:0/1095351837 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb31c0f5410 0x7fb31c0f5830 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:00:35.624 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:35.621+0000 7fb323286700 1 -- 172.21.15.178:0/1095351837 >> 172.21.15.178:0/1095351837 conn(0x7fb31c0f0b70 msgr2=0x7fb31c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:00:35.624 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:35.622+0000 7fb323286700 1 -- 172.21.15.178:0/1095351837 shutdown_connections 2024-02-11T17:00:35.625 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:35.622+0000 7fb323286700 1 -- 172.21.15.178:0/1095351837 wait complete. 2024-02-11T17:00:35.625 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:35.622+0000 7fb323286700 1 Processor -- start 2024-02-11T17:00:35.625 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:35.622+0000 7fb323286700 1 -- start start 2024-02-11T17:00:35.625 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:35.622+0000 7fb323286700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb31c105720 0x7fb31c105b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:00:35.625 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:35.622+0000 7fb323286700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb31c106030 con 0x7fb31c105720 2024-02-11T17:00:35.627 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:35.623+0000 7fb321022700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb31c105720 0x7fb31c105b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:00:35.627 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:35.623+0000 7fb321022700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb31c105720 0x7fb31c105b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:39012/0 (socket says 172.21.15.178:39012) 2024-02-11T17:00:35.627 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:35.623+0000 7fb321022700 1 -- 172.21.15.178:0/2437033187 learned_addr learned my addr 172.21.15.178:0/2437033187 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:00:35.627 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:35.623+0000 7fb321022700 1 -- 172.21.15.178:0/2437033187 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb30c005040 con 0x7fb31c105720 2024-02-11T17:00:35.627 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:35.623+0000 7fb321022700 1 --2- 172.21.15.178:0/2437033187 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb31c105720 0x7fb31c105b40 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fb30c009f80 tx=0x7fb30c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:00:35.627 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:35.623+0000 7fb311ffb700 1 -- 172.21.15.178:0/2437033187 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb30c0136c0 con 0x7fb31c105720 2024-02-11T17:00:35.627 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:35.623+0000 7fb311ffb700 1 -- 172.21.15.178:0/2437033187 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb30c0095f0 con 0x7fb31c105720 2024-02-11T17:00:35.627 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:35.623+0000 7fb323286700 1 -- 172.21.15.178:0/2437033187 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb31c106230 con 0x7fb31c105720 2024-02-11T17:00:35.627 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:35.623+0000 7fb311ffb700 1 -- 172.21.15.178:0/2437033187 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb30c014420 con 0x7fb31c105720 2024-02-11T17:00:35.627 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:35.624+0000 7fb323286700 1 -- 172.21.15.178:0/2437033187 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb31c106d80 con 0x7fb31c105720 2024-02-11T17:00:35.627 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:35.625+0000 7fb311ffb700 1 -- 172.21.15.178:0/2437033187 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb30c025060 con 0x7fb31c105720 2024-02-11T17:00:35.627 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:35.625+0000 7fb323286700 1 -- 172.21.15.178:0/2437033187 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb31c059100 con 0x7fb31c105720 2024-02-11T17:00:35.627 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:35.625+0000 7fb311ffb700 1 --2- 172.21.15.178:0/2437033187 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb30802ff60 0x7fb308032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:00:35.628 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:35.625+0000 7fb311ffb700 1 -- 172.21.15.178:0/2437033187 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb30c040da0 con 0x7fb31c105720 2024-02-11T17:00:35.628 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:35.625+0000 7fb320821700 1 --2- 172.21.15.178:0/2437033187 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb30802ff60 0x7fb308032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:00:35.628 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:35.625+0000 7fb320821700 1 --2- 172.21.15.178:0/2437033187 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb30802ff60 0x7fb308032420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fb3180096a0 tx=0x7fb318006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:00:35.631 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:35.630+0000 7fb311ffb700 1 -- 172.21.15.178:0/2437033187 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb30c01cb60 con 0x7fb31c105720 2024-02-11T17:00:35.801 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:35.799+0000 7fb323286700 1 -- 172.21.15.178:0/2437033187 --> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi178", "target": ["mon-mgr", ""]}) v1 -- 0x7fb31c107560 con 0x7fb30802ff60 2024-02-11T17:00:36.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:00:36 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:00:35.130385+0000 mgr.smithi037.eofnaq (mgr.14162) 12 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:00:36.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:00:36 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:00:35.801570+0000 mgr.smithi037.eofnaq (mgr.14162) 13 : audit [DBG] from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi178", "target": ["mon-mgr", ""]}]: dispatch 2024-02-11T17:00:36.396 INFO:teuthology.orchestra.run.smithi178.stdout:Added host 'smithi178' 2024-02-11T17:00:36.396 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:36.393+0000 7fb311ffb700 1 -- 172.21.15.178:0/2437033187 <== mgr.14162 v2:172.21.15.37:6800/3714037575 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7fb31c107560 con 0x7fb30802ff60 2024-02-11T17:00:36.397 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:36.396+0000 7fb323286700 1 -- 172.21.15.178:0/2437033187 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb30802ff60 msgr2=0x7fb308032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:00:36.398 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:36.396+0000 7fb323286700 1 --2- 172.21.15.178:0/2437033187 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb30802ff60 0x7fb308032420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fb3180096a0 tx=0x7fb318006b40).stop 2024-02-11T17:00:36.398 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:36.396+0000 7fb323286700 1 -- 172.21.15.178:0/2437033187 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb31c105720 msgr2=0x7fb31c105b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:00:36.398 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:36.396+0000 7fb323286700 1 --2- 172.21.15.178:0/2437033187 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb31c105720 0x7fb31c105b40 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fb30c009f80 tx=0x7fb30c009b30).stop 2024-02-11T17:00:36.398 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:36.396+0000 7fb323286700 1 -- 172.21.15.178:0/2437033187 shutdown_connections 2024-02-11T17:00:36.398 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:36.396+0000 7fb323286700 1 --2- 172.21.15.178:0/2437033187 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb30802ff60 0x7fb308032420 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:00:36.398 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:36.396+0000 7fb323286700 1 --2- 172.21.15.178:0/2437033187 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb31c105720 0x7fb31c105b40 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:00:36.398 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:36.396+0000 7fb323286700 1 -- 172.21.15.178:0/2437033187 >> 172.21.15.178:0/2437033187 conn(0x7fb31c0f0b70 msgr2=0x7fb31c0f1680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:00:36.398 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:36.396+0000 7fb323286700 1 -- 172.21.15.178:0/2437033187 shutdown_connections 2024-02-11T17:00:36.398 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:36.396+0000 7fb323286700 1 -- 172.21.15.178:0/2437033187 wait complete. 2024-02-11T17:00:37.051 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph orch host ls --format=json 2024-02-11T17:00:37.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:00:37 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:00:36.393824+0000 mon.smithi037 (mon.0) 116 : audit [INF] from='mgr.14162 172.21.15.37:0/2745465012' entity='mgr.smithi037.eofnaq' 2024-02-11T17:00:37.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:00:37 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cephadm 2024-02-11T17:00:36.394241+0000 mgr.smithi037.eofnaq (mgr.14162) 14 : cephadm [INF] Added host smithi178 2024-02-11T17:00:38.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:00:38 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:00:37.130884+0000 mgr.smithi037.eofnaq (mgr.14162) 15 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:00:38.771 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:38.768+0000 7f896ce25700 1 -- 172.21.15.178:0/580006308 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f89680f6510 msgr2=0x7f89680f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:00:38.771 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:38.768+0000 7f896ce25700 1 --2- 172.21.15.178:0/580006308 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f89680f6510 0x7f89680f6930 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f895c009fb0 tx=0x7f895c009b30).stop 2024-02-11T17:00:38.771 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:38.768+0000 7f896ce25700 1 -- 172.21.15.178:0/580006308 shutdown_connections 2024-02-11T17:00:38.771 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:38.768+0000 7f896ce25700 1 --2- 172.21.15.178:0/580006308 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f89680f6510 0x7f89680f6930 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:00:38.772 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:38.768+0000 7f896ce25700 1 -- 172.21.15.178:0/580006308 >> 172.21.15.178:0/580006308 conn(0x7f89680f1c70 msgr2=0x7f89680f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:00:38.772 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:38.768+0000 7f896ce25700 1 -- 172.21.15.178:0/580006308 shutdown_connections 2024-02-11T17:00:38.772 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:38.768+0000 7f896ce25700 1 -- 172.21.15.178:0/580006308 wait complete. 2024-02-11T17:00:38.772 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:38.769+0000 7f896ce25700 1 Processor -- start 2024-02-11T17:00:38.772 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:38.769+0000 7f896ce25700 1 -- start start 2024-02-11T17:00:38.772 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:38.769+0000 7f896ce25700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f89680f6510 0x7f8968106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:00:38.772 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:38.769+0000 7f896ce25700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8968106c20 con 0x7f89680f6510 2024-02-11T17:00:38.772 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:38.769+0000 7f896659c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f89680f6510 0x7f8968106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:00:38.772 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:38.769+0000 7f896659c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f89680f6510 0x7f8968106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:39020/0 (socket says 172.21.15.178:39020) 2024-02-11T17:00:38.772 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:38.769+0000 7f896659c700 1 -- 172.21.15.178:0/1021958170 learned_addr learned my addr 172.21.15.178:0/1021958170 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:00:38.772 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:38.770+0000 7f896659c700 1 -- 172.21.15.178:0/1021958170 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f895c005040 con 0x7f89680f6510 2024-02-11T17:00:38.772 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:38.770+0000 7f896659c700 1 --2- 172.21.15.178:0/1021958170 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f89680f6510 0x7f8968106730 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f895c013510 tx=0x7f895c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:00:38.774 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:38.770+0000 7f8956ffd700 1 -- 172.21.15.178:0/1021958170 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f895c009320 con 0x7f89680f6510 2024-02-11T17:00:38.774 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:38.770+0000 7f896ce25700 1 -- 172.21.15.178:0/1021958170 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8968106e20 con 0x7f89680f6510 2024-02-11T17:00:38.774 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:38.770+0000 7f8956ffd700 1 -- 172.21.15.178:0/1021958170 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f895c013f00 con 0x7f89680f6510 2024-02-11T17:00:38.774 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:38.770+0000 7f8956ffd700 1 -- 172.21.15.178:0/1021958170 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f895c014d90 con 0x7f89680f6510 2024-02-11T17:00:38.774 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:38.770+0000 7f896ce25700 1 -- 172.21.15.178:0/1021958170 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89681072b0 con 0x7f89680f6510 2024-02-11T17:00:38.774 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:38.771+0000 7f8956ffd700 1 -- 172.21.15.178:0/1021958170 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f895c0095f0 con 0x7f89680f6510 2024-02-11T17:00:38.774 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:38.771+0000 7f896ce25700 1 -- 172.21.15.178:0/1021958170 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8968045ad0 con 0x7f89680f6510 2024-02-11T17:00:38.774 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:38.771+0000 7f8956ffd700 1 --2- 172.21.15.178:0/1021958170 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f895002ff60 0x7f8950032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:00:38.775 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:38.771+0000 7f8956ffd700 1 -- 172.21.15.178:0/1021958170 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f895c041000 con 0x7f89680f6510 2024-02-11T17:00:38.775 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:38.771+0000 7f8965d9b700 1 --2- 172.21.15.178:0/1021958170 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f895002ff60 0x7f8950032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:00:38.775 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:38.772+0000 7f8965d9b700 1 --2- 172.21.15.178:0/1021958170 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f895002ff60 0x7f8950032420 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f89580096a0 tx=0x7f8958006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:00:38.777 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:38.776+0000 7f8956ffd700 1 -- 172.21.15.178:0/1021958170 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f895c014420 con 0x7f89680f6510 2024-02-11T17:00:38.945 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:38.944+0000 7f896ce25700 1 -- 172.21.15.178:0/1021958170 --> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f89680f2e70 con 0x7f895002ff60 2024-02-11T17:00:38.947 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:38.946+0000 7f8956ffd700 1 -- 172.21.15.178:0/1021958170 <== mgr.14162 v2:172.21.15.37:6800/3714037575 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+156 (secure 0 0 0) 0x7f89680f2e70 con 0x7f895002ff60 2024-02-11T17:00:38.947 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:00:38.948 INFO:teuthology.orchestra.run.smithi178.stdout:[{"addr": "172.21.15.37", "hostname": "smithi037", "labels": [], "status": ""}, {"addr": "smithi178", "hostname": "smithi178", "labels": [], "status": ""}] 2024-02-11T17:00:38.949 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:38.947+0000 7f896ce25700 1 -- 172.21.15.178:0/1021958170 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f895002ff60 msgr2=0x7f8950032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:00:38.949 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:38.947+0000 7f896ce25700 1 --2- 172.21.15.178:0/1021958170 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f895002ff60 0x7f8950032420 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f89580096a0 tx=0x7f8958006b40).stop 2024-02-11T17:00:38.949 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:38.947+0000 7f896ce25700 1 -- 172.21.15.178:0/1021958170 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f89680f6510 msgr2=0x7f8968106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:00:38.949 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:38.947+0000 7f896ce25700 1 --2- 172.21.15.178:0/1021958170 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f89680f6510 0x7f8968106730 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f895c013510 tx=0x7f895c00b040).stop 2024-02-11T17:00:38.949 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:38.947+0000 7f896ce25700 1 -- 172.21.15.178:0/1021958170 shutdown_connections 2024-02-11T17:00:38.949 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:38.947+0000 7f896ce25700 1 --2- 172.21.15.178:0/1021958170 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f895002ff60 0x7f8950032420 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:00:38.949 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:38.947+0000 7f896ce25700 1 --2- 172.21.15.178:0/1021958170 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f89680f6510 0x7f8968106730 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:00:38.949 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:38.947+0000 7f896ce25700 1 -- 172.21.15.178:0/1021958170 >> 172.21.15.178:0/1021958170 conn(0x7f89680f1c70 msgr2=0x7f89680f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:00:38.949 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:38.947+0000 7f896ce25700 1 -- 172.21.15.178:0/1021958170 shutdown_connections 2024-02-11T17:00:38.949 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:38.947+0000 7f896ce25700 1 -- 172.21.15.178:0/1021958170 wait complete. 2024-02-11T17:00:39.622 INFO:tasks.cephadm:Setting crush tunables to default 2024-02-11T17:00:39.622 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph osd crush tunables default 2024-02-11T17:00:40.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:00:40 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:00:38.946263+0000 mgr.smithi037.eofnaq (mgr.14162) 16 : audit [DBG] from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-11T17:00:40.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:00:40 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:00:39.131224+0000 mgr.smithi037.eofnaq (mgr.14162) 17 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:00:41.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:41.400+0000 7f674b59e700 1 -- 172.21.15.37:0/2246872681 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f674c0601e0 msgr2=0x7f674c0f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:00:41.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:41.400+0000 7f674b59e700 1 --2- 172.21.15.37:0/2246872681 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f674c0601e0 0x7f674c0f8f20 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f673c009fb0 tx=0x7f673c009b30).stop 2024-02-11T17:00:41.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:41.401+0000 7f674b59e700 1 -- 172.21.15.37:0/2246872681 shutdown_connections 2024-02-11T17:00:41.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:41.401+0000 7f674b59e700 1 --2- 172.21.15.37:0/2246872681 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f674c0601e0 0x7f674c0f8f20 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:00:41.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:41.401+0000 7f674b59e700 1 -- 172.21.15.37:0/2246872681 >> 172.21.15.37:0/2246872681 conn(0x7f674c0f09a0 msgr2=0x7f674c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:00:41.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:41.401+0000 7f674b59e700 1 -- 172.21.15.37:0/2246872681 shutdown_connections 2024-02-11T17:00:41.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:41.401+0000 7f674b59e700 1 -- 172.21.15.37:0/2246872681 wait complete. 2024-02-11T17:00:41.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:41.401+0000 7f674b59e700 1 Processor -- start 2024-02-11T17:00:41.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:41.401+0000 7f674b59e700 1 -- start start 2024-02-11T17:00:41.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:41.401+0000 7f674b59e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f674c0601e0 0x7f674c0f69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:00:41.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:41.401+0000 7f674b59e700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f674c0f4e20 con 0x7f674c0601e0 2024-02-11T17:00:41.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:41.401+0000 7f674a59c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f674c0601e0 0x7f674c0f69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:00:41.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:41.401+0000 7f674a59c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f674c0601e0 0x7f674c0f69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:42046/0 (socket says 172.21.15.37:42046) 2024-02-11T17:00:41.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:41.401+0000 7f674a59c700 1 -- 172.21.15.37:0/1078401632 learned_addr learned my addr 172.21.15.37:0/1078401632 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-02-11T17:00:41.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:41.402+0000 7f674a59c700 1 -- 172.21.15.37:0/1078401632 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f673c005040 con 0x7f674c0601e0 2024-02-11T17:00:41.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:41.402+0000 7f674a59c700 1 --2- 172.21.15.37:0/1078401632 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f674c0601e0 0x7f674c0f69a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f673c013510 tx=0x7f673c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:00:41.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:41.402+0000 7f673affd700 1 -- 172.21.15.37:0/1078401632 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f673c009320 con 0x7f674c0601e0 2024-02-11T17:00:41.405 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:41.402+0000 7f673affd700 1 -- 172.21.15.37:0/1078401632 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f673c013f00 con 0x7f674c0601e0 2024-02-11T17:00:41.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:41.402+0000 7f674b59e700 1 -- 172.21.15.37:0/1078401632 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f674c0f4fc0 con 0x7f674c0601e0 2024-02-11T17:00:41.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:41.402+0000 7f673affd700 1 -- 172.21.15.37:0/1078401632 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f673c014d90 con 0x7f674c0601e0 2024-02-11T17:00:41.406 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:41.402+0000 7f674b59e700 1 -- 172.21.15.37:0/1078401632 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f674c0f5450 con 0x7f674c0601e0 2024-02-11T17:00:41.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:41.403+0000 7f673affd700 1 -- 172.21.15.37:0/1078401632 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f673c0095f0 con 0x7f674c0601e0 2024-02-11T17:00:41.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:41.403+0000 7f673affd700 1 --2- 172.21.15.37:0/1078401632 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f673402ffb0 0x7f6734032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:00:41.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:41.403+0000 7f673affd700 1 -- 172.21.15.37:0/1078401632 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f673c040f90 con 0x7f674c0601e0 2024-02-11T17:00:41.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:41.403+0000 7f674b59e700 1 -- 172.21.15.37:0/1078401632 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f674c059100 con 0x7f674c0601e0 2024-02-11T17:00:41.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:41.403+0000 7f6749d9b700 1 --2- 172.21.15.37:0/1078401632 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f673402ffb0 0x7f6734032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:00:41.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:41.403+0000 7f6749d9b700 1 --2- 172.21.15.37:0/1078401632 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f673402ffb0 0x7f6734032470 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f67400096a0 tx=0x7f6740006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:00:41.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:41.407+0000 7f673affd700 1 -- 172.21.15.37:0/1078401632 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f673c014420 con 0x7f674c0601e0 2024-02-11T17:00:41.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:41.572+0000 7f674b59e700 1 -- 172.21.15.37:0/1078401632 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f674c045ad0 con 0x7f674c0601e0 2024-02-11T17:00:42.404 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:42.402+0000 7f673affd700 1 -- 172.21.15.37:0/1078401632 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v5) v1 ==== 124+0+0 (secure 0 0 0) 0x7f673c02b030 con 0x7f674c0601e0 2024-02-11T17:00:42.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:42.404+0000 7f674b59e700 1 -- 172.21.15.37:0/1078401632 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f673402ffb0 msgr2=0x7f6734032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:00:42.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:42.404+0000 7f674b59e700 1 --2- 172.21.15.37:0/1078401632 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f673402ffb0 0x7f6734032470 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f67400096a0 tx=0x7f6740006b40).stop 2024-02-11T17:00:42.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:42.404+0000 7f674b59e700 1 -- 172.21.15.37:0/1078401632 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f674c0601e0 msgr2=0x7f674c0f69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:00:42.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:42.404+0000 7f674b59e700 1 --2- 172.21.15.37:0/1078401632 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f674c0601e0 0x7f674c0f69a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f673c013510 tx=0x7f673c00b040).stop 2024-02-11T17:00:42.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:42.405+0000 7f674b59e700 1 -- 172.21.15.37:0/1078401632 shutdown_connections 2024-02-11T17:00:42.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:42.405+0000 7f674b59e700 1 --2- 172.21.15.37:0/1078401632 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f673402ffb0 0x7f6734032470 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:00:42.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:42.405+0000 7f674b59e700 1 --2- 172.21.15.37:0/1078401632 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f674c0601e0 0x7f674c0f69a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:00:42.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:42.405+0000 7f674b59e700 1 -- 172.21.15.37:0/1078401632 >> 172.21.15.37:0/1078401632 conn(0x7f674c0f09a0 msgr2=0x7f674c0f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:00:42.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:42.405+0000 7f674b59e700 1 -- 172.21.15.37:0/1078401632 shutdown_connections 2024-02-11T17:00:42.407 INFO:teuthology.orchestra.run.smithi037.stderr:2024-02-11T17:00:42.405+0000 7f674b59e700 1 -- 172.21.15.37:0/1078401632 wait complete. 2024-02-11T17:00:42.408 INFO:teuthology.orchestra.run.smithi037.stderr:adjusted tunables profile to default 2024-02-11T17:00:42.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:00:42 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:00:41.131514+0000 mgr.smithi037.eofnaq (mgr.14162) 18 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:00:42.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:00:42 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:00:41.573854+0000 mon.smithi037 (mon.0) 117 : audit [INF] from='client.? 172.21.15.37:0/1078401632' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-11T17:00:43.110 INFO:tasks.cephadm:Adding mon.smithi037 on smithi037 2024-02-11T17:00:43.111 INFO:tasks.cephadm:Adding mon.smithi178 on smithi178 2024-02-11T17:00:43.111 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph orch apply mon '2;smithi037:172.21.15.37=smithi037;smithi178:172.21.15.178=smithi178' 2024-02-11T17:00:43.684 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:00:43 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:00:42.404119+0000 mon.smithi037 (mon.0) 118 : audit [INF] from='client.? 172.21.15.37:0/1078401632' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-11T17:00:43.684 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:00:43 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:00:42.404176+0000 mon.smithi037 (mon.0) 119 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2024-02-11T17:00:44.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:00:44 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:00:43.131842+0000 mgr.smithi037.eofnaq (mgr.14162) 19 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:00:45.013 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:45.010+0000 7f266a113700 1 -- 172.21.15.178:0/617375316 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26640f5410 msgr2=0x7f26640f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:00:45.013 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:45.010+0000 7f266a113700 1 --2- 172.21.15.178:0/617375316 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26640f5410 0x7f26640f5830 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f2654009fb0 tx=0x7f2654009b30).stop 2024-02-11T17:00:45.013 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:45.010+0000 7f266a113700 1 -- 172.21.15.178:0/617375316 shutdown_connections 2024-02-11T17:00:45.013 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:45.010+0000 7f266a113700 1 --2- 172.21.15.178:0/617375316 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26640f5410 0x7f26640f5830 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:00:45.013 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:45.010+0000 7f266a113700 1 -- 172.21.15.178:0/617375316 >> 172.21.15.178:0/617375316 conn(0x7f26640f0b70 msgr2=0x7f26640f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:00:45.013 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:45.010+0000 7f266a113700 1 -- 172.21.15.178:0/617375316 shutdown_connections 2024-02-11T17:00:45.013 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:45.010+0000 7f266a113700 1 -- 172.21.15.178:0/617375316 wait complete. 2024-02-11T17:00:45.013 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:45.010+0000 7f266a113700 1 Processor -- start 2024-02-11T17:00:45.013 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:45.010+0000 7f266a113700 1 -- start start 2024-02-11T17:00:45.014 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:45.011+0000 7f266a113700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26640f5410 0x7f2664106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:00:45.014 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:45.011+0000 7f266a113700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2664106c10 con 0x7f26640f5410 2024-02-11T17:00:45.014 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:45.011+0000 7f26637fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26640f5410 0x7f2664106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:00:45.014 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:45.011+0000 7f26637fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26640f5410 0x7f2664106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:34978/0 (socket says 172.21.15.178:34978) 2024-02-11T17:00:45.014 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:45.011+0000 7f26637fe700 1 -- 172.21.15.178:0/1484652919 learned_addr learned my addr 172.21.15.178:0/1484652919 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:00:45.014 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:45.011+0000 7f26637fe700 1 -- 172.21.15.178:0/1484652919 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2654005040 con 0x7f26640f5410 2024-02-11T17:00:45.014 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:45.011+0000 7f26637fe700 1 --2- 172.21.15.178:0/1484652919 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26640f5410 0x7f2664106720 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f2654013510 tx=0x7f265400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:00:45.015 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:45.011+0000 7f264bfff700 1 -- 172.21.15.178:0/1484652919 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2654009320 con 0x7f26640f5410 2024-02-11T17:00:45.015 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:45.012+0000 7f266a113700 1 -- 172.21.15.178:0/1484652919 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2664106e10 con 0x7f26640f5410 2024-02-11T17:00:45.015 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:45.012+0000 7f264bfff700 1 -- 172.21.15.178:0/1484652919 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2654013f00 con 0x7f26640f5410 2024-02-11T17:00:45.015 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:45.012+0000 7f264bfff700 1 -- 172.21.15.178:0/1484652919 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2654014d90 con 0x7f26640f5410 2024-02-11T17:00:45.015 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:45.012+0000 7f266a113700 1 -- 172.21.15.178:0/1484652919 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26641072a0 con 0x7f26640f5410 2024-02-11T17:00:45.016 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:45.012+0000 7f264bfff700 1 -- 172.21.15.178:0/1484652919 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f26540095f0 con 0x7f26640f5410 2024-02-11T17:00:45.016 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:45.013+0000 7f266a113700 1 -- 172.21.15.178:0/1484652919 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2664045ad0 con 0x7f26640f5410 2024-02-11T17:00:45.016 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:45.013+0000 7f264bfff700 1 --2- 172.21.15.178:0/1484652919 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f264c02ffb0 0x7f264c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:00:45.016 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:45.013+0000 7f264bfff700 1 -- 172.21.15.178:0/1484652919 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2654041050 con 0x7f26640f5410 2024-02-11T17:00:45.016 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:45.013+0000 7f2662ffd700 1 --2- 172.21.15.178:0/1484652919 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f264c02ffb0 0x7f264c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:00:45.016 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:45.013+0000 7f2662ffd700 1 --2- 172.21.15.178:0/1484652919 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f264c02ffb0 0x7f264c032470 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f26580096a0 tx=0x7f2658006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:00:45.019 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:45.017+0000 7f264bfff700 1 -- 172.21.15.178:0/1484652919 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2654014420 con 0x7f26640f5410 2024-02-11T17:00:45.193 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:45.191+0000 7f266a113700 1 -- 172.21.15.178:0/1484652919 --> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi037:172.21.15.37=smithi037;smithi178:172.21.15.178=smithi178", "target": ["mon-mgr", ""]}) v1 -- 0x7f26640f2a80 con 0x7f264c02ffb0 2024-02-11T17:00:45.198 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:45.197+0000 7f264bfff700 1 -- 172.21.15.178:0/1484652919 <== mgr.14162 v2:172.21.15.37:6800/3714037575 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f26640f2a80 con 0x7f264c02ffb0 2024-02-11T17:00:45.198 INFO:teuthology.orchestra.run.smithi178.stdout:Scheduled mon update... 2024-02-11T17:00:45.200 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:45.198+0000 7f266a113700 1 -- 172.21.15.178:0/1484652919 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f264c02ffb0 msgr2=0x7f264c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:00:45.200 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:45.198+0000 7f266a113700 1 --2- 172.21.15.178:0/1484652919 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f264c02ffb0 0x7f264c032470 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f26580096a0 tx=0x7f2658006b40).stop 2024-02-11T17:00:45.200 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:45.198+0000 7f266a113700 1 -- 172.21.15.178:0/1484652919 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26640f5410 msgr2=0x7f2664106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:00:45.200 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:45.198+0000 7f266a113700 1 --2- 172.21.15.178:0/1484652919 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26640f5410 0x7f2664106720 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f2654013510 tx=0x7f265400b040).stop 2024-02-11T17:00:45.200 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:45.198+0000 7f266a113700 1 -- 172.21.15.178:0/1484652919 shutdown_connections 2024-02-11T17:00:45.200 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:45.198+0000 7f266a113700 1 --2- 172.21.15.178:0/1484652919 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f264c02ffb0 0x7f264c032470 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:00:45.201 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:45.198+0000 7f266a113700 1 --2- 172.21.15.178:0/1484652919 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26640f5410 0x7f2664106720 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:00:45.201 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:45.198+0000 7f266a113700 1 -- 172.21.15.178:0/1484652919 >> 172.21.15.178:0/1484652919 conn(0x7f26640f0b70 msgr2=0x7f266405fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:00:45.201 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:45.198+0000 7f266a113700 1 -- 172.21.15.178:0/1484652919 shutdown_connections 2024-02-11T17:00:45.201 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:45.198+0000 7f266a113700 1 -- 172.21.15.178:0/1484652919 wait complete. 2024-02-11T17:00:46.197 DEBUG:teuthology.orchestra.run.smithi178:mon.smithi178> sudo journalctl -f -n 0 -u ceph-c65cd888-c8fe-11ee-95b9-87774f69a715@mon.smithi178.service 2024-02-11T17:00:46.200 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:00:46.200 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:00:46.232 INFO:journalctl@ceph.mon.smithi178.smithi178.stdout:-- Logs begin at Sun 2024-02-11 16:48:50 UTC. -- 2024-02-11T17:00:46.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:00:46 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:00:45.132207+0000 mgr.smithi037.eofnaq (mgr.14162) 20 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:00:46.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:00:46 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:00:45.193983+0000 mgr.smithi037.eofnaq (mgr.14162) 21 : audit [DBG] from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi037:172.21.15.37=smithi037;smithi178:172.21.15.178=smithi178", "target": ["mon-mgr", ""]}]: dispatch 2024-02-11T17:00:46.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:00:46 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cephadm 2024-02-11T17:00:45.195489+0000 mgr.smithi037.eofnaq (mgr.14162) 22 : cephadm [INF] Saving service mon spec with placement smithi037:172.21.15.37=smithi037;smithi178:172.21.15.178=smithi178;count:2 2024-02-11T17:00:46.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:00:46 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:00:45.197763+0000 mon.smithi037 (mon.0) 120 : audit [INF] from='mgr.14162 172.21.15.37:0/2745465012' entity='mgr.smithi037.eofnaq' 2024-02-11T17:00:47.951 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:47.940+0000 7f4b0eb56700 1 -- 172.21.15.178:0/1483990637 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4b08060280 msgr2=0x7f4b080f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:00:47.951 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:47.940+0000 7f4b0eb56700 1 --2- 172.21.15.178:0/1483990637 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4b08060280 0x7f4b080f90f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f4afc009fb0 tx=0x7f4afc009b30).stop 2024-02-11T17:00:47.951 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:47.940+0000 7f4b0eb56700 1 -- 172.21.15.178:0/1483990637 shutdown_connections 2024-02-11T17:00:47.951 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:47.940+0000 7f4b0eb56700 1 --2- 172.21.15.178:0/1483990637 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4b08060280 0x7f4b080f90f0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:00:47.951 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:47.940+0000 7f4b0eb56700 1 -- 172.21.15.178:0/1483990637 >> 172.21.15.178:0/1483990637 conn(0x7f4b080f0b70 msgr2=0x7f4b080f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:00:47.951 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:47.940+0000 7f4b0eb56700 1 -- 172.21.15.178:0/1483990637 shutdown_connections 2024-02-11T17:00:47.951 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:47.940+0000 7f4b0eb56700 1 -- 172.21.15.178:0/1483990637 wait complete. 2024-02-11T17:00:47.951 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:47.941+0000 7f4b0eb56700 1 Processor -- start 2024-02-11T17:00:47.951 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:47.941+0000 7f4b0eb56700 1 -- start start 2024-02-11T17:00:47.952 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:47.941+0000 7f4b0eb56700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4b08060280 0x7f4b081066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:00:47.952 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:47.941+0000 7f4b0eb56700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b08106bd0 con 0x7f4b08060280 2024-02-11T17:00:47.952 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:47.941+0000 7f4b0c8f2700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4b08060280 0x7f4b081066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:00:47.952 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:47.941+0000 7f4b0c8f2700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4b08060280 0x7f4b081066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:34994/0 (socket says 172.21.15.178:34994) 2024-02-11T17:00:47.952 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:47.941+0000 7f4b0c8f2700 1 -- 172.21.15.178:0/1506004735 learned_addr learned my addr 172.21.15.178:0/1506004735 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:00:47.952 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:47.942+0000 7f4b0c8f2700 1 -- 172.21.15.178:0/1506004735 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4afc005040 con 0x7f4b08060280 2024-02-11T17:00:47.952 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:47.942+0000 7f4b0c8f2700 1 --2- 172.21.15.178:0/1506004735 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4b08060280 0x7f4b081066e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f4afc013510 tx=0x7f4afc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:00:47.952 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:47.942+0000 7f4b057fa700 1 -- 172.21.15.178:0/1506004735 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4afc009320 con 0x7f4b08060280 2024-02-11T17:00:47.952 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:47.942+0000 7f4b0eb56700 1 -- 172.21.15.178:0/1506004735 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b08106dd0 con 0x7f4b08060280 2024-02-11T17:00:47.952 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:47.942+0000 7f4b057fa700 1 -- 172.21.15.178:0/1506004735 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4afc013f00 con 0x7f4b08060280 2024-02-11T17:00:47.953 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:47.942+0000 7f4b057fa700 1 -- 172.21.15.178:0/1506004735 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4afc014dc0 con 0x7f4b08060280 2024-02-11T17:00:47.953 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:47.942+0000 7f4b0eb56700 1 -- 172.21.15.178:0/1506004735 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b08107260 con 0x7f4b08060280 2024-02-11T17:00:47.953 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:47.943+0000 7f4b0eb56700 1 -- 172.21.15.178:0/1506004735 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b08045ad0 con 0x7f4b08060280 2024-02-11T17:00:47.953 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:47.943+0000 7f4b057fa700 1 -- 172.21.15.178:0/1506004735 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4afc0095f0 con 0x7f4b08060280 2024-02-11T17:00:47.953 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:47.943+0000 7f4b057fa700 1 --2- 172.21.15.178:0/1506004735 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f4af802ff60 0x7f4af8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:00:47.953 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:47.943+0000 7f4b057fa700 1 -- 172.21.15.178:0/1506004735 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4afc041000 con 0x7f4b08060280 2024-02-11T17:00:47.953 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:47.943+0000 7f4b07fff700 1 --2- 172.21.15.178:0/1506004735 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f4af802ff60 0x7f4af8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:00:47.953 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:47.944+0000 7f4b07fff700 1 --2- 172.21.15.178:0/1506004735 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f4af802ff60 0x7f4af8032420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f4af40096a0 tx=0x7f4af4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:00:47.953 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:47.948+0000 7f4b057fa700 1 -- 172.21.15.178:0/1506004735 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4afc01c790 con 0x7f4b08060280 2024-02-11T17:00:48.151 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:48.149+0000 7f4b0eb56700 1 -- 172.21.15.178:0/1506004735 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4b081080b0 con 0x7f4b08060280 2024-02-11T17:00:48.152 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:48.150+0000 7f4b057fa700 1 -- 172.21.15.178:0/1506004735 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4afc01c790 con 0x7f4b08060280 2024-02-11T17:00:48.152 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:00:48.152 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:00:48.153 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:48.152+0000 7f4b0eb56700 1 -- 172.21.15.178:0/1506004735 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f4af802ff60 msgr2=0x7f4af8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:00:48.154 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:48.152+0000 7f4b0eb56700 1 --2- 172.21.15.178:0/1506004735 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f4af802ff60 0x7f4af8032420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f4af40096a0 tx=0x7f4af4006b40).stop 2024-02-11T17:00:48.154 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:48.152+0000 7f4b0eb56700 1 -- 172.21.15.178:0/1506004735 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4b08060280 msgr2=0x7f4b081066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:00:48.154 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:48.152+0000 7f4b0eb56700 1 --2- 172.21.15.178:0/1506004735 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4b08060280 0x7f4b081066e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f4afc013510 tx=0x7f4afc00b040).stop 2024-02-11T17:00:48.154 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:48.152+0000 7f4b0eb56700 1 -- 172.21.15.178:0/1506004735 shutdown_connections 2024-02-11T17:00:48.154 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:48.152+0000 7f4b0eb56700 1 --2- 172.21.15.178:0/1506004735 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f4af802ff60 0x7f4af8032420 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:00:48.154 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:48.152+0000 7f4b0eb56700 1 --2- 172.21.15.178:0/1506004735 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4b08060280 0x7f4b081066e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:00:48.154 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:48.152+0000 7f4b0eb56700 1 -- 172.21.15.178:0/1506004735 >> 172.21.15.178:0/1506004735 conn(0x7f4b080f0b70 msgr2=0x7f4b080f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:00:48.154 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:48.152+0000 7f4b0eb56700 1 -- 172.21.15.178:0/1506004735 shutdown_connections 2024-02-11T17:00:48.154 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:48.152+0000 7f4b0eb56700 1 -- 172.21.15.178:0/1506004735 wait complete. 2024-02-11T17:00:48.155 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:00:48.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:00:48 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:00:47.132584+0000 mgr.smithi037.eofnaq (mgr.14162) 23 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:00:48.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:00:48 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:00:48.151497+0000 mon.smithi037 (mon.0) 121 : audit [DBG] from='client.? 172.21.15.178:0/1506004735' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:00:49.801 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:00:49.802 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:00:50.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:00:50 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:00:49.132909+0000 mgr.smithi037.eofnaq (mgr.14162) 24 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:00:51.602 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:51.599+0000 7ff566be9700 1 -- 172.21.15.178:0/997991980 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff5600f62a0 msgr2=0x7ff5600f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:00:51.602 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:51.599+0000 7ff566be9700 1 --2- 172.21.15.178:0/997991980 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff5600f62a0 0x7ff5600f66c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7ff55c009fb0 tx=0x7ff55c009b30).stop 2024-02-11T17:00:51.602 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:51.599+0000 7ff566be9700 1 -- 172.21.15.178:0/997991980 shutdown_connections 2024-02-11T17:00:51.602 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:51.599+0000 7ff566be9700 1 --2- 172.21.15.178:0/997991980 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff5600f62a0 0x7ff5600f66c0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:00:51.602 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:51.599+0000 7ff566be9700 1 -- 172.21.15.178:0/997991980 >> 172.21.15.178:0/997991980 conn(0x7ff5600f1aa0 msgr2=0x7ff5600f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:00:51.602 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:51.599+0000 7ff566be9700 1 -- 172.21.15.178:0/997991980 shutdown_connections 2024-02-11T17:00:51.602 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:51.599+0000 7ff566be9700 1 -- 172.21.15.178:0/997991980 wait complete. 2024-02-11T17:00:51.602 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:51.599+0000 7ff566be9700 1 Processor -- start 2024-02-11T17:00:51.602 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:51.600+0000 7ff566be9700 1 -- start start 2024-02-11T17:00:51.602 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:51.600+0000 7ff566be9700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff5600f62a0 0x7ff560108700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:00:51.603 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:51.600+0000 7ff566be9700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff560108bf0 con 0x7ff5600f62a0 2024-02-11T17:00:51.603 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:51.600+0000 7ff565be7700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff5600f62a0 0x7ff560108700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:00:51.603 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:51.600+0000 7ff565be7700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff5600f62a0 0x7ff560108700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:33700/0 (socket says 172.21.15.178:33700) 2024-02-11T17:00:51.603 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:51.600+0000 7ff565be7700 1 -- 172.21.15.178:0/2281330354 learned_addr learned my addr 172.21.15.178:0/2281330354 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:00:51.603 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:51.600+0000 7ff565be7700 1 -- 172.21.15.178:0/2281330354 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff55c005040 con 0x7ff5600f62a0 2024-02-11T17:00:51.603 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:51.600+0000 7ff565be7700 1 --2- 172.21.15.178:0/2281330354 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff5600f62a0 0x7ff560108700 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7ff55c009830 tx=0x7ff55c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:00:51.604 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:51.601+0000 7ff5567fc700 1 -- 172.21.15.178:0/2281330354 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff55c009320 con 0x7ff5600f62a0 2024-02-11T17:00:51.604 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:51.601+0000 7ff5567fc700 1 -- 172.21.15.178:0/2281330354 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff55c0136c0 con 0x7ff5600f62a0 2024-02-11T17:00:51.604 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:51.601+0000 7ff566be9700 1 -- 172.21.15.178:0/2281330354 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff560108df0 con 0x7ff5600f62a0 2024-02-11T17:00:51.604 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:51.601+0000 7ff5567fc700 1 -- 172.21.15.178:0/2281330354 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff55c01ee90 con 0x7ff5600f62a0 2024-02-11T17:00:51.604 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:51.601+0000 7ff566be9700 1 -- 172.21.15.178:0/2281330354 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff560109280 con 0x7ff5600f62a0 2024-02-11T17:00:51.604 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:51.601+0000 7ff5567fc700 1 -- 172.21.15.178:0/2281330354 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff55c014440 con 0x7ff5600f62a0 2024-02-11T17:00:51.605 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:51.601+0000 7ff566be9700 1 -- 172.21.15.178:0/2281330354 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff548004fa0 con 0x7ff5600f62a0 2024-02-11T17:00:51.605 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:51.602+0000 7ff5567fc700 1 --2- 172.21.15.178:0/2281330354 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7ff54c02ff10 0x7ff54c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:00:51.605 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:51.602+0000 7ff5567fc700 1 -- 172.21.15.178:0/2281330354 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff55c040f60 con 0x7ff5600f62a0 2024-02-11T17:00:51.605 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:51.602+0000 7ff5653e6700 1 --2- 172.21.15.178:0/2281330354 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7ff54c02ff10 0x7ff54c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:00:51.605 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:51.602+0000 7ff5653e6700 1 --2- 172.21.15.178:0/2281330354 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7ff54c02ff10 0x7ff54c0323d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7ff5500096a0 tx=0x7ff550006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:00:51.608 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:51.606+0000 7ff5567fc700 1 -- 172.21.15.178:0/2281330354 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff55c06d2b0 con 0x7ff5600f62a0 2024-02-11T17:00:51.816 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:51.814+0000 7ff566be9700 1 -- 172.21.15.178:0/2281330354 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff548005670 con 0x7ff5600f62a0 2024-02-11T17:00:51.817 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:51.815+0000 7ff5567fc700 1 -- 172.21.15.178:0/2281330354 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff55c02c2d0 con 0x7ff5600f62a0 2024-02-11T17:00:51.817 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:00:51.817 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:00:51.818 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:51.817+0000 7ff566be9700 1 -- 172.21.15.178:0/2281330354 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7ff54c02ff10 msgr2=0x7ff54c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:00:51.818 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:51.817+0000 7ff566be9700 1 --2- 172.21.15.178:0/2281330354 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7ff54c02ff10 0x7ff54c0323d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7ff5500096a0 tx=0x7ff550006b40).stop 2024-02-11T17:00:51.819 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:51.817+0000 7ff566be9700 1 -- 172.21.15.178:0/2281330354 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff5600f62a0 msgr2=0x7ff560108700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:00:51.819 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:51.817+0000 7ff566be9700 1 --2- 172.21.15.178:0/2281330354 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff5600f62a0 0x7ff560108700 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7ff55c009830 tx=0x7ff55c00b040).stop 2024-02-11T17:00:51.819 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:51.817+0000 7ff566be9700 1 -- 172.21.15.178:0/2281330354 shutdown_connections 2024-02-11T17:00:51.819 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:51.817+0000 7ff566be9700 1 --2- 172.21.15.178:0/2281330354 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7ff54c02ff10 0x7ff54c0323d0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:00:51.819 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:51.817+0000 7ff566be9700 1 --2- 172.21.15.178:0/2281330354 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff5600f62a0 0x7ff560108700 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:00:51.819 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:51.817+0000 7ff566be9700 1 -- 172.21.15.178:0/2281330354 >> 172.21.15.178:0/2281330354 conn(0x7ff5600f1aa0 msgr2=0x7ff5600f2880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:00:51.819 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:51.817+0000 7ff566be9700 1 -- 172.21.15.178:0/2281330354 shutdown_connections 2024-02-11T17:00:51.819 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:51.817+0000 7ff566be9700 1 -- 172.21.15.178:0/2281330354 wait complete. 2024-02-11T17:00:51.819 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:00:52.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:00:52 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:00:51.133293+0000 mgr.smithi037.eofnaq (mgr.14162) 25 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:00:52.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:00:52 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:00:51.816409+0000 mon.smithi037 (mon.0) 122 : audit [DBG] from='client.? 172.21.15.178:0/2281330354' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:00:53.421 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:00:53.421 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:00:54.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:00:54 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:00:53.133616+0000 mgr.smithi037.eofnaq (mgr.14162) 26 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:00:55.182 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:55.180+0000 7f735a9c1700 1 -- 172.21.15.178:0/2874110159 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f73540f5410 msgr2=0x7f73540f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:00:55.182 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:55.180+0000 7f735a9c1700 1 --2- 172.21.15.178:0/2874110159 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f73540f5410 0x7f73540f5830 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f7344009fb0 tx=0x7f7344009b30).stop 2024-02-11T17:00:55.183 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:55.180+0000 7f735a9c1700 1 -- 172.21.15.178:0/2874110159 shutdown_connections 2024-02-11T17:00:55.183 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:55.180+0000 7f735a9c1700 1 --2- 172.21.15.178:0/2874110159 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f73540f5410 0x7f73540f5830 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:00:55.183 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:55.180+0000 7f735a9c1700 1 -- 172.21.15.178:0/2874110159 >> 172.21.15.178:0/2874110159 conn(0x7f73540f0b70 msgr2=0x7f73540f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:00:55.183 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:55.180+0000 7f735a9c1700 1 -- 172.21.15.178:0/2874110159 shutdown_connections 2024-02-11T17:00:55.183 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:55.180+0000 7f735a9c1700 1 -- 172.21.15.178:0/2874110159 wait complete. 2024-02-11T17:00:55.183 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:55.181+0000 7f735a9c1700 1 Processor -- start 2024-02-11T17:00:55.183 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:55.181+0000 7f735a9c1700 1 -- start start 2024-02-11T17:00:55.183 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:55.181+0000 7f735a9c1700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f73540f5410 0x7f7354109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:00:55.183 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:55.181+0000 7f735a9c1700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7354060240 con 0x7f73540f5410 2024-02-11T17:00:55.185 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:55.181+0000 7f7353fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f73540f5410 0x7f7354109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:00:55.185 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:55.181+0000 7f7353fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f73540f5410 0x7f7354109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:33716/0 (socket says 172.21.15.178:33716) 2024-02-11T17:00:55.185 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:55.181+0000 7f7353fff700 1 -- 172.21.15.178:0/4089677805 learned_addr learned my addr 172.21.15.178:0/4089677805 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:00:55.185 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:55.182+0000 7f7353fff700 1 -- 172.21.15.178:0/4089677805 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7344005040 con 0x7f73540f5410 2024-02-11T17:00:55.185 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:55.182+0000 7f7353fff700 1 --2- 172.21.15.178:0/4089677805 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f73540f5410 0x7f7354109430 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f7344013510 tx=0x7f734400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:00:55.185 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:55.182+0000 7f7350ff9700 1 -- 172.21.15.178:0/4089677805 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7344009320 con 0x7f73540f5410 2024-02-11T17:00:55.185 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:55.182+0000 7f7350ff9700 1 -- 172.21.15.178:0/4089677805 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7344013f00 con 0x7f73540f5410 2024-02-11T17:00:55.186 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:55.182+0000 7f735a9c1700 1 -- 172.21.15.178:0/4089677805 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7354060440 con 0x7f73540f5410 2024-02-11T17:00:55.186 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:55.182+0000 7f7350ff9700 1 -- 172.21.15.178:0/4089677805 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7344014d90 con 0x7f73540f5410 2024-02-11T17:00:55.186 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:55.182+0000 7f735a9c1700 1 -- 172.21.15.178:0/4089677805 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7354109ae0 con 0x7f73540f5410 2024-02-11T17:00:55.186 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:55.183+0000 7f7350ff9700 1 -- 172.21.15.178:0/4089677805 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f73440095f0 con 0x7f73540f5410 2024-02-11T17:00:55.186 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:55.183+0000 7f735a9c1700 1 -- 172.21.15.178:0/4089677805 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7354045ad0 con 0x7f73540f5410 2024-02-11T17:00:55.186 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:55.183+0000 7f7350ff9700 1 --2- 172.21.15.178:0/4089677805 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f733c02ff60 0x7f733c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:00:55.186 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:55.183+0000 7f7350ff9700 1 -- 172.21.15.178:0/4089677805 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7344040fe0 con 0x7f73540f5410 2024-02-11T17:00:55.186 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:55.183+0000 7f73537fe700 1 --2- 172.21.15.178:0/4089677805 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f733c02ff60 0x7f733c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:00:55.186 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:55.184+0000 7f73537fe700 1 --2- 172.21.15.178:0/4089677805 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f733c02ff60 0x7f733c032420 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f73480096a0 tx=0x7f7348006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:00:55.189 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:55.188+0000 7f7350ff9700 1 -- 172.21.15.178:0/4089677805 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f734401d7f0 con 0x7f73540f5410 2024-02-11T17:00:55.393 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:55.391+0000 7f735a9c1700 1 -- 172.21.15.178:0/4089677805 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f73540fae60 con 0x7f73540f5410 2024-02-11T17:00:55.393 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:55.392+0000 7f7350ff9700 1 -- 172.21.15.178:0/4089677805 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7344010230 con 0x7f73540f5410 2024-02-11T17:00:55.394 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:00:55.394 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:00:55.395 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:55.393+0000 7f735a9c1700 1 -- 172.21.15.178:0/4089677805 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f733c02ff60 msgr2=0x7f733c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:00:55.395 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:55.393+0000 7f735a9c1700 1 --2- 172.21.15.178:0/4089677805 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f733c02ff60 0x7f733c032420 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f73480096a0 tx=0x7f7348006b40).stop 2024-02-11T17:00:55.395 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:55.393+0000 7f735a9c1700 1 -- 172.21.15.178:0/4089677805 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f73540f5410 msgr2=0x7f7354109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:00:55.395 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:55.393+0000 7f735a9c1700 1 --2- 172.21.15.178:0/4089677805 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f73540f5410 0x7f7354109430 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f7344013510 tx=0x7f734400b040).stop 2024-02-11T17:00:55.395 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:55.393+0000 7f735a9c1700 1 -- 172.21.15.178:0/4089677805 shutdown_connections 2024-02-11T17:00:55.396 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:55.393+0000 7f735a9c1700 1 --2- 172.21.15.178:0/4089677805 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f733c02ff60 0x7f733c032420 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:00:55.396 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:55.393+0000 7f735a9c1700 1 --2- 172.21.15.178:0/4089677805 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f73540f5410 0x7f7354109430 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:00:55.396 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:55.393+0000 7f735a9c1700 1 -- 172.21.15.178:0/4089677805 >> 172.21.15.178:0/4089677805 conn(0x7f73540f0b70 msgr2=0x7f73540f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:00:55.396 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:55.393+0000 7f735a9c1700 1 -- 172.21.15.178:0/4089677805 shutdown_connections 2024-02-11T17:00:55.396 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:55.393+0000 7f735a9c1700 1 -- 172.21.15.178:0/4089677805 wait complete. 2024-02-11T17:00:55.396 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:00:56.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:00:56 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:00:55.133823+0000 mgr.smithi037.eofnaq (mgr.14162) 27 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:00:56.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:00:56 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:00:55.393056+0000 mon.smithi037 (mon.0) 123 : audit [DBG] from='client.? 172.21.15.178:0/4089677805' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:00:57.083 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:00:57.083 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:00:58.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:00:58 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:00:57.134111+0000 mgr.smithi037.eofnaq (mgr.14162) 28 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:00:59.853 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:59.839+0000 7fbddb86e700 1 -- 172.21.15.178:0/2329921076 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbdd40f6510 msgr2=0x7fbdd40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:00:59.853 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:59.839+0000 7fbddb86e700 1 --2- 172.21.15.178:0/2329921076 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbdd40f6510 0x7fbdd40f6930 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fbdc8009fb0 tx=0x7fbdc8009b30).stop 2024-02-11T17:00:59.853 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:59.839+0000 7fbddb86e700 1 -- 172.21.15.178:0/2329921076 shutdown_connections 2024-02-11T17:00:59.854 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:59.839+0000 7fbddb86e700 1 --2- 172.21.15.178:0/2329921076 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbdd40f6510 0x7fbdd40f6930 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:00:59.854 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:59.839+0000 7fbddb86e700 1 -- 172.21.15.178:0/2329921076 >> 172.21.15.178:0/2329921076 conn(0x7fbdd40f1c70 msgr2=0x7fbdd40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:00:59.854 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:59.839+0000 7fbddb86e700 1 -- 172.21.15.178:0/2329921076 shutdown_connections 2024-02-11T17:00:59.854 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:59.839+0000 7fbddb86e700 1 -- 172.21.15.178:0/2329921076 wait complete. 2024-02-11T17:00:59.854 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:59.840+0000 7fbddb86e700 1 Processor -- start 2024-02-11T17:00:59.854 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:59.840+0000 7fbddb86e700 1 -- start start 2024-02-11T17:00:59.854 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:59.840+0000 7fbddb86e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbdd40f6510 0x7fbdd406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:00:59.854 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:59.840+0000 7fbddb86e700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd4069e20 con 0x7fbdd40f6510 2024-02-11T17:00:59.854 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:59.840+0000 7fbdd960a700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbdd40f6510 0x7fbdd406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:00:59.854 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:59.840+0000 7fbdd960a700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbdd40f6510 0x7fbdd406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:44026/0 (socket says 172.21.15.178:44026) 2024-02-11T17:00:59.854 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:59.840+0000 7fbdd960a700 1 -- 172.21.15.178:0/3836150045 learned_addr learned my addr 172.21.15.178:0/3836150045 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:00:59.854 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:59.841+0000 7fbdd960a700 1 -- 172.21.15.178:0/3836150045 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdc8005040 con 0x7fbdd40f6510 2024-02-11T17:00:59.854 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:59.841+0000 7fbdd960a700 1 --2- 172.21.15.178:0/3836150045 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbdd40f6510 0x7fbdd406b9a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fbdc8013510 tx=0x7fbdc800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:00:59.854 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:59.841+0000 7fbdc5ffb700 1 -- 172.21.15.178:0/3836150045 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbdc8009320 con 0x7fbdd40f6510 2024-02-11T17:00:59.854 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:59.841+0000 7fbddb86e700 1 -- 172.21.15.178:0/3836150045 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdd4069fc0 con 0x7fbdd40f6510 2024-02-11T17:00:59.854 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:59.841+0000 7fbdc5ffb700 1 -- 172.21.15.178:0/3836150045 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbdc8013f00 con 0x7fbdd40f6510 2024-02-11T17:00:59.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:59.841+0000 7fbdc5ffb700 1 -- 172.21.15.178:0/3836150045 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbdc8014d90 con 0x7fbdd40f6510 2024-02-11T17:00:59.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:59.841+0000 7fbddb86e700 1 -- 172.21.15.178:0/3836150045 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdd406a450 con 0x7fbdd40f6510 2024-02-11T17:00:59.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:59.842+0000 7fbdc5ffb700 1 -- 172.21.15.178:0/3836150045 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbdc80263a0 con 0x7fbdd40f6510 2024-02-11T17:00:59.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:59.842+0000 7fbdc5ffb700 1 --2- 172.21.15.178:0/3836150045 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fbdc002ff60 0x7fbdc0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:00:59.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:59.842+0000 7fbdc5ffb700 1 -- 172.21.15.178:0/3836150045 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbdc80410f0 con 0x7fbdd40f6510 2024-02-11T17:00:59.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:59.842+0000 7fbddb86e700 1 -- 172.21.15.178:0/3836150045 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdd4059100 con 0x7fbdd40f6510 2024-02-11T17:00:59.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:59.842+0000 7fbdd8e09700 1 --2- 172.21.15.178:0/3836150045 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fbdc002ff60 0x7fbdc0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:00:59.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:59.843+0000 7fbdd8e09700 1 --2- 172.21.15.178:0/3836150045 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fbdc002ff60 0x7fbdc0032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fbdd00096a0 tx=0x7fbdd0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:00:59.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:00:59.847+0000 7fbdc5ffb700 1 -- 172.21.15.178:0/3836150045 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbdc8012920 con 0x7fbdd40f6510 2024-02-11T17:01:00.053 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:00.052+0000 7fbddb86e700 1 -- 172.21.15.178:0/3836150045 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbdd40fbf60 con 0x7fbdd40f6510 2024-02-11T17:01:00.054 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:00.052+0000 7fbdc5ffb700 1 -- 172.21.15.178:0/3836150045 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbdc802b020 con 0x7fbdd40f6510 2024-02-11T17:01:00.054 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:01:00.055 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:01:00.056 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:00.054+0000 7fbddb86e700 1 -- 172.21.15.178:0/3836150045 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fbdc002ff60 msgr2=0x7fbdc0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:01:00.056 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:00.054+0000 7fbddb86e700 1 --2- 172.21.15.178:0/3836150045 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fbdc002ff60 0x7fbdc0032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fbdd00096a0 tx=0x7fbdd0006b40).stop 2024-02-11T17:01:00.056 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:00.054+0000 7fbddb86e700 1 -- 172.21.15.178:0/3836150045 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbdd40f6510 msgr2=0x7fbdd406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:01:00.056 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:00.054+0000 7fbddb86e700 1 --2- 172.21.15.178:0/3836150045 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbdd40f6510 0x7fbdd406b9a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fbdc8013510 tx=0x7fbdc800b040).stop 2024-02-11T17:01:00.056 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:00.054+0000 7fbddb86e700 1 -- 172.21.15.178:0/3836150045 shutdown_connections 2024-02-11T17:01:00.056 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:00.054+0000 7fbddb86e700 1 --2- 172.21.15.178:0/3836150045 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fbdc002ff60 0x7fbdc0032420 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:01:00.056 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:00.054+0000 7fbddb86e700 1 --2- 172.21.15.178:0/3836150045 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbdd40f6510 0x7fbdd406b9a0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:01:00.057 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:00.054+0000 7fbddb86e700 1 -- 172.21.15.178:0/3836150045 >> 172.21.15.178:0/3836150045 conn(0x7fbdd40f1c70 msgr2=0x7fbdd40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:01:00.057 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:00.054+0000 7fbddb86e700 1 -- 172.21.15.178:0/3836150045 shutdown_connections 2024-02-11T17:01:00.057 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:00.054+0000 7fbddb86e700 1 -- 172.21.15.178:0/3836150045 wait complete. 2024-02-11T17:01:00.057 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:01:00.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:01:00 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:00:59.134487+0000 mgr.smithi037.eofnaq (mgr.14162) 29 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:01:00.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:01:00 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:01:00.053841+0000 mon.smithi037 (mon.0) 124 : audit [DBG] from='client.? 172.21.15.178:0/3836150045' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:01:01.726 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:01:01.727 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:01:02.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:01:02 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:01:01.134881+0000 mgr.smithi037.eofnaq (mgr.14162) 30 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:01:04.548 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:03.992+0000 7f9ba2833700 1 -- 172.21.15.178:0/3977143396 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9b9c060280 msgr2=0x7f9b9c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:01:04.548 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:03.992+0000 7f9ba2833700 1 --2- 172.21.15.178:0/3977143396 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9b9c060280 0x7f9b9c0f90f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f9b8c009fb0 tx=0x7f9b8c009b30).stop 2024-02-11T17:01:04.549 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:03.992+0000 7f9ba2833700 1 -- 172.21.15.178:0/3977143396 shutdown_connections 2024-02-11T17:01:04.549 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:03.992+0000 7f9ba2833700 1 --2- 172.21.15.178:0/3977143396 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9b9c060280 0x7f9b9c0f90f0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:01:04.549 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:03.992+0000 7f9ba2833700 1 -- 172.21.15.178:0/3977143396 >> 172.21.15.178:0/3977143396 conn(0x7f9b9c0f0b70 msgr2=0x7f9b9c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:01:04.549 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:03.992+0000 7f9ba2833700 1 -- 172.21.15.178:0/3977143396 shutdown_connections 2024-02-11T17:01:04.549 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:03.993+0000 7f9ba2833700 1 -- 172.21.15.178:0/3977143396 wait complete. 2024-02-11T17:01:04.550 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:03.993+0000 7f9ba2833700 1 Processor -- start 2024-02-11T17:01:04.550 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:03.993+0000 7f9ba2833700 1 -- start start 2024-02-11T17:01:04.550 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:03.994+0000 7f9ba2833700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9b9c060280 0x7f9b9c1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:01:04.550 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:03.994+0000 7f9ba2833700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b9c106bd0 con 0x7f9b9c060280 2024-02-11T17:01:04.550 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:03.994+0000 7f9b9bfff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9b9c060280 0x7f9b9c1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:01:04.550 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:03.994+0000 7f9b9bfff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9b9c060280 0x7f9b9c1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:44052/0 (socket says 172.21.15.178:44052) 2024-02-11T17:01:04.551 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:03.994+0000 7f9b9bfff700 1 -- 172.21.15.178:0/1801245043 learned_addr learned my addr 172.21.15.178:0/1801245043 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:01:04.551 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:03.994+0000 7f9b9bfff700 1 -- 172.21.15.178:0/1801245043 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b8c005040 con 0x7f9b9c060280 2024-02-11T17:01:04.551 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:03.994+0000 7f9b9bfff700 1 --2- 172.21.15.178:0/1801245043 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9b9c060280 0x7f9b9c1066e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f9b8c013510 tx=0x7f9b8c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:01:04.551 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:03.994+0000 7f9b98ff9700 1 -- 172.21.15.178:0/1801245043 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b8c009320 con 0x7f9b9c060280 2024-02-11T17:01:04.551 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:03.995+0000 7f9b98ff9700 1 -- 172.21.15.178:0/1801245043 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9b8c013f00 con 0x7f9b9c060280 2024-02-11T17:01:04.551 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:03.995+0000 7f9ba2833700 1 -- 172.21.15.178:0/1801245043 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b9c106dd0 con 0x7f9b9c060280 2024-02-11T17:01:04.552 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:03.995+0000 7f9b98ff9700 1 -- 172.21.15.178:0/1801245043 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b8c014dc0 con 0x7f9b9c060280 2024-02-11T17:01:04.552 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:03.995+0000 7f9ba2833700 1 -- 172.21.15.178:0/1801245043 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b9c107260 con 0x7f9b9c060280 2024-02-11T17:01:04.552 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:03.995+0000 7f9b98ff9700 1 -- 172.21.15.178:0/1801245043 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9b8c01d3d0 con 0x7f9b9c060280 2024-02-11T17:01:04.552 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:03.996+0000 7f9b98ff9700 1 --2- 172.21.15.178:0/1801245043 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f9b8802ff60 0x7f9b88032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:01:04.552 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:03.996+0000 7f9b98ff9700 1 -- 172.21.15.178:0/1801245043 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9b8c041050 con 0x7f9b9c060280 2024-02-11T17:01:04.552 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:03.996+0000 7f9ba2833700 1 -- 172.21.15.178:0/1801245043 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b9c045ad0 con 0x7f9b9c060280 2024-02-11T17:01:04.552 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:03.996+0000 7f9b9b7fe700 1 --2- 172.21.15.178:0/1801245043 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f9b8802ff60 0x7f9b88032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:01:04.552 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:03.996+0000 7f9b9b7fe700 1 --2- 172.21.15.178:0/1801245043 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f9b8802ff60 0x7f9b88032420 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f9b840096a0 tx=0x7f9b84006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:01:04.552 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:04.001+0000 7f9b98ff9700 1 -- 172.21.15.178:0/1801245043 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9b8c012920 con 0x7f9b9c060280 2024-02-11T17:01:04.552 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:04.204+0000 7f9ba2833700 1 -- 172.21.15.178:0/1801245043 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9b9c0fbf50 con 0x7f9b9c060280 2024-02-11T17:01:04.552 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:04.205+0000 7f9b98ff9700 1 -- 172.21.15.178:0/1801245043 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9b8c012330 con 0x7f9b9c060280 2024-02-11T17:01:04.553 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:01:04.553 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:01:04.553 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:04.207+0000 7f9ba2833700 1 -- 172.21.15.178:0/1801245043 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f9b8802ff60 msgr2=0x7f9b88032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:01:04.553 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:04.207+0000 7f9ba2833700 1 --2- 172.21.15.178:0/1801245043 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f9b8802ff60 0x7f9b88032420 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f9b840096a0 tx=0x7f9b84006b40).stop 2024-02-11T17:01:04.553 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:04.207+0000 7f9ba2833700 1 -- 172.21.15.178:0/1801245043 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9b9c060280 msgr2=0x7f9b9c1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:01:04.553 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:04.207+0000 7f9ba2833700 1 --2- 172.21.15.178:0/1801245043 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9b9c060280 0x7f9b9c1066e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f9b8c013510 tx=0x7f9b8c00b040).stop 2024-02-11T17:01:04.553 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:04.207+0000 7f9ba2833700 1 -- 172.21.15.178:0/1801245043 shutdown_connections 2024-02-11T17:01:04.553 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:04.207+0000 7f9ba2833700 1 --2- 172.21.15.178:0/1801245043 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f9b8802ff60 0x7f9b88032420 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:01:04.554 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:04.207+0000 7f9ba2833700 1 --2- 172.21.15.178:0/1801245043 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9b9c060280 0x7f9b9c1066e0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:01:04.554 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:04.207+0000 7f9ba2833700 1 -- 172.21.15.178:0/1801245043 >> 172.21.15.178:0/1801245043 conn(0x7f9b9c0f0b70 msgr2=0x7f9b9c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:01:04.554 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:04.207+0000 7f9ba2833700 1 -- 172.21.15.178:0/1801245043 shutdown_connections 2024-02-11T17:01:04.554 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:04.207+0000 7f9ba2833700 1 -- 172.21.15.178:0/1801245043 wait complete. 2024-02-11T17:01:04.554 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:01:04.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:01:04 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:01:03.135268+0000 mgr.smithi037.eofnaq (mgr.14162) 31 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:01:04.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:01:04 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:01:03.270525+0000 mon.smithi037 (mon.0) 125 : audit [INF] from='mgr.14162 172.21.15.37:0/2745465012' entity='mgr.smithi037.eofnaq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi037.eofnaq/mirror_snapshot_schedule"}]: dispatch 2024-02-11T17:01:04.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:01:04 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:01:03.274580+0000 mon.smithi037 (mon.0) 126 : audit [INF] from='mgr.14162 172.21.15.37:0/2745465012' entity='mgr.smithi037.eofnaq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi037.eofnaq/trash_purge_schedule"}]: dispatch 2024-02-11T17:01:05.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:01:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:01:04.206619+0000 mon.smithi037 (mon.0) 127 : audit [DBG] from='client.? 172.21.15.178:0/1801245043' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:01:06.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:01:06 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:01:05.135607+0000 mgr.smithi037.eofnaq (mgr.14162) 32 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:01:08.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:01:08 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:01:07.135952+0000 mgr.smithi037.eofnaq (mgr.14162) 33 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:01:09.934 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:01:09.935 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:01:10.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:01:10 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:01:09.136323+0000 mgr.smithi037.eofnaq (mgr.14162) 34 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:01:12.645 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:12.642+0000 7f514d785700 1 -- 172.21.15.178:0/1588598893 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f51480f47c0 msgr2=0x7f51480f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:01:12.646 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:12.642+0000 7f514d785700 1 --2- 172.21.15.178:0/1588598893 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f51480f47c0 0x7f51480f4be0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f513c009fb0 tx=0x7f513c009b30).stop 2024-02-11T17:01:12.646 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:12.642+0000 7f514d785700 1 -- 172.21.15.178:0/1588598893 shutdown_connections 2024-02-11T17:01:12.646 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:12.642+0000 7f514d785700 1 --2- 172.21.15.178:0/1588598893 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f51480f47c0 0x7f51480f4be0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:01:12.646 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:12.642+0000 7f514d785700 1 -- 172.21.15.178:0/1588598893 >> 172.21.15.178:0/1588598893 conn(0x7f51480f02e0 msgr2=0x7f51480f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:01:12.646 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:12.642+0000 7f514d785700 1 -- 172.21.15.178:0/1588598893 shutdown_connections 2024-02-11T17:01:12.646 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:12.642+0000 7f514d785700 1 -- 172.21.15.178:0/1588598893 wait complete. 2024-02-11T17:01:12.646 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:12.643+0000 7f514d785700 1 Processor -- start 2024-02-11T17:01:12.646 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:12.643+0000 7f514d785700 1 -- start start 2024-02-11T17:01:12.646 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:12.643+0000 7f514d785700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f51480f47c0 0x7f5148106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:01:12.646 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:12.643+0000 7f514d785700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5148106bf0 con 0x7f51480f47c0 2024-02-11T17:01:12.646 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:12.643+0000 7f5146ffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f51480f47c0 0x7f5148106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:01:12.647 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:12.643+0000 7f5146ffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f51480f47c0 0x7f5148106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:52286/0 (socket says 172.21.15.178:52286) 2024-02-11T17:01:12.647 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:12.643+0000 7f5146ffd700 1 -- 172.21.15.178:0/4021651621 learned_addr learned my addr 172.21.15.178:0/4021651621 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:01:12.647 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:12.644+0000 7f5146ffd700 1 -- 172.21.15.178:0/4021651621 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f513c005040 con 0x7f51480f47c0 2024-02-11T17:01:12.647 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:12.644+0000 7f5146ffd700 1 --2- 172.21.15.178:0/4021651621 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f51480f47c0 0x7f5148106700 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f513c014000 tx=0x7f513c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:01:12.647 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:12.644+0000 7f512f7fe700 1 -- 172.21.15.178:0/4021651621 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f513c0095f0 con 0x7f51480f47c0 2024-02-11T17:01:12.647 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:12.644+0000 7f514d785700 1 -- 172.21.15.178:0/4021651621 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5148106df0 con 0x7f51480f47c0 2024-02-11T17:01:12.647 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:12.644+0000 7f512f7fe700 1 -- 172.21.15.178:0/4021651621 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f513c007e70 con 0x7f51480f47c0 2024-02-11T17:01:12.647 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:12.644+0000 7f512f7fe700 1 -- 172.21.15.178:0/4021651621 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f513c014ea0 con 0x7f51480f47c0 2024-02-11T17:01:12.647 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:12.644+0000 7f514d785700 1 -- 172.21.15.178:0/4021651621 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5148107280 con 0x7f51480f47c0 2024-02-11T17:01:12.648 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:12.645+0000 7f512f7fe700 1 -- 172.21.15.178:0/4021651621 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f513c014690 con 0x7f51480f47c0 2024-02-11T17:01:12.648 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:12.645+0000 7f514d785700 1 -- 172.21.15.178:0/4021651621 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5148045ad0 con 0x7f51480f47c0 2024-02-11T17:01:12.648 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:12.645+0000 7f512f7fe700 1 --2- 172.21.15.178:0/4021651621 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f513002ff60 0x7f5130032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:01:12.648 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:12.645+0000 7f512f7fe700 1 -- 172.21.15.178:0/4021651621 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f513c040870 con 0x7f51480f47c0 2024-02-11T17:01:12.648 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:12.645+0000 7f51467fc700 1 --2- 172.21.15.178:0/4021651621 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f513002ff60 0x7f5130032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:01:12.648 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:12.646+0000 7f51467fc700 1 --2- 172.21.15.178:0/4021651621 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f513002ff60 0x7f5130032420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f51380096a0 tx=0x7f5138006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:01:12.651 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:12.650+0000 7f512f7fe700 1 -- 172.21.15.178:0/4021651621 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f513c00f300 con 0x7f51480f47c0 2024-02-11T17:01:12.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:01:12 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:01:11.136657+0000 mgr.smithi037.eofnaq (mgr.14162) 35 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:01:12.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:12.856+0000 7f514d785700 1 -- 172.21.15.178:0/4021651621 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5148108130 con 0x7f51480f47c0 2024-02-11T17:01:12.859 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:12.857+0000 7f512f7fe700 1 -- 172.21.15.178:0/4021651621 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f513c00f300 con 0x7f51480f47c0 2024-02-11T17:01:12.859 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:01:12.859 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:01:12.860 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:12.858+0000 7f514d785700 1 -- 172.21.15.178:0/4021651621 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f513002ff60 msgr2=0x7f5130032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:01:12.860 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:12.858+0000 7f514d785700 1 --2- 172.21.15.178:0/4021651621 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f513002ff60 0x7f5130032420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f51380096a0 tx=0x7f5138006b40).stop 2024-02-11T17:01:12.860 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:12.859+0000 7f514d785700 1 -- 172.21.15.178:0/4021651621 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f51480f47c0 msgr2=0x7f5148106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:01:12.861 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:12.859+0000 7f514d785700 1 --2- 172.21.15.178:0/4021651621 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f51480f47c0 0x7f5148106700 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f513c014000 tx=0x7f513c00b040).stop 2024-02-11T17:01:12.861 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:12.859+0000 7f514d785700 1 -- 172.21.15.178:0/4021651621 shutdown_connections 2024-02-11T17:01:12.861 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:12.859+0000 7f514d785700 1 --2- 172.21.15.178:0/4021651621 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f513002ff60 0x7f5130032420 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:01:12.861 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:12.859+0000 7f514d785700 1 --2- 172.21.15.178:0/4021651621 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f51480f47c0 0x7f5148106700 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:01:12.861 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:12.859+0000 7f514d785700 1 -- 172.21.15.178:0/4021651621 >> 172.21.15.178:0/4021651621 conn(0x7f51480f02e0 msgr2=0x7f51480f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:01:12.861 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:12.859+0000 7f514d785700 1 -- 172.21.15.178:0/4021651621 shutdown_connections 2024-02-11T17:01:12.861 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:12.859+0000 7f514d785700 1 -- 172.21.15.178:0/4021651621 wait complete. 2024-02-11T17:01:12.861 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:01:13.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:01:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:01:12.858385+0000 mon.smithi037 (mon.0) 128 : audit [DBG] from='client.? 172.21.15.178:0/4021651621' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:01:14.608 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:01:14.609 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:01:14.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:01:14 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:01:13.136971+0000 mgr.smithi037.eofnaq (mgr.14162) 36 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:01:16.315 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:16.303+0000 7f2b6b59e700 1 -- 172.21.15.178:0/2102619897 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2b6c0f51a0 msgr2=0x7f2b6c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:01:16.315 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:16.303+0000 7f2b6b59e700 1 --2- 172.21.15.178:0/2102619897 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2b6c0f51a0 0x7f2b6c0f55c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f2b5c009fb0 tx=0x7f2b5c009b30).stop 2024-02-11T17:01:16.315 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:16.303+0000 7f2b6b59e700 1 -- 172.21.15.178:0/2102619897 shutdown_connections 2024-02-11T17:01:16.315 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:16.303+0000 7f2b6b59e700 1 --2- 172.21.15.178:0/2102619897 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2b6c0f51a0 0x7f2b6c0f55c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:01:16.315 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:16.303+0000 7f2b6b59e700 1 -- 172.21.15.178:0/2102619897 >> 172.21.15.178:0/2102619897 conn(0x7f2b6c0f09a0 msgr2=0x7f2b6c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:01:16.315 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:16.303+0000 7f2b6b59e700 1 -- 172.21.15.178:0/2102619897 shutdown_connections 2024-02-11T17:01:16.315 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:16.303+0000 7f2b6b59e700 1 -- 172.21.15.178:0/2102619897 wait complete. 2024-02-11T17:01:16.315 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:16.303+0000 7f2b6b59e700 1 Processor -- start 2024-02-11T17:01:16.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:16.304+0000 7f2b6b59e700 1 -- start start 2024-02-11T17:01:16.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:16.304+0000 7f2b6b59e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2b6c0f51a0 0x7f2b6c1091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:01:16.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:16.304+0000 7f2b6b59e700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b6c060240 con 0x7f2b6c0f51a0 2024-02-11T17:01:16.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:16.304+0000 7f2b6a59c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2b6c0f51a0 0x7f2b6c1091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:01:16.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:16.304+0000 7f2b6a59c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2b6c0f51a0 0x7f2b6c1091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:52316/0 (socket says 172.21.15.178:52316) 2024-02-11T17:01:16.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:16.304+0000 7f2b6a59c700 1 -- 172.21.15.178:0/3556117762 learned_addr learned my addr 172.21.15.178:0/3556117762 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:01:16.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:16.304+0000 7f2b6a59c700 1 -- 172.21.15.178:0/3556117762 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b5c005040 con 0x7f2b6c0f51a0 2024-02-11T17:01:16.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:16.304+0000 7f2b6a59c700 1 --2- 172.21.15.178:0/3556117762 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2b6c0f51a0 0x7f2b6c1091b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f2b5c009830 tx=0x7f2b5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:01:16.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:16.304+0000 7f2b5affd700 1 -- 172.21.15.178:0/3556117762 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b5c009320 con 0x7f2b6c0f51a0 2024-02-11T17:01:16.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:16.305+0000 7f2b5affd700 1 -- 172.21.15.178:0/3556117762 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2b5c0136c0 con 0x7f2b6c0f51a0 2024-02-11T17:01:16.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:16.305+0000 7f2b5affd700 1 -- 172.21.15.178:0/3556117762 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b5c01ee90 con 0x7f2b6c0f51a0 2024-02-11T17:01:16.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:16.305+0000 7f2b6b59e700 1 -- 172.21.15.178:0/3556117762 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b6c060440 con 0x7f2b6c0f51a0 2024-02-11T17:01:16.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:16.305+0000 7f2b6b59e700 1 -- 172.21.15.178:0/3556117762 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b6c109860 con 0x7f2b6c0f51a0 2024-02-11T17:01:16.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:16.305+0000 7f2b5affd700 1 -- 172.21.15.178:0/3556117762 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2b5c01e420 con 0x7f2b6c0f51a0 2024-02-11T17:01:16.317 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:16.305+0000 7f2b6b59e700 1 -- 172.21.15.178:0/3556117762 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b6c045ad0 con 0x7f2b6c0f51a0 2024-02-11T17:01:16.317 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:16.306+0000 7f2b5affd700 1 --2- 172.21.15.178:0/3556117762 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f2b5402ff10 0x7f2b540323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:01:16.317 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:16.306+0000 7f2b5affd700 1 -- 172.21.15.178:0/3556117762 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2b5c041080 con 0x7f2b6c0f51a0 2024-02-11T17:01:16.317 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:16.306+0000 7f2b69d9b700 1 --2- 172.21.15.178:0/3556117762 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f2b5402ff10 0x7f2b540323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:01:16.317 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:16.306+0000 7f2b69d9b700 1 --2- 172.21.15.178:0/3556117762 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f2b5402ff10 0x7f2b540323d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f2b600096a0 tx=0x7f2b60006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:01:16.317 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:16.311+0000 7f2b5affd700 1 -- 172.21.15.178:0/3556117762 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2b5c06d2b0 con 0x7f2b6c0f51a0 2024-02-11T17:01:16.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:01:16 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:01:15.137309+0000 mgr.smithi037.eofnaq (mgr.14162) 37 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:01:16.517 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:16.515+0000 7f2b6b59e700 1 -- 172.21.15.178:0/3556117762 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2b6c10a080 con 0x7f2b6c0f51a0 2024-02-11T17:01:16.517 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:16.516+0000 7f2b5affd700 1 -- 172.21.15.178:0/3556117762 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2b5c02c2d0 con 0x7f2b6c0f51a0 2024-02-11T17:01:16.518 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:01:16.518 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:01:16.520 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:16.517+0000 7f2b6b59e700 1 -- 172.21.15.178:0/3556117762 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f2b5402ff10 msgr2=0x7f2b540323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:01:16.520 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:16.517+0000 7f2b6b59e700 1 --2- 172.21.15.178:0/3556117762 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f2b5402ff10 0x7f2b540323d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f2b600096a0 tx=0x7f2b60006b40).stop 2024-02-11T17:01:16.520 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:16.517+0000 7f2b6b59e700 1 -- 172.21.15.178:0/3556117762 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2b6c0f51a0 msgr2=0x7f2b6c1091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:01:16.520 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:16.517+0000 7f2b6b59e700 1 --2- 172.21.15.178:0/3556117762 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2b6c0f51a0 0x7f2b6c1091b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f2b5c009830 tx=0x7f2b5c00b040).stop 2024-02-11T17:01:16.520 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:16.517+0000 7f2b6b59e700 1 -- 172.21.15.178:0/3556117762 shutdown_connections 2024-02-11T17:01:16.520 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:16.517+0000 7f2b6b59e700 1 --2- 172.21.15.178:0/3556117762 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f2b5402ff10 0x7f2b540323d0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:01:16.520 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:16.518+0000 7f2b6b59e700 1 --2- 172.21.15.178:0/3556117762 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2b6c0f51a0 0x7f2b6c1091b0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:01:16.520 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:16.518+0000 7f2b6b59e700 1 -- 172.21.15.178:0/3556117762 >> 172.21.15.178:0/3556117762 conn(0x7f2b6c0f09a0 msgr2=0x7f2b6c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:01:16.520 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:16.518+0000 7f2b6b59e700 1 -- 172.21.15.178:0/3556117762 shutdown_connections 2024-02-11T17:01:16.520 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:16.518+0000 7f2b6b59e700 1 -- 172.21.15.178:0/3556117762 wait complete. 2024-02-11T17:01:16.521 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:01:17.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:01:17 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:01:16.517436+0000 mon.smithi037 (mon.0) 129 : audit [DBG] from='client.? 172.21.15.178:0/3556117762' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:01:18.208 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:01:18.209 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:01:18.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:01:18 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:01:17.137683+0000 mgr.smithi037.eofnaq (mgr.14162) 38 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:01:19.962 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:19.959+0000 7fc66f0ed700 1 -- 172.21.15.178:0/631813879 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc6680f6510 msgr2=0x7fc6680f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:01:19.962 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:19.959+0000 7fc66f0ed700 1 --2- 172.21.15.178:0/631813879 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc6680f6510 0x7fc6680f6930 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fc65c009fb0 tx=0x7fc65c009b30).stop 2024-02-11T17:01:19.962 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:19.959+0000 7fc66f0ed700 1 -- 172.21.15.178:0/631813879 shutdown_connections 2024-02-11T17:01:19.962 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:19.959+0000 7fc66f0ed700 1 --2- 172.21.15.178:0/631813879 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc6680f6510 0x7fc6680f6930 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:01:19.962 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:19.959+0000 7fc66f0ed700 1 -- 172.21.15.178:0/631813879 >> 172.21.15.178:0/631813879 conn(0x7fc6680f1c70 msgr2=0x7fc6680f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:01:19.962 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:19.959+0000 7fc66f0ed700 1 -- 172.21.15.178:0/631813879 shutdown_connections 2024-02-11T17:01:19.962 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:19.959+0000 7fc66f0ed700 1 -- 172.21.15.178:0/631813879 wait complete. 2024-02-11T17:01:19.963 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:19.960+0000 7fc66f0ed700 1 Processor -- start 2024-02-11T17:01:19.963 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:19.960+0000 7fc66f0ed700 1 -- start start 2024-02-11T17:01:19.963 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:19.960+0000 7fc66f0ed700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc6680f6510 0x7fc6681066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:01:19.963 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:19.960+0000 7fc66f0ed700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc668106bc0 con 0x7fc6680f6510 2024-02-11T17:01:19.963 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:19.961+0000 7fc66ce89700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc6680f6510 0x7fc6681066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:01:19.963 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:19.961+0000 7fc66ce89700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc6680f6510 0x7fc6681066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:47740/0 (socket says 172.21.15.178:47740) 2024-02-11T17:01:19.963 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:19.961+0000 7fc66ce89700 1 -- 172.21.15.178:0/1736542987 learned_addr learned my addr 172.21.15.178:0/1736542987 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:01:19.963 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:19.961+0000 7fc66ce89700 1 -- 172.21.15.178:0/1736542987 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc65c005040 con 0x7fc6680f6510 2024-02-11T17:01:19.963 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:19.961+0000 7fc66ce89700 1 --2- 172.21.15.178:0/1736542987 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc6680f6510 0x7fc6681066d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fc65c013510 tx=0x7fc65c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:01:19.963 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:19.961+0000 7fc6657fa700 1 -- 172.21.15.178:0/1736542987 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc65c009320 con 0x7fc6680f6510 2024-02-11T17:01:19.963 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:19.961+0000 7fc66f0ed700 1 -- 172.21.15.178:0/1736542987 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc668106dc0 con 0x7fc6680f6510 2024-02-11T17:01:19.963 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:19.961+0000 7fc6657fa700 1 -- 172.21.15.178:0/1736542987 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc65c013f00 con 0x7fc6680f6510 2024-02-11T17:01:19.963 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:19.961+0000 7fc6657fa700 1 -- 172.21.15.178:0/1736542987 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc65c014d90 con 0x7fc6680f6510 2024-02-11T17:01:19.963 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:19.961+0000 7fc66f0ed700 1 -- 172.21.15.178:0/1736542987 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc668107250 con 0x7fc6680f6510 2024-02-11T17:01:19.964 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:19.962+0000 7fc6657fa700 1 -- 172.21.15.178:0/1736542987 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc65c0263a0 con 0x7fc6680f6510 2024-02-11T17:01:19.964 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:19.962+0000 7fc66f0ed700 1 -- 172.21.15.178:0/1736542987 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc668045ad0 con 0x7fc6680f6510 2024-02-11T17:01:19.964 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:19.962+0000 7fc6657fa700 1 --2- 172.21.15.178:0/1736542987 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fc65802ff60 0x7fc658032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:01:19.965 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:19.962+0000 7fc6657fa700 1 -- 172.21.15.178:0/1736542987 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc65c040ff0 con 0x7fc6680f6510 2024-02-11T17:01:19.965 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:19.962+0000 7fc667fff700 1 --2- 172.21.15.178:0/1736542987 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fc65802ff60 0x7fc658032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:01:19.965 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:19.963+0000 7fc667fff700 1 --2- 172.21.15.178:0/1736542987 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fc65802ff60 0x7fc658032420 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7fc6540096a0 tx=0x7fc654006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:01:19.967 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:19.967+0000 7fc6657fa700 1 -- 172.21.15.178:0/1736542987 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc65c0095f0 con 0x7fc6680f6510 2024-02-11T17:01:20.171 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:20.171+0000 7fc66f0ed700 1 -- 172.21.15.178:0/1736542987 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc668108100 con 0x7fc6680f6510 2024-02-11T17:01:20.172 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:20.171+0000 7fc6657fa700 1 -- 172.21.15.178:0/1736542987 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc65c01db70 con 0x7fc6680f6510 2024-02-11T17:01:20.172 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:01:20.172 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:01:20.174 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:20.173+0000 7fc66f0ed700 1 -- 172.21.15.178:0/1736542987 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fc65802ff60 msgr2=0x7fc658032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:01:20.174 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:20.173+0000 7fc66f0ed700 1 --2- 172.21.15.178:0/1736542987 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fc65802ff60 0x7fc658032420 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7fc6540096a0 tx=0x7fc654006b40).stop 2024-02-11T17:01:20.174 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:20.173+0000 7fc66f0ed700 1 -- 172.21.15.178:0/1736542987 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc6680f6510 msgr2=0x7fc6681066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:01:20.174 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:20.173+0000 7fc66f0ed700 1 --2- 172.21.15.178:0/1736542987 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc6680f6510 0x7fc6681066d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fc65c013510 tx=0x7fc65c00b040).stop 2024-02-11T17:01:20.175 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:20.173+0000 7fc66f0ed700 1 -- 172.21.15.178:0/1736542987 shutdown_connections 2024-02-11T17:01:20.175 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:20.173+0000 7fc66f0ed700 1 --2- 172.21.15.178:0/1736542987 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fc65802ff60 0x7fc658032420 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:01:20.175 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:20.173+0000 7fc66f0ed700 1 --2- 172.21.15.178:0/1736542987 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc6680f6510 0x7fc6681066d0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:01:20.175 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:20.173+0000 7fc66f0ed700 1 -- 172.21.15.178:0/1736542987 >> 172.21.15.178:0/1736542987 conn(0x7fc6680f1c70 msgr2=0x7fc6680f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:01:20.175 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:20.173+0000 7fc66f0ed700 1 -- 172.21.15.178:0/1736542987 shutdown_connections 2024-02-11T17:01:20.175 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:20.173+0000 7fc66f0ed700 1 -- 172.21.15.178:0/1736542987 wait complete. 2024-02-11T17:01:20.175 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:01:20.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:01:20 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:01:19.137953+0000 mgr.smithi037.eofnaq (mgr.14162) 39 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:01:20.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:01:20 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:01:20.171696+0000 mon.smithi037 (mon.0) 130 : audit [DBG] from='client.? 172.21.15.178:0/1736542987' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:01:21.785 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:01:21.785 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:01:22.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:01:22 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:01:21.138287+0000 mgr.smithi037.eofnaq (mgr.14162) 40 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:01:23.692 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:23.689+0000 7f3da29d5700 1 -- 172.21.15.178:0/374254185 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3d9c0f5410 msgr2=0x7f3d9c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:01:23.692 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:23.689+0000 7f3da29d5700 1 --2- 172.21.15.178:0/374254185 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3d9c0f5410 0x7f3d9c0f5830 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f3d8c009fb0 tx=0x7f3d8c009b30).stop 2024-02-11T17:01:23.692 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:23.689+0000 7f3da29d5700 1 -- 172.21.15.178:0/374254185 shutdown_connections 2024-02-11T17:01:23.692 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:23.689+0000 7f3da29d5700 1 --2- 172.21.15.178:0/374254185 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3d9c0f5410 0x7f3d9c0f5830 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:01:23.692 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:23.689+0000 7f3da29d5700 1 -- 172.21.15.178:0/374254185 >> 172.21.15.178:0/374254185 conn(0x7f3d9c0f0b70 msgr2=0x7f3d9c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:01:23.692 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:23.689+0000 7f3da29d5700 1 -- 172.21.15.178:0/374254185 shutdown_connections 2024-02-11T17:01:23.692 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:23.689+0000 7f3da29d5700 1 -- 172.21.15.178:0/374254185 wait complete. 2024-02-11T17:01:23.692 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:23.690+0000 7f3da29d5700 1 Processor -- start 2024-02-11T17:01:23.693 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:23.690+0000 7f3da29d5700 1 -- start start 2024-02-11T17:01:23.693 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:23.690+0000 7f3da29d5700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3d9c0f5410 0x7f3d9c105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:01:23.693 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:23.690+0000 7f3da29d5700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d9c105b20 con 0x7f3d9c0f5410 2024-02-11T17:01:23.693 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:23.691+0000 7f3d9bfff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3d9c0f5410 0x7f3d9c105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:01:23.693 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:23.691+0000 7f3d9bfff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3d9c0f5410 0x7f3d9c105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:47748/0 (socket says 172.21.15.178:47748) 2024-02-11T17:01:23.693 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:23.691+0000 7f3d9bfff700 1 -- 172.21.15.178:0/3234395226 learned_addr learned my addr 172.21.15.178:0/3234395226 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:01:23.693 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:23.691+0000 7f3d9bfff700 1 -- 172.21.15.178:0/3234395226 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d8c005040 con 0x7f3d9c0f5410 2024-02-11T17:01:23.693 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:23.691+0000 7f3d9bfff700 1 --2- 172.21.15.178:0/3234395226 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3d9c0f5410 0x7f3d9c105630 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f3d8c013510 tx=0x7f3d8c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:01:23.693 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:23.691+0000 7f3d98ff9700 1 -- 172.21.15.178:0/3234395226 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d8c009320 con 0x7f3d9c0f5410 2024-02-11T17:01:23.693 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:23.691+0000 7f3da29d5700 1 -- 172.21.15.178:0/3234395226 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d9c105d20 con 0x7f3d9c0f5410 2024-02-11T17:01:23.693 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:23.691+0000 7f3d98ff9700 1 -- 172.21.15.178:0/3234395226 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3d8c013f00 con 0x7f3d9c0f5410 2024-02-11T17:01:23.693 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:23.691+0000 7f3d98ff9700 1 -- 172.21.15.178:0/3234395226 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d8c014d90 con 0x7f3d9c0f5410 2024-02-11T17:01:23.693 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:23.691+0000 7f3da29d5700 1 -- 172.21.15.178:0/3234395226 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d9c1061b0 con 0x7f3d9c0f5410 2024-02-11T17:01:23.694 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:23.692+0000 7f3d98ff9700 1 -- 172.21.15.178:0/3234395226 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3d8c0263a0 con 0x7f3d9c0f5410 2024-02-11T17:01:23.694 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:23.692+0000 7f3d98ff9700 1 --2- 172.21.15.178:0/3234395226 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f3d8802ff60 0x7f3d88032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:01:23.694 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:23.692+0000 7f3da29d5700 1 -- 172.21.15.178:0/3234395226 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d9c045ad0 con 0x7f3d9c0f5410 2024-02-11T17:01:23.694 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:23.692+0000 7f3d98ff9700 1 -- 172.21.15.178:0/3234395226 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3d8c0410b0 con 0x7f3d9c0f5410 2024-02-11T17:01:23.695 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:23.692+0000 7f3d9b7fe700 1 --2- 172.21.15.178:0/3234395226 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f3d8802ff60 0x7f3d88032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:01:23.695 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:23.693+0000 7f3d9b7fe700 1 --2- 172.21.15.178:0/3234395226 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f3d8802ff60 0x7f3d88032420 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f3d840096a0 tx=0x7f3d84006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:01:23.697 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:23.697+0000 7f3d98ff9700 1 -- 172.21.15.178:0/3234395226 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3d8c06d2b0 con 0x7f3d9c0f5410 2024-02-11T17:01:23.901 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:23.900+0000 7f3da29d5700 1 -- 172.21.15.178:0/3234395226 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3d9c0faf50 con 0x7f3d9c0f5410 2024-02-11T17:01:23.901 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:23.901+0000 7f3d98ff9700 1 -- 172.21.15.178:0/3234395226 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3d8c012510 con 0x7f3d9c0f5410 2024-02-11T17:01:23.901 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:01:23.902 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:01:23.903 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:23.902+0000 7f3da29d5700 1 -- 172.21.15.178:0/3234395226 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f3d8802ff60 msgr2=0x7f3d88032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:01:23.903 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:23.902+0000 7f3da29d5700 1 --2- 172.21.15.178:0/3234395226 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f3d8802ff60 0x7f3d88032420 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f3d840096a0 tx=0x7f3d84006b40).stop 2024-02-11T17:01:23.903 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:23.902+0000 7f3da29d5700 1 -- 172.21.15.178:0/3234395226 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3d9c0f5410 msgr2=0x7f3d9c105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:01:23.903 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:23.902+0000 7f3da29d5700 1 --2- 172.21.15.178:0/3234395226 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3d9c0f5410 0x7f3d9c105630 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f3d8c013510 tx=0x7f3d8c00b040).stop 2024-02-11T17:01:23.903 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:23.902+0000 7f3da29d5700 1 -- 172.21.15.178:0/3234395226 shutdown_connections 2024-02-11T17:01:23.903 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:23.902+0000 7f3da29d5700 1 --2- 172.21.15.178:0/3234395226 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f3d8802ff60 0x7f3d88032420 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:01:23.904 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:23.902+0000 7f3da29d5700 1 --2- 172.21.15.178:0/3234395226 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3d9c0f5410 0x7f3d9c105630 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:01:23.904 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:23.902+0000 7f3da29d5700 1 -- 172.21.15.178:0/3234395226 >> 172.21.15.178:0/3234395226 conn(0x7f3d9c0f0b70 msgr2=0x7f3d9c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:01:23.904 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:23.902+0000 7f3da29d5700 1 -- 172.21.15.178:0/3234395226 shutdown_connections 2024-02-11T17:01:23.904 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:23.902+0000 7f3da29d5700 1 -- 172.21.15.178:0/3234395226 wait complete. 2024-02-11T17:01:23.904 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:01:24.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:01:24 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:01:23.138680+0000 mgr.smithi037.eofnaq (mgr.14162) 41 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:01:24.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:01:24 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:01:23.901062+0000 mon.smithi037 (mon.0) 131 : audit [DBG] from='client.? 172.21.15.178:0/3234395226' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:01:25.567 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:01:25.567 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:01:26.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:01:26 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:01:25.139039+0000 mgr.smithi037.eofnaq (mgr.14162) 42 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:01:27.355 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:27.352+0000 7f96e2ff2700 1 -- 172.21.15.178:0/1959766909 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f96dc06b960 msgr2=0x7f96dc069e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:01:27.355 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:27.352+0000 7f96e2ff2700 1 --2- 172.21.15.178:0/1959766909 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f96dc06b960 0x7f96dc069e60 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f96d0009fb0 tx=0x7f96d0009b30).stop 2024-02-11T17:01:27.356 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:27.352+0000 7f96e2ff2700 1 -- 172.21.15.178:0/1959766909 shutdown_connections 2024-02-11T17:01:27.356 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:27.352+0000 7f96e2ff2700 1 --2- 172.21.15.178:0/1959766909 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f96dc06b960 0x7f96dc069e60 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:01:27.356 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:27.352+0000 7f96e2ff2700 1 -- 172.21.15.178:0/1959766909 >> 172.21.15.178:0/1959766909 conn(0x7f96dc0f1c70 msgr2=0x7f96dc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:01:27.356 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:27.352+0000 7f96e2ff2700 1 -- 172.21.15.178:0/1959766909 shutdown_connections 2024-02-11T17:01:27.356 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:27.352+0000 7f96e2ff2700 1 -- 172.21.15.178:0/1959766909 wait complete. 2024-02-11T17:01:27.356 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:27.353+0000 7f96e2ff2700 1 Processor -- start 2024-02-11T17:01:27.356 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:27.353+0000 7f96e2ff2700 1 -- start start 2024-02-11T17:01:27.356 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:27.353+0000 7f96e2ff2700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f96dc06b960 0x7f96dc1088e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:01:27.356 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:27.353+0000 7f96e2ff2700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96dc108dd0 con 0x7f96dc06b960 2024-02-11T17:01:27.356 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:27.353+0000 7f96e0d8e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f96dc06b960 0x7f96dc1088e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:01:27.356 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:27.354+0000 7f96e0d8e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f96dc06b960 0x7f96dc1088e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:47768/0 (socket says 172.21.15.178:47768) 2024-02-11T17:01:27.357 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:27.354+0000 7f96e0d8e700 1 -- 172.21.15.178:0/2412987476 learned_addr learned my addr 172.21.15.178:0/2412987476 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:01:27.357 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:27.354+0000 7f96e0d8e700 1 -- 172.21.15.178:0/2412987476 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96d0005040 con 0x7f96dc06b960 2024-02-11T17:01:27.357 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:27.354+0000 7f96e0d8e700 1 --2- 172.21.15.178:0/2412987476 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f96dc06b960 0x7f96dc1088e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f96d0013510 tx=0x7f96d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:01:27.358 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:27.354+0000 7f96d97fa700 1 -- 172.21.15.178:0/2412987476 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f96d0009320 con 0x7f96dc06b960 2024-02-11T17:01:27.358 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:27.354+0000 7f96e2ff2700 1 -- 172.21.15.178:0/2412987476 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96dc108fd0 con 0x7f96dc06b960 2024-02-11T17:01:27.358 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:27.354+0000 7f96e2ff2700 1 -- 172.21.15.178:0/2412987476 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96dc109460 con 0x7f96dc06b960 2024-02-11T17:01:27.358 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:27.354+0000 7f96d97fa700 1 -- 172.21.15.178:0/2412987476 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f96d0013f00 con 0x7f96dc06b960 2024-02-11T17:01:27.358 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:27.354+0000 7f96d97fa700 1 -- 172.21.15.178:0/2412987476 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f96d0014d90 con 0x7f96dc06b960 2024-02-11T17:01:27.358 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:27.355+0000 7f96d97fa700 1 -- 172.21.15.178:0/2412987476 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f96d00263a0 con 0x7f96dc06b960 2024-02-11T17:01:27.358 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:27.355+0000 7f96e2ff2700 1 -- 172.21.15.178:0/2412987476 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96dc045ad0 con 0x7f96dc06b960 2024-02-11T17:01:27.359 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:27.355+0000 7f96d97fa700 1 --2- 172.21.15.178:0/2412987476 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f96cc02ff60 0x7f96cc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:01:27.359 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:27.355+0000 7f96d97fa700 1 -- 172.21.15.178:0/2412987476 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f96d00412d0 con 0x7f96dc06b960 2024-02-11T17:01:27.359 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:27.355+0000 7f96dbfff700 1 --2- 172.21.15.178:0/2412987476 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f96cc02ff60 0x7f96cc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:01:27.359 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:27.356+0000 7f96dbfff700 1 --2- 172.21.15.178:0/2412987476 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f96cc02ff60 0x7f96cc032420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f96c80096a0 tx=0x7f96c8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:01:27.361 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:27.360+0000 7f96d97fa700 1 -- 172.21.15.178:0/2412987476 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f96d001c320 con 0x7f96dc06b960 2024-02-11T17:01:27.568 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:27.566+0000 7f96e2ff2700 1 -- 172.21.15.178:0/2412987476 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f96dc10a310 con 0x7f96dc06b960 2024-02-11T17:01:27.568 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:27.567+0000 7f96d97fa700 1 -- 172.21.15.178:0/2412987476 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f96d0012b20 con 0x7f96dc06b960 2024-02-11T17:01:27.569 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:01:27.569 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:01:27.570 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:27.568+0000 7f96e2ff2700 1 -- 172.21.15.178:0/2412987476 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f96cc02ff60 msgr2=0x7f96cc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:01:27.570 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:27.568+0000 7f96e2ff2700 1 --2- 172.21.15.178:0/2412987476 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f96cc02ff60 0x7f96cc032420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f96c80096a0 tx=0x7f96c8006b40).stop 2024-02-11T17:01:27.570 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:27.568+0000 7f96e2ff2700 1 -- 172.21.15.178:0/2412987476 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f96dc06b960 msgr2=0x7f96dc1088e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:01:27.570 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:27.568+0000 7f96e2ff2700 1 --2- 172.21.15.178:0/2412987476 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f96dc06b960 0x7f96dc1088e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f96d0013510 tx=0x7f96d000b040).stop 2024-02-11T17:01:27.570 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:27.569+0000 7f96e2ff2700 1 -- 172.21.15.178:0/2412987476 shutdown_connections 2024-02-11T17:01:27.571 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:27.569+0000 7f96e2ff2700 1 --2- 172.21.15.178:0/2412987476 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f96cc02ff60 0x7f96cc032420 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:01:27.571 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:27.569+0000 7f96e2ff2700 1 --2- 172.21.15.178:0/2412987476 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f96dc06b960 0x7f96dc1088e0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:01:27.571 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:27.569+0000 7f96e2ff2700 1 -- 172.21.15.178:0/2412987476 >> 172.21.15.178:0/2412987476 conn(0x7f96dc0f1c70 msgr2=0x7f96dc0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:01:27.571 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:27.569+0000 7f96e2ff2700 1 -- 172.21.15.178:0/2412987476 shutdown_connections 2024-02-11T17:01:27.571 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:27.569+0000 7f96e2ff2700 1 -- 172.21.15.178:0/2412987476 wait complete. 2024-02-11T17:01:27.571 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:01:28.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:01:28 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:01:27.139302+0000 mgr.smithi037.eofnaq (mgr.14162) 43 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:01:28.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:01:28 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:01:27.568187+0000 mon.smithi037 (mon.0) 132 : audit [DBG] from='client.? 172.21.15.178:0/2412987476' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:01:29.197 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:01:29.198 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:01:30.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:01:30 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:01:29.139618+0000 mgr.smithi037.eofnaq (mgr.14162) 44 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:01:31.015 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:31.013+0000 7fba9e1fe700 1 -- 172.21.15.178:0/2380782583 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba980f62c0 msgr2=0x7fba980f66e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:01:31.015 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:31.013+0000 7fba9e1fe700 1 --2- 172.21.15.178:0/2380782583 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba980f62c0 0x7fba980f66e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7fba88009fb0 tx=0x7fba88009b30).stop 2024-02-11T17:01:31.016 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:31.013+0000 7fba9e1fe700 1 -- 172.21.15.178:0/2380782583 shutdown_connections 2024-02-11T17:01:31.016 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:31.013+0000 7fba9e1fe700 1 --2- 172.21.15.178:0/2380782583 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba980f62c0 0x7fba980f66e0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:01:31.016 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:31.014+0000 7fba9e1fe700 1 -- 172.21.15.178:0/2380782583 >> 172.21.15.178:0/2380782583 conn(0x7fba980f1ac0 msgr2=0x7fba980f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:01:31.016 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:31.014+0000 7fba9e1fe700 1 -- 172.21.15.178:0/2380782583 shutdown_connections 2024-02-11T17:01:31.016 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:31.014+0000 7fba9e1fe700 1 -- 172.21.15.178:0/2380782583 wait complete. 2024-02-11T17:01:31.018 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:31.014+0000 7fba9e1fe700 1 Processor -- start 2024-02-11T17:01:31.018 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:31.014+0000 7fba9e1fe700 1 -- start start 2024-02-11T17:01:31.018 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:31.014+0000 7fba9e1fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba980f62c0 0x7fba98106520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:01:31.018 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:31.014+0000 7fba9e1fe700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba98106a10 con 0x7fba980f62c0 2024-02-11T17:01:31.018 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:31.015+0000 7fba9d1fc700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba980f62c0 0x7fba98106520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:01:31.018 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:31.015+0000 7fba9d1fc700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba980f62c0 0x7fba98106520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:51806/0 (socket says 172.21.15.178:51806) 2024-02-11T17:01:31.019 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:31.015+0000 7fba9d1fc700 1 -- 172.21.15.178:0/2990113065 learned_addr learned my addr 172.21.15.178:0/2990113065 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:01:31.019 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:31.015+0000 7fba9d1fc700 1 -- 172.21.15.178:0/2990113065 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba88005040 con 0x7fba980f62c0 2024-02-11T17:01:31.019 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:31.015+0000 7fba9d1fc700 1 --2- 172.21.15.178:0/2990113065 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba980f62c0 0x7fba98106520 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fba88013510 tx=0x7fba8800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:01:31.019 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:31.015+0000 7fba8dffb700 1 -- 172.21.15.178:0/2990113065 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba88009320 con 0x7fba980f62c0 2024-02-11T17:01:31.019 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:31.015+0000 7fba9e1fe700 1 -- 172.21.15.178:0/2990113065 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba98106c10 con 0x7fba980f62c0 2024-02-11T17:01:31.019 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:31.015+0000 7fba8dffb700 1 -- 172.21.15.178:0/2990113065 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fba88013f00 con 0x7fba980f62c0 2024-02-11T17:01:31.019 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:31.015+0000 7fba8dffb700 1 -- 172.21.15.178:0/2990113065 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba88014dc0 con 0x7fba980f62c0 2024-02-11T17:01:31.019 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:31.015+0000 7fba9e1fe700 1 -- 172.21.15.178:0/2990113065 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba981070a0 con 0x7fba980f62c0 2024-02-11T17:01:31.019 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:31.016+0000 7fba8dffb700 1 -- 172.21.15.178:0/2990113065 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fba8801d3d0 con 0x7fba980f62c0 2024-02-11T17:01:31.019 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:31.016+0000 7fba8dffb700 1 --2- 172.21.15.178:0/2990113065 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fba8402ffb0 0x7fba84032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:01:31.019 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:31.016+0000 7fba8dffb700 1 -- 172.21.15.178:0/2990113065 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fba880410b0 con 0x7fba980f62c0 2024-02-11T17:01:31.019 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:31.016+0000 7fba9e1fe700 1 -- 172.21.15.178:0/2990113065 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba98045ad0 con 0x7fba980f62c0 2024-02-11T17:01:31.019 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:31.017+0000 7fba9c9fb700 1 --2- 172.21.15.178:0/2990113065 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fba8402ffb0 0x7fba84032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:01:31.020 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:31.017+0000 7fba9c9fb700 1 --2- 172.21.15.178:0/2990113065 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fba8402ffb0 0x7fba84032470 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fba940096a0 tx=0x7fba94006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:01:31.023 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:31.021+0000 7fba8dffb700 1 -- 172.21.15.178:0/2990113065 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fba88012920 con 0x7fba980f62c0 2024-02-11T17:01:31.225 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:31.223+0000 7fba9e1fe700 1 -- 172.21.15.178:0/2990113065 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fba98107de0 con 0x7fba980f62c0 2024-02-11T17:01:31.226 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:31.224+0000 7fba8dffb700 1 -- 172.21.15.178:0/2990113065 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fba88012330 con 0x7fba980f62c0 2024-02-11T17:01:31.226 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:01:31.226 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:01:31.228 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:31.226+0000 7fba9e1fe700 1 -- 172.21.15.178:0/2990113065 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fba8402ffb0 msgr2=0x7fba84032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:01:31.228 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:31.226+0000 7fba9e1fe700 1 --2- 172.21.15.178:0/2990113065 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fba8402ffb0 0x7fba84032470 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fba940096a0 tx=0x7fba94006b40).stop 2024-02-11T17:01:31.228 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:31.226+0000 7fba9e1fe700 1 -- 172.21.15.178:0/2990113065 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba980f62c0 msgr2=0x7fba98106520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:01:31.228 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:31.226+0000 7fba9e1fe700 1 --2- 172.21.15.178:0/2990113065 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba980f62c0 0x7fba98106520 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fba88013510 tx=0x7fba8800b040).stop 2024-02-11T17:01:31.228 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:31.226+0000 7fba9e1fe700 1 -- 172.21.15.178:0/2990113065 shutdown_connections 2024-02-11T17:01:31.228 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:31.226+0000 7fba9e1fe700 1 --2- 172.21.15.178:0/2990113065 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fba8402ffb0 0x7fba84032470 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:01:31.228 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:31.226+0000 7fba9e1fe700 1 --2- 172.21.15.178:0/2990113065 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba980f62c0 0x7fba98106520 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:01:31.228 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:31.226+0000 7fba9e1fe700 1 -- 172.21.15.178:0/2990113065 >> 172.21.15.178:0/2990113065 conn(0x7fba980f1ac0 msgr2=0x7fba980f2530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:01:31.229 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:31.226+0000 7fba9e1fe700 1 -- 172.21.15.178:0/2990113065 shutdown_connections 2024-02-11T17:01:31.229 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:31.226+0000 7fba9e1fe700 1 -- 172.21.15.178:0/2990113065 wait complete. 2024-02-11T17:01:31.229 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:01:32.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:01:32 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:01:31.139902+0000 mgr.smithi037.eofnaq (mgr.14162) 45 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:01:32.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:01:32 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:01:31.225505+0000 mon.smithi037 (mon.0) 133 : audit [DBG] from='client.? 172.21.15.178:0/2990113065' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:01:32.869 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:01:32.869 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:01:34.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:01:34 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:01:33.140306+0000 mgr.smithi037.eofnaq (mgr.14162) 46 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:01:34.698 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:34.695+0000 7f8e95238700 1 -- 172.21.15.178:0/1614349195 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8e900f51a0 msgr2=0x7f8e900f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:01:34.698 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:34.695+0000 7f8e95238700 1 --2- 172.21.15.178:0/1614349195 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8e900f51a0 0x7f8e900f55c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f8e80009fb0 tx=0x7f8e80009b30).stop 2024-02-11T17:01:34.698 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:34.695+0000 7f8e95238700 1 -- 172.21.15.178:0/1614349195 shutdown_connections 2024-02-11T17:01:34.698 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:34.695+0000 7f8e95238700 1 --2- 172.21.15.178:0/1614349195 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8e900f51a0 0x7f8e900f55c0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:01:34.698 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:34.695+0000 7f8e95238700 1 -- 172.21.15.178:0/1614349195 >> 172.21.15.178:0/1614349195 conn(0x7f8e900f09a0 msgr2=0x7f8e900f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:01:34.698 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:34.695+0000 7f8e95238700 1 -- 172.21.15.178:0/1614349195 shutdown_connections 2024-02-11T17:01:34.698 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:34.695+0000 7f8e95238700 1 -- 172.21.15.178:0/1614349195 wait complete. 2024-02-11T17:01:34.698 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:34.695+0000 7f8e95238700 1 Processor -- start 2024-02-11T17:01:34.698 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:34.695+0000 7f8e95238700 1 -- start start 2024-02-11T17:01:34.699 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:34.696+0000 7f8e95238700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8e900f51a0 0x7f8e901064c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:01:34.699 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:34.696+0000 7f8e95238700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e901069b0 con 0x7f8e900f51a0 2024-02-11T17:01:34.699 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:34.696+0000 7f8e8ffff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8e900f51a0 0x7f8e901064c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:01:34.699 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:34.696+0000 7f8e8ffff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8e900f51a0 0x7f8e901064c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:51828/0 (socket says 172.21.15.178:51828) 2024-02-11T17:01:34.699 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:34.696+0000 7f8e8ffff700 1 -- 172.21.15.178:0/2116618547 learned_addr learned my addr 172.21.15.178:0/2116618547 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:01:34.699 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:34.696+0000 7f8e8ffff700 1 -- 172.21.15.178:0/2116618547 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e80005040 con 0x7f8e900f51a0 2024-02-11T17:01:34.699 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:34.696+0000 7f8e8ffff700 1 --2- 172.21.15.178:0/2116618547 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8e900f51a0 0x7f8e901064c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f8e80013510 tx=0x7f8e8000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:01:34.699 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:34.696+0000 7f8e8cff9700 1 -- 172.21.15.178:0/2116618547 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e80009320 con 0x7f8e900f51a0 2024-02-11T17:01:34.699 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:34.696+0000 7f8e95238700 1 -- 172.21.15.178:0/2116618547 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e90106bb0 con 0x7f8e900f51a0 2024-02-11T17:01:34.699 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:34.696+0000 7f8e8cff9700 1 -- 172.21.15.178:0/2116618547 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8e80013f00 con 0x7f8e900f51a0 2024-02-11T17:01:34.699 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:34.696+0000 7f8e8cff9700 1 -- 172.21.15.178:0/2116618547 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e80014dc0 con 0x7f8e900f51a0 2024-02-11T17:01:34.699 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:34.696+0000 7f8e95238700 1 -- 172.21.15.178:0/2116618547 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e90107040 con 0x7f8e900f51a0 2024-02-11T17:01:34.700 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:34.697+0000 7f8e8cff9700 1 -- 172.21.15.178:0/2116618547 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8e80014420 con 0x7f8e900f51a0 2024-02-11T17:01:34.700 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:34.697+0000 7f8e95238700 1 -- 172.21.15.178:0/2116618547 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e90045ad0 con 0x7f8e900f51a0 2024-02-11T17:01:34.700 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:34.697+0000 7f8e8cff9700 1 --2- 172.21.15.178:0/2116618547 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f8e7802ffb0 0x7f8e78032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:01:34.700 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:34.697+0000 7f8e8cff9700 1 -- 172.21.15.178:0/2116618547 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8e80041090 con 0x7f8e900f51a0 2024-02-11T17:01:34.700 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:34.698+0000 7f8e8f7fe700 1 --2- 172.21.15.178:0/2116618547 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f8e7802ffb0 0x7f8e78032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:01:34.701 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:34.698+0000 7f8e8f7fe700 1 --2- 172.21.15.178:0/2116618547 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f8e7802ffb0 0x7f8e78032470 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f8e840096a0 tx=0x7f8e84006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:01:34.704 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:34.702+0000 7f8e8cff9700 1 -- 172.21.15.178:0/2116618547 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8e8006d2b0 con 0x7f8e900f51a0 2024-02-11T17:01:34.907 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:34.906+0000 7f8e95238700 1 -- 172.21.15.178:0/2116618547 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8e90107ef0 con 0x7f8e900f51a0 2024-02-11T17:01:34.908 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:34.907+0000 7f8e8cff9700 1 -- 172.21.15.178:0/2116618547 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8e80026bf0 con 0x7f8e900f51a0 2024-02-11T17:01:34.909 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:01:34.909 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:01:34.910 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:34.908+0000 7f8e95238700 1 -- 172.21.15.178:0/2116618547 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f8e7802ffb0 msgr2=0x7f8e78032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:01:34.910 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:34.908+0000 7f8e95238700 1 --2- 172.21.15.178:0/2116618547 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f8e7802ffb0 0x7f8e78032470 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f8e840096a0 tx=0x7f8e84006b40).stop 2024-02-11T17:01:34.910 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:34.908+0000 7f8e95238700 1 -- 172.21.15.178:0/2116618547 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8e900f51a0 msgr2=0x7f8e901064c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:01:34.910 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:34.908+0000 7f8e95238700 1 --2- 172.21.15.178:0/2116618547 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8e900f51a0 0x7f8e901064c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f8e80013510 tx=0x7f8e8000b040).stop 2024-02-11T17:01:34.910 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:34.908+0000 7f8e95238700 1 -- 172.21.15.178:0/2116618547 shutdown_connections 2024-02-11T17:01:34.911 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:34.908+0000 7f8e95238700 1 --2- 172.21.15.178:0/2116618547 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f8e7802ffb0 0x7f8e78032470 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:01:34.911 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:34.908+0000 7f8e95238700 1 --2- 172.21.15.178:0/2116618547 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8e900f51a0 0x7f8e901064c0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:01:34.911 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:34.908+0000 7f8e95238700 1 -- 172.21.15.178:0/2116618547 >> 172.21.15.178:0/2116618547 conn(0x7f8e900f09a0 msgr2=0x7f8e900f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:01:34.911 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:34.908+0000 7f8e95238700 1 -- 172.21.15.178:0/2116618547 shutdown_connections 2024-02-11T17:01:34.911 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:34.908+0000 7f8e95238700 1 -- 172.21.15.178:0/2116618547 wait complete. 2024-02-11T17:01:34.911 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:01:35.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:01:35 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:01:34.907919+0000 mon.smithi037 (mon.0) 134 : audit [DBG] from='client.? 172.21.15.178:0/2116618547' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:01:36.482 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:01:36.482 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:01:36.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:01:36 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:01:35.140712+0000 mgr.smithi037.eofnaq (mgr.14162) 47 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:01:38.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:01:38 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:01:37.141100+0000 mgr.smithi037.eofnaq (mgr.14162) 48 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:01:39.101 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:39.091+0000 7fbbd713a700 1 -- 172.21.15.178:0/4062104528 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbbd00f51a0 msgr2=0x7fbbd00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:01:39.102 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:39.091+0000 7fbbd713a700 1 --2- 172.21.15.178:0/4062104528 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbbd00f51a0 0x7fbbd00f55c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7fbbc0009fb0 tx=0x7fbbc0009b30).stop 2024-02-11T17:01:39.102 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:39.091+0000 7fbbd713a700 1 -- 172.21.15.178:0/4062104528 shutdown_connections 2024-02-11T17:01:39.102 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:39.091+0000 7fbbd713a700 1 --2- 172.21.15.178:0/4062104528 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbbd00f51a0 0x7fbbd00f55c0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:01:39.102 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:39.091+0000 7fbbd713a700 1 -- 172.21.15.178:0/4062104528 >> 172.21.15.178:0/4062104528 conn(0x7fbbd00f09a0 msgr2=0x7fbbd00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:01:39.102 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:39.091+0000 7fbbd713a700 1 -- 172.21.15.178:0/4062104528 shutdown_connections 2024-02-11T17:01:39.102 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:39.091+0000 7fbbd713a700 1 -- 172.21.15.178:0/4062104528 wait complete. 2024-02-11T17:01:39.102 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:39.091+0000 7fbbd713a700 1 Processor -- start 2024-02-11T17:01:39.102 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:39.092+0000 7fbbd713a700 1 -- start start 2024-02-11T17:01:39.102 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:39.092+0000 7fbbd713a700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbbd00f51a0 0x7fbbd01053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:01:39.102 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:39.092+0000 7fbbd713a700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbd01058a0 con 0x7fbbd00f51a0 2024-02-11T17:01:39.102 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:39.092+0000 7fbbd6138700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbbd00f51a0 0x7fbbd01053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:01:39.103 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:39.092+0000 7fbbd6138700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbbd00f51a0 0x7fbbd01053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:51850/0 (socket says 172.21.15.178:51850) 2024-02-11T17:01:39.103 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:39.092+0000 7fbbd6138700 1 -- 172.21.15.178:0/2867456615 learned_addr learned my addr 172.21.15.178:0/2867456615 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:01:39.103 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:39.092+0000 7fbbd6138700 1 -- 172.21.15.178:0/2867456615 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbc0005040 con 0x7fbbd00f51a0 2024-02-11T17:01:39.103 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:39.092+0000 7fbbd6138700 1 --2- 172.21.15.178:0/2867456615 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbbd00f51a0 0x7fbbd01053b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fbbc0013510 tx=0x7fbbc000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:01:39.103 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:39.093+0000 7fbbc6ffd700 1 -- 172.21.15.178:0/2867456615 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbbc0009320 con 0x7fbbd00f51a0 2024-02-11T17:01:39.103 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:39.093+0000 7fbbd713a700 1 -- 172.21.15.178:0/2867456615 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbd0105aa0 con 0x7fbbd00f51a0 2024-02-11T17:01:39.103 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:39.093+0000 7fbbc6ffd700 1 -- 172.21.15.178:0/2867456615 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbbc0013f00 con 0x7fbbd00f51a0 2024-02-11T17:01:39.103 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:39.093+0000 7fbbc6ffd700 1 -- 172.21.15.178:0/2867456615 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbbc0014dc0 con 0x7fbbd00f51a0 2024-02-11T17:01:39.103 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:39.093+0000 7fbbd713a700 1 -- 172.21.15.178:0/2867456615 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbd0105f30 con 0x7fbbd00f51a0 2024-02-11T17:01:39.103 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:39.093+0000 7fbbc6ffd700 1 -- 172.21.15.178:0/2867456615 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbbc00095f0 con 0x7fbbd00f51a0 2024-02-11T17:01:39.103 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:39.093+0000 7fbbd713a700 1 -- 172.21.15.178:0/2867456615 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbd0045ad0 con 0x7fbbd00f51a0 2024-02-11T17:01:39.103 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:39.093+0000 7fbbc6ffd700 1 --2- 172.21.15.178:0/2867456615 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fbbbc02ff60 0x7fbbbc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:01:39.103 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:39.093+0000 7fbbc6ffd700 1 -- 172.21.15.178:0/2867456615 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbbc0041000 con 0x7fbbd00f51a0 2024-02-11T17:01:39.103 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:39.094+0000 7fbbd5937700 1 --2- 172.21.15.178:0/2867456615 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fbbbc02ff60 0x7fbbbc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:01:39.104 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:39.094+0000 7fbbd5937700 1 --2- 172.21.15.178:0/2867456615 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fbbbc02ff60 0x7fbbbc032420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7fbbcc0096a0 tx=0x7fbbcc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:01:39.104 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:39.098+0000 7fbbc6ffd700 1 -- 172.21.15.178:0/2867456615 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbbc0014420 con 0x7fbbd00f51a0 2024-02-11T17:01:39.307 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:39.305+0000 7fbbd713a700 1 -- 172.21.15.178:0/2867456615 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbbd0106de0 con 0x7fbbd00f51a0 2024-02-11T17:01:39.307 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:39.306+0000 7fbbc6ffd700 1 -- 172.21.15.178:0/2867456615 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbbc0010230 con 0x7fbbd00f51a0 2024-02-11T17:01:39.308 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:01:39.308 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:01:39.309 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:39.307+0000 7fbbd713a700 1 -- 172.21.15.178:0/2867456615 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fbbbc02ff60 msgr2=0x7fbbbc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:01:39.309 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:39.307+0000 7fbbd713a700 1 --2- 172.21.15.178:0/2867456615 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fbbbc02ff60 0x7fbbbc032420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7fbbcc0096a0 tx=0x7fbbcc006b40).stop 2024-02-11T17:01:39.309 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:39.307+0000 7fbbd713a700 1 -- 172.21.15.178:0/2867456615 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbbd00f51a0 msgr2=0x7fbbd01053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:01:39.310 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:39.307+0000 7fbbd713a700 1 --2- 172.21.15.178:0/2867456615 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbbd00f51a0 0x7fbbd01053b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fbbc0013510 tx=0x7fbbc000b040).stop 2024-02-11T17:01:39.310 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:39.307+0000 7fbbd713a700 1 -- 172.21.15.178:0/2867456615 shutdown_connections 2024-02-11T17:01:39.310 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:39.307+0000 7fbbd713a700 1 --2- 172.21.15.178:0/2867456615 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fbbbc02ff60 0x7fbbbc032420 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:01:39.310 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:39.307+0000 7fbbd713a700 1 --2- 172.21.15.178:0/2867456615 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbbd00f51a0 0x7fbbd01053b0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:01:39.310 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:39.307+0000 7fbbd713a700 1 -- 172.21.15.178:0/2867456615 >> 172.21.15.178:0/2867456615 conn(0x7fbbd00f09a0 msgr2=0x7fbbd00f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:01:39.310 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:39.307+0000 7fbbd713a700 1 -- 172.21.15.178:0/2867456615 shutdown_connections 2024-02-11T17:01:39.310 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:39.307+0000 7fbbd713a700 1 -- 172.21.15.178:0/2867456615 wait complete. 2024-02-11T17:01:39.311 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:01:40.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:01:40 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:01:39.141462+0000 mgr.smithi037.eofnaq (mgr.14162) 49 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:01:40.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:01:40 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:01:39.307022+0000 mon.smithi037 (mon.0) 135 : audit [DBG] from='client.? 172.21.15.178:0/2867456615' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:01:40.958 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:01:40.958 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:01:42.660 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:42.657+0000 7fd18d09c700 1 -- 172.21.15.178:0/3555034794 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd1880f6510 msgr2=0x7fd1880f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:01:42.661 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:42.657+0000 7fd18d09c700 1 --2- 172.21.15.178:0/3555034794 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd1880f6510 0x7fd1880f6930 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fd17c009fb0 tx=0x7fd17c009b30).stop 2024-02-11T17:01:42.661 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:42.657+0000 7fd18d09c700 1 -- 172.21.15.178:0/3555034794 shutdown_connections 2024-02-11T17:01:42.661 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:42.657+0000 7fd18d09c700 1 --2- 172.21.15.178:0/3555034794 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd1880f6510 0x7fd1880f6930 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:01:42.661 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:42.657+0000 7fd18d09c700 1 -- 172.21.15.178:0/3555034794 >> 172.21.15.178:0/3555034794 conn(0x7fd1880f1c70 msgr2=0x7fd1880f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:01:42.661 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:42.657+0000 7fd18d09c700 1 -- 172.21.15.178:0/3555034794 shutdown_connections 2024-02-11T17:01:42.661 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:42.657+0000 7fd18d09c700 1 -- 172.21.15.178:0/3555034794 wait complete. 2024-02-11T17:01:42.661 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:42.658+0000 7fd18d09c700 1 Processor -- start 2024-02-11T17:01:42.661 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:42.658+0000 7fd18d09c700 1 -- start start 2024-02-11T17:01:42.661 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:42.658+0000 7fd18d09c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd1880f6510 0x7fd18806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:01:42.661 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:42.658+0000 7fd18d09c700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd188069e20 con 0x7fd1880f6510 2024-02-11T17:01:42.661 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:42.658+0000 7fd186d9d700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd1880f6510 0x7fd18806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:01:42.662 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:42.658+0000 7fd186d9d700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd1880f6510 0x7fd18806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:42330/0 (socket says 172.21.15.178:42330) 2024-02-11T17:01:42.662 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:42.658+0000 7fd186d9d700 1 -- 172.21.15.178:0/1815135695 learned_addr learned my addr 172.21.15.178:0/1815135695 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:01:42.662 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:42.659+0000 7fd186d9d700 1 -- 172.21.15.178:0/1815135695 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd17c005040 con 0x7fd1880f6510 2024-02-11T17:01:42.662 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:42.659+0000 7fd186d9d700 1 --2- 172.21.15.178:0/1815135695 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd1880f6510 0x7fd18806b9a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fd17c013510 tx=0x7fd17c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:01:42.662 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:42.659+0000 7fd16f7fe700 1 -- 172.21.15.178:0/1815135695 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd17c009320 con 0x7fd1880f6510 2024-02-11T17:01:42.662 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:42.659+0000 7fd18d09c700 1 -- 172.21.15.178:0/1815135695 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd188069fc0 con 0x7fd1880f6510 2024-02-11T17:01:42.662 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:42.659+0000 7fd16f7fe700 1 -- 172.21.15.178:0/1815135695 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd17c013f00 con 0x7fd1880f6510 2024-02-11T17:01:42.662 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:42.659+0000 7fd16f7fe700 1 -- 172.21.15.178:0/1815135695 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd17c014d90 con 0x7fd1880f6510 2024-02-11T17:01:42.662 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:42.659+0000 7fd18d09c700 1 -- 172.21.15.178:0/1815135695 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd18806a450 con 0x7fd1880f6510 2024-02-11T17:01:42.663 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:42.660+0000 7fd16f7fe700 1 -- 172.21.15.178:0/1815135695 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd17c0263a0 con 0x7fd1880f6510 2024-02-11T17:01:42.663 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:42.660+0000 7fd18d09c700 1 -- 172.21.15.178:0/1815135695 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd188059100 con 0x7fd1880f6510 2024-02-11T17:01:42.663 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:42.660+0000 7fd16f7fe700 1 --2- 172.21.15.178:0/1815135695 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fd17002ff60 0x7fd170032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:01:42.663 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:42.660+0000 7fd16f7fe700 1 -- 172.21.15.178:0/1815135695 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd17c041090 con 0x7fd1880f6510 2024-02-11T17:01:42.663 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:42.660+0000 7fd18659c700 1 --2- 172.21.15.178:0/1815135695 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fd17002ff60 0x7fd170032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:01:42.663 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:42.660+0000 7fd18659c700 1 --2- 172.21.15.178:0/1815135695 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fd17002ff60 0x7fd170032420 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fd1780096a0 tx=0x7fd178006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:01:42.666 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:42.665+0000 7fd16f7fe700 1 -- 172.21.15.178:0/1815135695 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd17c0095f0 con 0x7fd1880f6510 2024-02-11T17:01:42.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:01:42 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:01:41.141895+0000 mgr.smithi037.eofnaq (mgr.14162) 50 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:01:42.870 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:42.869+0000 7fd18d09c700 1 -- 172.21.15.178:0/1815135695 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd188045ad0 con 0x7fd1880f6510 2024-02-11T17:01:42.871 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:42.869+0000 7fd16f7fe700 1 -- 172.21.15.178:0/1815135695 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd17c01db70 con 0x7fd1880f6510 2024-02-11T17:01:42.871 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:01:42.871 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:01:42.873 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:42.871+0000 7fd18d09c700 1 -- 172.21.15.178:0/1815135695 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fd17002ff60 msgr2=0x7fd170032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:01:42.873 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:42.871+0000 7fd18d09c700 1 --2- 172.21.15.178:0/1815135695 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fd17002ff60 0x7fd170032420 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fd1780096a0 tx=0x7fd178006b40).stop 2024-02-11T17:01:42.873 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:42.871+0000 7fd18d09c700 1 -- 172.21.15.178:0/1815135695 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd1880f6510 msgr2=0x7fd18806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:01:42.873 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:42.871+0000 7fd18d09c700 1 --2- 172.21.15.178:0/1815135695 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd1880f6510 0x7fd18806b9a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fd17c013510 tx=0x7fd17c00b040).stop 2024-02-11T17:01:42.873 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:42.871+0000 7fd18d09c700 1 -- 172.21.15.178:0/1815135695 shutdown_connections 2024-02-11T17:01:42.873 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:42.871+0000 7fd18d09c700 1 --2- 172.21.15.178:0/1815135695 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fd17002ff60 0x7fd170032420 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:01:42.873 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:42.871+0000 7fd18d09c700 1 --2- 172.21.15.178:0/1815135695 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd1880f6510 0x7fd18806b9a0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:01:42.873 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:42.871+0000 7fd18d09c700 1 -- 172.21.15.178:0/1815135695 >> 172.21.15.178:0/1815135695 conn(0x7fd1880f1c70 msgr2=0x7fd1880f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:01:42.873 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:42.871+0000 7fd18d09c700 1 -- 172.21.15.178:0/1815135695 shutdown_connections 2024-02-11T17:01:42.873 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:42.871+0000 7fd18d09c700 1 -- 172.21.15.178:0/1815135695 wait complete. 2024-02-11T17:01:42.874 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:01:43.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:01:43 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:01:42.870763+0000 mon.smithi037 (mon.0) 136 : audit [DBG] from='client.? 172.21.15.178:0/1815135695' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:01:44.490 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:01:44.490 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:01:44.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:01:44 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:01:43.142162+0000 mgr.smithi037.eofnaq (mgr.14162) 51 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:01:46.278 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:46.275+0000 7f9c0a54d700 1 -- 172.21.15.178:0/2844210974 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9c040f6510 msgr2=0x7f9c040f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:01:46.278 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:46.275+0000 7f9c0a54d700 1 --2- 172.21.15.178:0/2844210974 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9c040f6510 0x7f9c040f6930 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f9bf8009fb0 tx=0x7f9bf8009b30).stop 2024-02-11T17:01:46.279 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:46.275+0000 7f9c0a54d700 1 -- 172.21.15.178:0/2844210974 shutdown_connections 2024-02-11T17:01:46.279 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:46.275+0000 7f9c0a54d700 1 --2- 172.21.15.178:0/2844210974 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9c040f6510 0x7f9c040f6930 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:01:46.279 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:46.275+0000 7f9c0a54d700 1 -- 172.21.15.178:0/2844210974 >> 172.21.15.178:0/2844210974 conn(0x7f9c040f1c70 msgr2=0x7f9c040f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:01:46.279 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:46.275+0000 7f9c0a54d700 1 -- 172.21.15.178:0/2844210974 shutdown_connections 2024-02-11T17:01:46.279 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:46.275+0000 7f9c0a54d700 1 -- 172.21.15.178:0/2844210974 wait complete. 2024-02-11T17:01:46.279 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:46.275+0000 7f9c0a54d700 1 Processor -- start 2024-02-11T17:01:46.279 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:46.276+0000 7f9c0a54d700 1 -- start start 2024-02-11T17:01:46.279 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:46.276+0000 7f9c0a54d700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9c040f6510 0x7f9c04106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:01:46.279 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:46.276+0000 7f9c0a54d700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c04106c20 con 0x7f9c040f6510 2024-02-11T17:01:46.279 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:46.276+0000 7f9c03fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9c040f6510 0x7f9c04106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:01:46.279 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:46.276+0000 7f9c03fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9c040f6510 0x7f9c04106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:42350/0 (socket says 172.21.15.178:42350) 2024-02-11T17:01:46.279 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:46.276+0000 7f9c03fff700 1 -- 172.21.15.178:0/1908487262 learned_addr learned my addr 172.21.15.178:0/1908487262 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:01:46.280 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:46.276+0000 7f9c03fff700 1 -- 172.21.15.178:0/1908487262 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf8005040 con 0x7f9c040f6510 2024-02-11T17:01:46.280 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:46.276+0000 7f9c03fff700 1 --2- 172.21.15.178:0/1908487262 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9c040f6510 0x7f9c04106730 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f9bf8013510 tx=0x7f9bf800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:01:46.280 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:46.277+0000 7f9c00ff9700 1 -- 172.21.15.178:0/1908487262 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9bf8009320 con 0x7f9c040f6510 2024-02-11T17:01:46.280 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:46.277+0000 7f9c0a54d700 1 -- 172.21.15.178:0/1908487262 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c04106e20 con 0x7f9c040f6510 2024-02-11T17:01:46.280 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:46.277+0000 7f9c00ff9700 1 -- 172.21.15.178:0/1908487262 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9bf8013f00 con 0x7f9c040f6510 2024-02-11T17:01:46.280 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:46.277+0000 7f9c00ff9700 1 -- 172.21.15.178:0/1908487262 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9bf8014d90 con 0x7f9c040f6510 2024-02-11T17:01:46.280 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:46.277+0000 7f9c0a54d700 1 -- 172.21.15.178:0/1908487262 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c041072b0 con 0x7f9c040f6510 2024-02-11T17:01:46.281 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:46.277+0000 7f9c00ff9700 1 -- 172.21.15.178:0/1908487262 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9bf8014420 con 0x7f9c040f6510 2024-02-11T17:01:46.281 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:46.278+0000 7f9c00ff9700 1 --2- 172.21.15.178:0/1908487262 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f9bec02ff60 0x7f9bec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:01:46.281 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:46.278+0000 7f9c0a54d700 1 -- 172.21.15.178:0/1908487262 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c04045ad0 con 0x7f9c040f6510 2024-02-11T17:01:46.281 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:46.278+0000 7f9c00ff9700 1 -- 172.21.15.178:0/1908487262 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9bf80410c0 con 0x7f9c040f6510 2024-02-11T17:01:46.281 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:46.278+0000 7f9c037fe700 1 --2- 172.21.15.178:0/1908487262 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f9bec02ff60 0x7f9bec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:01:46.281 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:46.278+0000 7f9c037fe700 1 --2- 172.21.15.178:0/1908487262 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f9bec02ff60 0x7f9bec032420 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f9bf40096a0 tx=0x7f9bf4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:01:46.284 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:46.283+0000 7f9c00ff9700 1 -- 172.21.15.178:0/1908487262 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9bf806d2b0 con 0x7f9c040f6510 2024-02-11T17:01:46.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:01:46 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:01:45.142507+0000 mgr.smithi037.eofnaq (mgr.14162) 52 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:01:46.489 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:46.488+0000 7f9c0a54d700 1 -- 172.21.15.178:0/1908487262 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9c040fc050 con 0x7f9c040f6510 2024-02-11T17:01:46.490 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:46.488+0000 7f9c00ff9700 1 -- 172.21.15.178:0/1908487262 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9bf801dd50 con 0x7f9c040f6510 2024-02-11T17:01:46.490 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:01:46.490 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:01:46.492 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:46.490+0000 7f9c0a54d700 1 -- 172.21.15.178:0/1908487262 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f9bec02ff60 msgr2=0x7f9bec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:01:46.492 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:46.490+0000 7f9c0a54d700 1 --2- 172.21.15.178:0/1908487262 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f9bec02ff60 0x7f9bec032420 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f9bf40096a0 tx=0x7f9bf4006b40).stop 2024-02-11T17:01:46.492 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:46.490+0000 7f9c0a54d700 1 -- 172.21.15.178:0/1908487262 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9c040f6510 msgr2=0x7f9c04106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:01:46.492 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:46.490+0000 7f9c0a54d700 1 --2- 172.21.15.178:0/1908487262 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9c040f6510 0x7f9c04106730 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f9bf8013510 tx=0x7f9bf800b040).stop 2024-02-11T17:01:46.492 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:46.490+0000 7f9c0a54d700 1 -- 172.21.15.178:0/1908487262 shutdown_connections 2024-02-11T17:01:46.492 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:46.490+0000 7f9c0a54d700 1 --2- 172.21.15.178:0/1908487262 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f9bec02ff60 0x7f9bec032420 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:01:46.492 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:46.490+0000 7f9c0a54d700 1 --2- 172.21.15.178:0/1908487262 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9c040f6510 0x7f9c04106730 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:01:46.492 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:46.490+0000 7f9c0a54d700 1 -- 172.21.15.178:0/1908487262 >> 172.21.15.178:0/1908487262 conn(0x7f9c040f1c70 msgr2=0x7f9c040f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:01:46.492 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:46.490+0000 7f9c0a54d700 1 -- 172.21.15.178:0/1908487262 shutdown_connections 2024-02-11T17:01:46.492 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:46.490+0000 7f9c0a54d700 1 -- 172.21.15.178:0/1908487262 wait complete. 2024-02-11T17:01:46.493 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:01:47.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:01:47 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:01:46.489726+0000 mon.smithi037 (mon.0) 137 : audit [DBG] from='client.? 172.21.15.178:0/1908487262' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:01:48.060 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:01:48.060 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:01:48.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:01:48 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:01:47.142883+0000 mgr.smithi037.eofnaq (mgr.14162) 53 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:01:49.860 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:49.855+0000 7f414759e700 1 -- 172.21.15.178:0/1253917773 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41480f51a0 msgr2=0x7f41480f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:01:49.861 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:49.855+0000 7f414759e700 1 --2- 172.21.15.178:0/1253917773 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41480f51a0 0x7f41480f55c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f4138009fb0 tx=0x7f4138009b30).stop 2024-02-11T17:01:49.861 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:49.855+0000 7f414759e700 1 -- 172.21.15.178:0/1253917773 shutdown_connections 2024-02-11T17:01:49.861 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:49.855+0000 7f414759e700 1 --2- 172.21.15.178:0/1253917773 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41480f51a0 0x7f41480f55c0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:01:49.861 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:49.856+0000 7f414759e700 1 -- 172.21.15.178:0/1253917773 >> 172.21.15.178:0/1253917773 conn(0x7f41480f09a0 msgr2=0x7f41480f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:01:49.861 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:49.856+0000 7f414759e700 1 -- 172.21.15.178:0/1253917773 shutdown_connections 2024-02-11T17:01:49.861 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:49.856+0000 7f414759e700 1 -- 172.21.15.178:0/1253917773 wait complete. 2024-02-11T17:01:49.861 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:49.856+0000 7f414759e700 1 Processor -- start 2024-02-11T17:01:49.861 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:49.856+0000 7f414759e700 1 -- start start 2024-02-11T17:01:49.862 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:49.856+0000 7f414759e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41480f51a0 0x7f41481064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:01:49.862 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:49.856+0000 7f414759e700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4148106990 con 0x7f41480f51a0 2024-02-11T17:01:49.862 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:49.856+0000 7f414659c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41480f51a0 0x7f41481064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:01:49.862 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:49.856+0000 7f414659c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41480f51a0 0x7f41481064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:59410/0 (socket says 172.21.15.178:59410) 2024-02-11T17:01:49.862 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:49.856+0000 7f414659c700 1 -- 172.21.15.178:0/3354551985 learned_addr learned my addr 172.21.15.178:0/3354551985 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:01:49.862 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:49.857+0000 7f414659c700 1 -- 172.21.15.178:0/3354551985 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4138005040 con 0x7f41480f51a0 2024-02-11T17:01:49.862 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:49.857+0000 7f414659c700 1 --2- 172.21.15.178:0/3354551985 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41480f51a0 0x7f41481064a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f4138009830 tx=0x7f413800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:01:49.862 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:49.857+0000 7f4136ffd700 1 -- 172.21.15.178:0/3354551985 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4138009320 con 0x7f41480f51a0 2024-02-11T17:01:49.862 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:49.857+0000 7f414759e700 1 -- 172.21.15.178:0/3354551985 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4148106b90 con 0x7f41480f51a0 2024-02-11T17:01:49.862 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:49.857+0000 7f4136ffd700 1 -- 172.21.15.178:0/3354551985 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f41380136c0 con 0x7f41480f51a0 2024-02-11T17:01:49.862 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:49.857+0000 7f4136ffd700 1 -- 172.21.15.178:0/3354551985 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f413801ee90 con 0x7f41480f51a0 2024-02-11T17:01:49.862 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:49.857+0000 7f414759e700 1 -- 172.21.15.178:0/3354551985 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4148107020 con 0x7f41480f51a0 2024-02-11T17:01:49.862 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:49.858+0000 7f4136ffd700 1 -- 172.21.15.178:0/3354551985 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4138014440 con 0x7f41480f51a0 2024-02-11T17:01:49.862 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:49.858+0000 7f4136ffd700 1 --2- 172.21.15.178:0/3354551985 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f413002ff10 0x7f41300323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:01:49.863 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:49.858+0000 7f4136ffd700 1 -- 172.21.15.178:0/3354551985 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4138040f10 con 0x7f41480f51a0 2024-02-11T17:01:49.863 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:49.858+0000 7f414759e700 1 -- 172.21.15.178:0/3354551985 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4148045ad0 con 0x7f41480f51a0 2024-02-11T17:01:49.863 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:49.858+0000 7f4145d9b700 1 --2- 172.21.15.178:0/3354551985 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f413002ff10 0x7f41300323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:01:49.863 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:49.859+0000 7f4145d9b700 1 --2- 172.21.15.178:0/3354551985 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f413002ff10 0x7f41300323d0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f413c0096a0 tx=0x7f413c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:01:49.864 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:49.863+0000 7f4136ffd700 1 -- 172.21.15.178:0/3354551985 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f41380273f0 con 0x7f41480f51a0 2024-02-11T17:01:50.068 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:50.067+0000 7f414759e700 1 -- 172.21.15.178:0/3354551985 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f41480facd0 con 0x7f41480f51a0 2024-02-11T17:01:50.069 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:50.068+0000 7f4136ffd700 1 -- 172.21.15.178:0/3354551985 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f413801c320 con 0x7f41480f51a0 2024-02-11T17:01:50.069 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:01:50.069 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:01:50.071 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:50.069+0000 7f414759e700 1 -- 172.21.15.178:0/3354551985 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f413002ff10 msgr2=0x7f41300323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:01:50.071 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:50.069+0000 7f414759e700 1 --2- 172.21.15.178:0/3354551985 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f413002ff10 0x7f41300323d0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f413c0096a0 tx=0x7f413c006b40).stop 2024-02-11T17:01:50.071 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:50.069+0000 7f414759e700 1 -- 172.21.15.178:0/3354551985 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41480f51a0 msgr2=0x7f41481064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:01:50.071 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:50.069+0000 7f414759e700 1 --2- 172.21.15.178:0/3354551985 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41480f51a0 0x7f41481064a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f4138009830 tx=0x7f413800b040).stop 2024-02-11T17:01:50.071 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:50.069+0000 7f414759e700 1 -- 172.21.15.178:0/3354551985 shutdown_connections 2024-02-11T17:01:50.071 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:50.069+0000 7f414759e700 1 --2- 172.21.15.178:0/3354551985 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f413002ff10 0x7f41300323d0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:01:50.071 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:50.069+0000 7f414759e700 1 --2- 172.21.15.178:0/3354551985 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41480f51a0 0x7f41481064a0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:01:50.071 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:50.069+0000 7f414759e700 1 -- 172.21.15.178:0/3354551985 >> 172.21.15.178:0/3354551985 conn(0x7f41480f09a0 msgr2=0x7f414805fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:01:50.072 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:50.069+0000 7f414759e700 1 -- 172.21.15.178:0/3354551985 shutdown_connections 2024-02-11T17:01:50.072 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:50.069+0000 7f414759e700 1 -- 172.21.15.178:0/3354551985 wait complete. 2024-02-11T17:01:50.072 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:01:50.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:01:50 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:01:49.143264+0000 mgr.smithi037.eofnaq (mgr.14162) 54 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:01:50.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:01:50 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:01:50.068845+0000 mon.smithi037 (mon.0) 138 : audit [DBG] from='client.? 172.21.15.178:0/3354551985' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:01:51.724 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:01:51.724 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:01:52.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:01:52 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:01:51.143701+0000 mgr.smithi037.eofnaq (mgr.14162) 55 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:01:53.639 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:53.636+0000 7f3aa7e80700 1 -- 172.21.15.178:0/1298000765 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3aa0060220 msgr2=0x7f3aa00f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:01:53.639 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:53.636+0000 7f3aa7e80700 1 --2- 172.21.15.178:0/1298000765 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3aa0060220 0x7f3aa00f9150 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f3a9c009fb0 tx=0x7f3a9c009b30).stop 2024-02-11T17:01:53.640 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:53.636+0000 7f3aa7e80700 1 -- 172.21.15.178:0/1298000765 shutdown_connections 2024-02-11T17:01:53.640 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:53.636+0000 7f3aa7e80700 1 --2- 172.21.15.178:0/1298000765 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3aa0060220 0x7f3aa00f9150 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:01:53.640 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:53.636+0000 7f3aa7e80700 1 -- 172.21.15.178:0/1298000765 >> 172.21.15.178:0/1298000765 conn(0x7f3aa00f0b70 msgr2=0x7f3aa00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:01:53.640 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:53.636+0000 7f3aa7e80700 1 -- 172.21.15.178:0/1298000765 shutdown_connections 2024-02-11T17:01:53.640 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:53.636+0000 7f3aa7e80700 1 -- 172.21.15.178:0/1298000765 wait complete. 2024-02-11T17:01:53.640 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:53.637+0000 7f3aa7e80700 1 Processor -- start 2024-02-11T17:01:53.640 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:53.637+0000 7f3aa7e80700 1 -- start start 2024-02-11T17:01:53.640 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:53.637+0000 7f3aa7e80700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3aa0060220 0x7f3aa01066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:01:53.640 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:53.637+0000 7f3aa7e80700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa0106bd0 con 0x7f3aa0060220 2024-02-11T17:01:53.640 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:53.637+0000 7f3aa5c1c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3aa0060220 0x7f3aa01066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:01:53.641 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:53.637+0000 7f3aa5c1c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3aa0060220 0x7f3aa01066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:59430/0 (socket says 172.21.15.178:59430) 2024-02-11T17:01:53.641 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:53.637+0000 7f3aa5c1c700 1 -- 172.21.15.178:0/3679912262 learned_addr learned my addr 172.21.15.178:0/3679912262 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:01:53.641 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:53.638+0000 7f3aa5c1c700 1 -- 172.21.15.178:0/3679912262 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a9c005040 con 0x7f3aa0060220 2024-02-11T17:01:53.641 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:53.638+0000 7f3aa5c1c700 1 --2- 172.21.15.178:0/3679912262 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3aa0060220 0x7f3aa01066e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f3a9c013510 tx=0x7f3a9c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:01:53.641 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:53.638+0000 7f3a967fc700 1 -- 172.21.15.178:0/3679912262 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a9c009320 con 0x7f3aa0060220 2024-02-11T17:01:53.641 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:53.638+0000 7f3a967fc700 1 -- 172.21.15.178:0/3679912262 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3a9c013f00 con 0x7f3aa0060220 2024-02-11T17:01:53.641 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:53.638+0000 7f3aa7e80700 1 -- 172.21.15.178:0/3679912262 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3aa0106dd0 con 0x7f3aa0060220 2024-02-11T17:01:53.641 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:53.638+0000 7f3a967fc700 1 -- 172.21.15.178:0/3679912262 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a9c014dc0 con 0x7f3aa0060220 2024-02-11T17:01:53.641 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:53.638+0000 7f3aa7e80700 1 -- 172.21.15.178:0/3679912262 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3aa0107260 con 0x7f3aa0060220 2024-02-11T17:01:53.642 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:53.639+0000 7f3a967fc700 1 -- 172.21.15.178:0/3679912262 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3a9c014420 con 0x7f3aa0060220 2024-02-11T17:01:53.642 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:53.639+0000 7f3aa7e80700 1 -- 172.21.15.178:0/3679912262 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3aa0045ad0 con 0x7f3aa0060220 2024-02-11T17:01:53.642 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:53.639+0000 7f3a967fc700 1 --2- 172.21.15.178:0/3679912262 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f3a8c02ff60 0x7f3a8c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:01:53.642 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:53.639+0000 7f3a967fc700 1 -- 172.21.15.178:0/3679912262 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3a9c041110 con 0x7f3aa0060220 2024-02-11T17:01:53.642 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:53.639+0000 7f3aa541b700 1 --2- 172.21.15.178:0/3679912262 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f3a8c02ff60 0x7f3a8c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:01:53.642 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:53.639+0000 7f3aa541b700 1 --2- 172.21.15.178:0/3679912262 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f3a8c02ff60 0x7f3a8c032420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f3a900096a0 tx=0x7f3a90006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:01:53.645 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:53.644+0000 7f3a967fc700 1 -- 172.21.15.178:0/3679912262 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3a9c0095f0 con 0x7f3aa0060220 2024-02-11T17:01:53.850 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:53.848+0000 7f3aa7e80700 1 -- 172.21.15.178:0/3679912262 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3aa0108110 con 0x7f3aa0060220 2024-02-11T17:01:53.850 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:53.849+0000 7f3a967fc700 1 -- 172.21.15.178:0/3679912262 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3a9c01cb10 con 0x7f3aa0060220 2024-02-11T17:01:53.850 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:01:53.851 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:01:53.852 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:53.850+0000 7f3aa7e80700 1 -- 172.21.15.178:0/3679912262 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f3a8c02ff60 msgr2=0x7f3a8c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:01:53.852 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:53.850+0000 7f3aa7e80700 1 --2- 172.21.15.178:0/3679912262 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f3a8c02ff60 0x7f3a8c032420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f3a900096a0 tx=0x7f3a90006b40).stop 2024-02-11T17:01:53.852 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:53.850+0000 7f3aa7e80700 1 -- 172.21.15.178:0/3679912262 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3aa0060220 msgr2=0x7f3aa01066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:01:53.852 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:53.850+0000 7f3aa7e80700 1 --2- 172.21.15.178:0/3679912262 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3aa0060220 0x7f3aa01066e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f3a9c013510 tx=0x7f3a9c00b040).stop 2024-02-11T17:01:53.852 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:53.850+0000 7f3aa7e80700 1 -- 172.21.15.178:0/3679912262 shutdown_connections 2024-02-11T17:01:53.852 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:53.850+0000 7f3aa7e80700 1 --2- 172.21.15.178:0/3679912262 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f3a8c02ff60 0x7f3a8c032420 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:01:53.852 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:53.850+0000 7f3aa7e80700 1 --2- 172.21.15.178:0/3679912262 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3aa0060220 0x7f3aa01066e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:01:53.853 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:53.850+0000 7f3aa7e80700 1 -- 172.21.15.178:0/3679912262 >> 172.21.15.178:0/3679912262 conn(0x7f3aa00f0b70 msgr2=0x7f3aa00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:01:53.853 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:53.850+0000 7f3aa7e80700 1 -- 172.21.15.178:0/3679912262 shutdown_connections 2024-02-11T17:01:53.853 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:53.850+0000 7f3aa7e80700 1 -- 172.21.15.178:0/3679912262 wait complete. 2024-02-11T17:01:53.853 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:01:54.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:01:54 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:01:53.143933+0000 mgr.smithi037.eofnaq (mgr.14162) 56 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:01:54.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:01:54 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:01:53.850228+0000 mon.smithi037 (mon.0) 139 : audit [DBG] from='client.? 172.21.15.178:0/3679912262' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:01:55.481 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:01:55.481 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:01:56.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:01:56 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:01:55.144207+0000 mgr.smithi037.eofnaq (mgr.14162) 57 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:01:57.433 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:57.430+0000 7f29e925e700 1 -- 172.21.15.178:0/136179071 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f29e40f5410 msgr2=0x7f29e40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:01:57.433 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:57.430+0000 7f29e925e700 1 --2- 172.21.15.178:0/136179071 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f29e40f5410 0x7f29e40f5830 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f29d8009fb0 tx=0x7f29d8009b30).stop 2024-02-11T17:01:57.433 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:57.430+0000 7f29e925e700 1 -- 172.21.15.178:0/136179071 shutdown_connections 2024-02-11T17:01:57.433 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:57.430+0000 7f29e925e700 1 --2- 172.21.15.178:0/136179071 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f29e40f5410 0x7f29e40f5830 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:01:57.434 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:57.430+0000 7f29e925e700 1 -- 172.21.15.178:0/136179071 >> 172.21.15.178:0/136179071 conn(0x7f29e40f0b70 msgr2=0x7f29e40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:01:57.434 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:57.430+0000 7f29e925e700 1 -- 172.21.15.178:0/136179071 shutdown_connections 2024-02-11T17:01:57.434 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:57.430+0000 7f29e925e700 1 -- 172.21.15.178:0/136179071 wait complete. 2024-02-11T17:01:57.434 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:57.430+0000 7f29e925e700 1 Processor -- start 2024-02-11T17:01:57.434 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:57.431+0000 7f29e925e700 1 -- start start 2024-02-11T17:01:57.434 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:57.431+0000 7f29e925e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f29e40f5410 0x7f29e4106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:01:57.434 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:57.431+0000 7f29e925e700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e4106c10 con 0x7f29e40f5410 2024-02-11T17:01:57.434 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:57.431+0000 7f29e2d9d700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f29e40f5410 0x7f29e4106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:01:57.434 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:57.431+0000 7f29e2d9d700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f29e40f5410 0x7f29e4106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:59444/0 (socket says 172.21.15.178:59444) 2024-02-11T17:01:57.434 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:57.431+0000 7f29e2d9d700 1 -- 172.21.15.178:0/2712534025 learned_addr learned my addr 172.21.15.178:0/2712534025 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:01:57.434 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:57.431+0000 7f29e2d9d700 1 -- 172.21.15.178:0/2712534025 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29d8005040 con 0x7f29e40f5410 2024-02-11T17:01:57.434 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:57.431+0000 7f29e2d9d700 1 --2- 172.21.15.178:0/2712534025 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f29e40f5410 0x7f29e4106720 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f29d8013510 tx=0x7f29d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:01:57.434 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:57.431+0000 7f29cb7fe700 1 -- 172.21.15.178:0/2712534025 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29d8009320 con 0x7f29e40f5410 2024-02-11T17:01:57.435 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:57.432+0000 7f29e925e700 1 -- 172.21.15.178:0/2712534025 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29e4106e10 con 0x7f29e40f5410 2024-02-11T17:01:57.435 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:57.432+0000 7f29cb7fe700 1 -- 172.21.15.178:0/2712534025 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f29d8013f00 con 0x7f29e40f5410 2024-02-11T17:01:57.435 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:57.432+0000 7f29cb7fe700 1 -- 172.21.15.178:0/2712534025 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29d8014d90 con 0x7f29e40f5410 2024-02-11T17:01:57.435 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:57.432+0000 7f29e925e700 1 -- 172.21.15.178:0/2712534025 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29e41072a0 con 0x7f29e40f5410 2024-02-11T17:01:57.436 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:57.432+0000 7f29cb7fe700 1 -- 172.21.15.178:0/2712534025 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f29d80095f0 con 0x7f29e40f5410 2024-02-11T17:01:57.436 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:57.432+0000 7f29e925e700 1 -- 172.21.15.178:0/2712534025 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29e4045ad0 con 0x7f29e40f5410 2024-02-11T17:01:57.436 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:57.432+0000 7f29cb7fe700 1 --2- 172.21.15.178:0/2712534025 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f29cc02ff60 0x7f29cc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:01:57.436 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:57.433+0000 7f29cb7fe700 1 -- 172.21.15.178:0/2712534025 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f29d8041000 con 0x7f29e40f5410 2024-02-11T17:01:57.436 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:57.433+0000 7f29e259c700 1 --2- 172.21.15.178:0/2712534025 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f29cc02ff60 0x7f29cc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:01:57.436 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:57.433+0000 7f29e259c700 1 --2- 172.21.15.178:0/2712534025 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f29cc02ff60 0x7f29cc032420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f29d40096a0 tx=0x7f29d4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:01:57.439 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:57.437+0000 7f29cb7fe700 1 -- 172.21.15.178:0/2712534025 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f29d8014420 con 0x7f29e40f5410 2024-02-11T17:01:57.643 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:57.641+0000 7f29e925e700 1 -- 172.21.15.178:0/2712534025 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f29e4108150 con 0x7f29e40f5410 2024-02-11T17:01:57.643 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:57.641+0000 7f29cb7fe700 1 -- 172.21.15.178:0/2712534025 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f29d8010230 con 0x7f29e40f5410 2024-02-11T17:01:57.643 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:01:57.643 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:01:57.645 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:57.643+0000 7f29e925e700 1 -- 172.21.15.178:0/2712534025 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f29cc02ff60 msgr2=0x7f29cc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:01:57.645 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:57.643+0000 7f29e925e700 1 --2- 172.21.15.178:0/2712534025 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f29cc02ff60 0x7f29cc032420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f29d40096a0 tx=0x7f29d4006b40).stop 2024-02-11T17:01:57.645 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:57.643+0000 7f29e925e700 1 -- 172.21.15.178:0/2712534025 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f29e40f5410 msgr2=0x7f29e4106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:01:57.645 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:57.643+0000 7f29e925e700 1 --2- 172.21.15.178:0/2712534025 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f29e40f5410 0x7f29e4106720 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f29d8013510 tx=0x7f29d800b040).stop 2024-02-11T17:01:57.645 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:57.643+0000 7f29e925e700 1 -- 172.21.15.178:0/2712534025 shutdown_connections 2024-02-11T17:01:57.645 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:57.643+0000 7f29e925e700 1 --2- 172.21.15.178:0/2712534025 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f29cc02ff60 0x7f29cc032420 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:01:57.645 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:57.643+0000 7f29e925e700 1 --2- 172.21.15.178:0/2712534025 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f29e40f5410 0x7f29e4106720 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:01:57.645 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:57.643+0000 7f29e925e700 1 -- 172.21.15.178:0/2712534025 >> 172.21.15.178:0/2712534025 conn(0x7f29e40f0b70 msgr2=0x7f29e405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:01:57.645 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:57.643+0000 7f29e925e700 1 -- 172.21.15.178:0/2712534025 shutdown_connections 2024-02-11T17:01:57.646 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:01:57.643+0000 7f29e925e700 1 -- 172.21.15.178:0/2712534025 wait complete. 2024-02-11T17:01:57.646 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:01:58.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:01:58 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:01:57.144531+0000 mgr.smithi037.eofnaq (mgr.14162) 58 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:01:58.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:01:58 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:01:57.642926+0000 mon.smithi037 (mon.0) 140 : audit [DBG] from='client.? 172.21.15.178:0/2712534025' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:01:59.332 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:01:59.332 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:02:00.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:02:00 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:01:59.144799+0000 mgr.smithi037.eofnaq (mgr.14162) 59 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:02:01.138 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:01.135+0000 7f5760bf2700 1 -- 172.21.15.178:0/2571149633 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f575c060280 msgr2=0x7f575c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:02:01.138 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:01.135+0000 7f5760bf2700 1 --2- 172.21.15.178:0/2571149633 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f575c060280 0x7f575c0f90f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f574c009fb0 tx=0x7f574c009b30).stop 2024-02-11T17:02:01.138 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:01.135+0000 7f5760bf2700 1 -- 172.21.15.178:0/2571149633 shutdown_connections 2024-02-11T17:02:01.138 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:01.135+0000 7f5760bf2700 1 --2- 172.21.15.178:0/2571149633 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f575c060280 0x7f575c0f90f0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:02:01.139 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:01.135+0000 7f5760bf2700 1 -- 172.21.15.178:0/2571149633 >> 172.21.15.178:0/2571149633 conn(0x7f575c0f0b70 msgr2=0x7f575c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:02:01.139 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:01.135+0000 7f5760bf2700 1 -- 172.21.15.178:0/2571149633 shutdown_connections 2024-02-11T17:02:01.139 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:01.135+0000 7f5760bf2700 1 -- 172.21.15.178:0/2571149633 wait complete. 2024-02-11T17:02:01.139 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:01.136+0000 7f5760bf2700 1 Processor -- start 2024-02-11T17:02:01.139 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:01.136+0000 7f5760bf2700 1 -- start start 2024-02-11T17:02:01.139 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:01.136+0000 7f5760bf2700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f575c060280 0x7f575c1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:02:01.139 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:01.136+0000 7f5760bf2700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f575c106bd0 con 0x7f575c060280 2024-02-11T17:02:01.139 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:01.136+0000 7f575a59c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f575c060280 0x7f575c1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:02:01.139 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:01.136+0000 7f575a59c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f575c060280 0x7f575c1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:54484/0 (socket says 172.21.15.178:54484) 2024-02-11T17:02:01.139 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:01.136+0000 7f575a59c700 1 -- 172.21.15.178:0/689287503 learned_addr learned my addr 172.21.15.178:0/689287503 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:02:01.139 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:01.136+0000 7f575a59c700 1 -- 172.21.15.178:0/689287503 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f574c005040 con 0x7f575c060280 2024-02-11T17:02:01.139 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:01.136+0000 7f575a59c700 1 --2- 172.21.15.178:0/689287503 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f575c060280 0x7f575c1066e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f574c013510 tx=0x7f574c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:02:01.139 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:01.137+0000 7f574affd700 1 -- 172.21.15.178:0/689287503 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f574c009320 con 0x7f575c060280 2024-02-11T17:02:01.139 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:01.137+0000 7f5760bf2700 1 -- 172.21.15.178:0/689287503 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f575c106dd0 con 0x7f575c060280 2024-02-11T17:02:01.139 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:01.137+0000 7f574affd700 1 -- 172.21.15.178:0/689287503 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f574c013f00 con 0x7f575c060280 2024-02-11T17:02:01.140 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:01.137+0000 7f574affd700 1 -- 172.21.15.178:0/689287503 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f574c014dc0 con 0x7f575c060280 2024-02-11T17:02:01.140 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:01.137+0000 7f5760bf2700 1 -- 172.21.15.178:0/689287503 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f575c107260 con 0x7f575c060280 2024-02-11T17:02:01.140 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:01.137+0000 7f574affd700 1 -- 172.21.15.178:0/689287503 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f574c0095f0 con 0x7f575c060280 2024-02-11T17:02:01.141 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:01.137+0000 7f5760bf2700 1 -- 172.21.15.178:0/689287503 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f575c045ad0 con 0x7f575c060280 2024-02-11T17:02:01.141 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:01.138+0000 7f574affd700 1 --2- 172.21.15.178:0/689287503 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f574402ff60 0x7f5744032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:02:01.141 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:01.138+0000 7f574affd700 1 -- 172.21.15.178:0/689287503 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f574c041040 con 0x7f575c060280 2024-02-11T17:02:01.141 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:01.138+0000 7f5759d9b700 1 --2- 172.21.15.178:0/689287503 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f574402ff60 0x7f5744032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:02:01.141 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:01.138+0000 7f5759d9b700 1 --2- 172.21.15.178:0/689287503 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f574402ff60 0x7f5744032420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f57500096a0 tx=0x7f5750006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:02:01.144 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:01.142+0000 7f574affd700 1 -- 172.21.15.178:0/689287503 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f574c014420 con 0x7f575c060280 2024-02-11T17:02:01.349 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:01.347+0000 7f5760bf2700 1 -- 172.21.15.178:0/689287503 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f575c108110 con 0x7f575c060280 2024-02-11T17:02:01.350 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:01.348+0000 7f574affd700 1 -- 172.21.15.178:0/689287503 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f574c010230 con 0x7f575c060280 2024-02-11T17:02:01.350 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:02:01.350 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:02:01.352 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:01.350+0000 7f5760bf2700 1 -- 172.21.15.178:0/689287503 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f574402ff60 msgr2=0x7f5744032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:02:01.352 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:01.350+0000 7f5760bf2700 1 --2- 172.21.15.178:0/689287503 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f574402ff60 0x7f5744032420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f57500096a0 tx=0x7f5750006b40).stop 2024-02-11T17:02:01.352 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:01.350+0000 7f5760bf2700 1 -- 172.21.15.178:0/689287503 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f575c060280 msgr2=0x7f575c1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:02:01.352 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:01.350+0000 7f5760bf2700 1 --2- 172.21.15.178:0/689287503 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f575c060280 0x7f575c1066e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f574c013510 tx=0x7f574c00b040).stop 2024-02-11T17:02:01.352 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:01.350+0000 7f5760bf2700 1 -- 172.21.15.178:0/689287503 shutdown_connections 2024-02-11T17:02:01.352 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:01.350+0000 7f5760bf2700 1 --2- 172.21.15.178:0/689287503 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f574402ff60 0x7f5744032420 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:02:01.353 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:01.350+0000 7f5760bf2700 1 --2- 172.21.15.178:0/689287503 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f575c060280 0x7f575c1066e0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:02:01.353 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:01.350+0000 7f5760bf2700 1 -- 172.21.15.178:0/689287503 >> 172.21.15.178:0/689287503 conn(0x7f575c0f0b70 msgr2=0x7f575c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:02:01.353 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:01.350+0000 7f5760bf2700 1 -- 172.21.15.178:0/689287503 shutdown_connections 2024-02-11T17:02:01.353 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:01.350+0000 7f5760bf2700 1 -- 172.21.15.178:0/689287503 wait complete. 2024-02-11T17:02:01.353 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:02:02.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:02:02 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:02:01.145036+0000 mgr.smithi037.eofnaq (mgr.14162) 60 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:02:02.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:02:02 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:02:01.349522+0000 mon.smithi037 (mon.0) 141 : audit [DBG] from='client.? 172.21.15.178:0/689287503' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:02:02.963 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:02:02.963 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:02:04.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:02:04 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:02:03.145289+0000 mgr.smithi037.eofnaq (mgr.14162) 61 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:02:04.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:02:04 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:02:03.271622+0000 mon.smithi037 (mon.0) 142 : audit [INF] from='mgr.14162 172.21.15.37:0/2745465012' entity='mgr.smithi037.eofnaq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi037.eofnaq/mirror_snapshot_schedule"}]: dispatch 2024-02-11T17:02:04.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:02:04 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:02:03.275479+0000 mon.smithi037 (mon.0) 143 : audit [INF] from='mgr.14162 172.21.15.37:0/2745465012' entity='mgr.smithi037.eofnaq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi037.eofnaq/trash_purge_schedule"}]: dispatch 2024-02-11T17:02:04.887 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:04.884+0000 7f0a3e4b8700 1 -- 172.21.15.178:0/1416694222 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0a380f5410 msgr2=0x7f0a380f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:02:04.887 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:04.884+0000 7f0a3e4b8700 1 --2- 172.21.15.178:0/1416694222 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0a380f5410 0x7f0a380f5830 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f0a28009fb0 tx=0x7f0a28009b30).stop 2024-02-11T17:02:04.887 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:04.885+0000 7f0a3e4b8700 1 -- 172.21.15.178:0/1416694222 shutdown_connections 2024-02-11T17:02:04.887 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:04.885+0000 7f0a3e4b8700 1 --2- 172.21.15.178:0/1416694222 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0a380f5410 0x7f0a380f5830 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:02:04.887 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:04.885+0000 7f0a3e4b8700 1 -- 172.21.15.178:0/1416694222 >> 172.21.15.178:0/1416694222 conn(0x7f0a380f0b70 msgr2=0x7f0a380f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:02:04.887 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:04.885+0000 7f0a3e4b8700 1 -- 172.21.15.178:0/1416694222 shutdown_connections 2024-02-11T17:02:04.887 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:04.885+0000 7f0a3e4b8700 1 -- 172.21.15.178:0/1416694222 wait complete. 2024-02-11T17:02:04.887 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:04.885+0000 7f0a3e4b8700 1 Processor -- start 2024-02-11T17:02:04.888 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:04.885+0000 7f0a3e4b8700 1 -- start start 2024-02-11T17:02:04.888 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:04.885+0000 7f0a3e4b8700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0a38060200 0x7f0a38109560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:02:04.888 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:04.885+0000 7f0a3e4b8700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a38109a50 con 0x7f0a38060200 2024-02-11T17:02:04.889 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:04.886+0000 7f0a37fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0a38060200 0x7f0a38109560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:02:04.889 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:04.886+0000 7f0a37fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0a38060200 0x7f0a38109560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:54504/0 (socket says 172.21.15.178:54504) 2024-02-11T17:02:04.889 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:04.886+0000 7f0a37fff700 1 -- 172.21.15.178:0/1668411574 learned_addr learned my addr 172.21.15.178:0/1668411574 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:02:04.889 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:04.886+0000 7f0a37fff700 1 -- 172.21.15.178:0/1668411574 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a28005040 con 0x7f0a38060200 2024-02-11T17:02:04.890 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:04.886+0000 7f0a37fff700 1 --2- 172.21.15.178:0/1668411574 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0a38060200 0x7f0a38109560 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f0a28009f80 tx=0x7f0a28009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:02:04.890 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:04.886+0000 7f0a34ff9700 1 -- 172.21.15.178:0/1668411574 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a280136c0 con 0x7f0a38060200 2024-02-11T17:02:04.890 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:04.886+0000 7f0a3e4b8700 1 -- 172.21.15.178:0/1668411574 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a38109b90 con 0x7f0a38060200 2024-02-11T17:02:04.890 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:04.886+0000 7f0a34ff9700 1 -- 172.21.15.178:0/1668411574 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0a280095f0 con 0x7f0a38060200 2024-02-11T17:02:04.890 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:04.886+0000 7f0a34ff9700 1 -- 172.21.15.178:0/1668411574 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a28014420 con 0x7f0a38060200 2024-02-11T17:02:04.890 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:04.886+0000 7f0a3e4b8700 1 -- 172.21.15.178:0/1668411574 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a38109fc0 con 0x7f0a38060200 2024-02-11T17:02:04.890 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:04.887+0000 7f0a34ff9700 1 -- 172.21.15.178:0/1668411574 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0a28025060 con 0x7f0a38060200 2024-02-11T17:02:04.890 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:04.887+0000 7f0a3e4b8700 1 -- 172.21.15.178:0/1668411574 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a38059100 con 0x7f0a38060200 2024-02-11T17:02:04.890 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:04.887+0000 7f0a34ff9700 1 --2- 172.21.15.178:0/1668411574 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f0a2002ff60 0x7f0a20032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:02:04.890 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:04.887+0000 7f0a34ff9700 1 -- 172.21.15.178:0/1668411574 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0a28040e60 con 0x7f0a38060200 2024-02-11T17:02:04.890 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:04.888+0000 7f0a377fe700 1 --2- 172.21.15.178:0/1668411574 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f0a2002ff60 0x7f0a20032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:02:04.890 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:04.888+0000 7f0a377fe700 1 --2- 172.21.15.178:0/1668411574 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f0a2002ff60 0x7f0a20032420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f0a2c0096a0 tx=0x7f0a2c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:02:04.893 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:04.892+0000 7f0a34ff9700 1 -- 172.21.15.178:0/1668411574 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0a2801c950 con 0x7f0a38060200 2024-02-11T17:02:05.098 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:05.096+0000 7f0a3e4b8700 1 -- 172.21.15.178:0/1668411574 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0a38045ad0 con 0x7f0a38060200 2024-02-11T17:02:05.098 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:05.097+0000 7f0a34ff9700 1 -- 172.21.15.178:0/1668411574 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0a2801cb60 con 0x7f0a38060200 2024-02-11T17:02:05.099 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:02:05.099 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:02:05.100 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:05.098+0000 7f0a3e4b8700 1 -- 172.21.15.178:0/1668411574 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f0a2002ff60 msgr2=0x7f0a20032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:02:05.100 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:05.098+0000 7f0a3e4b8700 1 --2- 172.21.15.178:0/1668411574 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f0a2002ff60 0x7f0a20032420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f0a2c0096a0 tx=0x7f0a2c006b40).stop 2024-02-11T17:02:05.100 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:05.098+0000 7f0a3e4b8700 1 -- 172.21.15.178:0/1668411574 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0a38060200 msgr2=0x7f0a38109560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:02:05.100 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:05.098+0000 7f0a3e4b8700 1 --2- 172.21.15.178:0/1668411574 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0a38060200 0x7f0a38109560 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f0a28009f80 tx=0x7f0a28009b30).stop 2024-02-11T17:02:05.100 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:05.098+0000 7f0a3e4b8700 1 -- 172.21.15.178:0/1668411574 shutdown_connections 2024-02-11T17:02:05.100 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:05.098+0000 7f0a3e4b8700 1 --2- 172.21.15.178:0/1668411574 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f0a2002ff60 0x7f0a20032420 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:02:05.101 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:05.098+0000 7f0a3e4b8700 1 --2- 172.21.15.178:0/1668411574 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0a38060200 0x7f0a38109560 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:02:05.101 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:05.098+0000 7f0a3e4b8700 1 -- 172.21.15.178:0/1668411574 >> 172.21.15.178:0/1668411574 conn(0x7f0a380f0b70 msgr2=0x7f0a380f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:02:05.101 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:05.098+0000 7f0a3e4b8700 1 -- 172.21.15.178:0/1668411574 shutdown_connections 2024-02-11T17:02:05.101 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:05.098+0000 7f0a3e4b8700 1 -- 172.21.15.178:0/1668411574 wait complete. 2024-02-11T17:02:05.101 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:02:06.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:02:06 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:02:05.098187+0000 mon.smithi037 (mon.0) 144 : audit [DBG] from='client.? 172.21.15.178:0/1668411574' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:02:06.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:02:06 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:02:05.145711+0000 mgr.smithi037.eofnaq (mgr.14162) 62 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:02:06.760 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:02:06.761 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:02:08.531 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:08.529+0000 7f267a542700 1 -- 172.21.15.178:0/11698908 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26740f51a0 msgr2=0x7f26740f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:02:08.531 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:08.529+0000 7f267a542700 1 --2- 172.21.15.178:0/11698908 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26740f51a0 0x7f26740f55c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f2670009fb0 tx=0x7f2670009b30).stop 2024-02-11T17:02:08.531 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:08.529+0000 7f267a542700 1 -- 172.21.15.178:0/11698908 shutdown_connections 2024-02-11T17:02:08.532 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:08.529+0000 7f267a542700 1 --2- 172.21.15.178:0/11698908 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26740f51a0 0x7f26740f55c0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:02:08.532 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:08.529+0000 7f267a542700 1 -- 172.21.15.178:0/11698908 >> 172.21.15.178:0/11698908 conn(0x7f26740f09a0 msgr2=0x7f26740f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:02:08.532 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:08.529+0000 7f267a542700 1 -- 172.21.15.178:0/11698908 shutdown_connections 2024-02-11T17:02:08.532 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:08.529+0000 7f267a542700 1 -- 172.21.15.178:0/11698908 wait complete. 2024-02-11T17:02:08.532 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:08.530+0000 7f267a542700 1 Processor -- start 2024-02-11T17:02:08.532 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:08.530+0000 7f267a542700 1 -- start start 2024-02-11T17:02:08.532 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:08.530+0000 7f267a542700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26740f51a0 0x7f26741091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:02:08.532 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:08.530+0000 7f267a542700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2674060240 con 0x7f26740f51a0 2024-02-11T17:02:08.534 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:08.530+0000 7f2679540700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26740f51a0 0x7f26741091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:02:08.534 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:08.530+0000 7f2679540700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26740f51a0 0x7f26741091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:54524/0 (socket says 172.21.15.178:54524) 2024-02-11T17:02:08.534 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:08.530+0000 7f2679540700 1 -- 172.21.15.178:0/747801383 learned_addr learned my addr 172.21.15.178:0/747801383 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:02:08.534 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:08.530+0000 7f2679540700 1 -- 172.21.15.178:0/747801383 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2670005040 con 0x7f26740f51a0 2024-02-11T17:02:08.534 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:08.531+0000 7f2679540700 1 --2- 172.21.15.178:0/747801383 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26740f51a0 0x7f26741091b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f2670013510 tx=0x7f267000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:02:08.534 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:08.531+0000 7f2669ffb700 1 -- 172.21.15.178:0/747801383 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2670009320 con 0x7f26740f51a0 2024-02-11T17:02:08.534 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:08.531+0000 7f267a542700 1 -- 172.21.15.178:0/747801383 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2674060440 con 0x7f26740f51a0 2024-02-11T17:02:08.534 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:08.531+0000 7f2669ffb700 1 -- 172.21.15.178:0/747801383 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2670013f00 con 0x7f26740f51a0 2024-02-11T17:02:08.534 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:08.531+0000 7f2669ffb700 1 -- 172.21.15.178:0/747801383 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2670014dc0 con 0x7f26740f51a0 2024-02-11T17:02:08.535 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:08.531+0000 7f267a542700 1 -- 172.21.15.178:0/747801383 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2674109860 con 0x7f26740f51a0 2024-02-11T17:02:08.535 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:08.532+0000 7f2669ffb700 1 -- 172.21.15.178:0/747801383 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f26700095f0 con 0x7f26740f51a0 2024-02-11T17:02:08.535 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:08.532+0000 7f267a542700 1 -- 172.21.15.178:0/747801383 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2674045ad0 con 0x7f26740f51a0 2024-02-11T17:02:08.535 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:08.532+0000 7f2669ffb700 1 --2- 172.21.15.178:0/747801383 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f266002ff60 0x7f2660032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:02:08.535 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:08.532+0000 7f2669ffb700 1 -- 172.21.15.178:0/747801383 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2670041040 con 0x7f26740f51a0 2024-02-11T17:02:08.535 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:08.532+0000 7f2678d3f700 1 --2- 172.21.15.178:0/747801383 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f266002ff60 0x7f2660032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:02:08.535 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:08.533+0000 7f2678d3f700 1 --2- 172.21.15.178:0/747801383 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f266002ff60 0x7f2660032420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f26640096a0 tx=0x7f2664006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:02:08.538 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:08.537+0000 7f2669ffb700 1 -- 172.21.15.178:0/747801383 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2670014420 con 0x7f26740f51a0 2024-02-11T17:02:08.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:02:08 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:02:07.146071+0000 mgr.smithi037.eofnaq (mgr.14162) 63 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:02:08.741 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:08.739+0000 7f267a542700 1 -- 172.21.15.178:0/747801383 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f267410a080 con 0x7f26740f51a0 2024-02-11T17:02:08.742 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:08.740+0000 7f2669ffb700 1 -- 172.21.15.178:0/747801383 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2670010230 con 0x7f26740f51a0 2024-02-11T17:02:08.742 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:02:08.742 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:02:08.743 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:08.742+0000 7f267a542700 1 -- 172.21.15.178:0/747801383 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f266002ff60 msgr2=0x7f2660032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:02:08.744 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:08.742+0000 7f267a542700 1 --2- 172.21.15.178:0/747801383 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f266002ff60 0x7f2660032420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f26640096a0 tx=0x7f2664006b40).stop 2024-02-11T17:02:08.744 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:08.742+0000 7f267a542700 1 -- 172.21.15.178:0/747801383 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26740f51a0 msgr2=0x7f26741091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:02:08.744 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:08.742+0000 7f267a542700 1 --2- 172.21.15.178:0/747801383 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26740f51a0 0x7f26741091b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f2670013510 tx=0x7f267000b040).stop 2024-02-11T17:02:08.744 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:08.742+0000 7f267a542700 1 -- 172.21.15.178:0/747801383 shutdown_connections 2024-02-11T17:02:08.744 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:08.742+0000 7f267a542700 1 --2- 172.21.15.178:0/747801383 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f266002ff60 0x7f2660032420 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:02:08.744 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:08.742+0000 7f267a542700 1 --2- 172.21.15.178:0/747801383 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26740f51a0 0x7f26741091b0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:02:08.744 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:08.742+0000 7f267a542700 1 -- 172.21.15.178:0/747801383 >> 172.21.15.178:0/747801383 conn(0x7f26740f09a0 msgr2=0x7f26740f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:02:08.744 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:08.742+0000 7f267a542700 1 -- 172.21.15.178:0/747801383 shutdown_connections 2024-02-11T17:02:08.744 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:08.742+0000 7f267a542700 1 -- 172.21.15.178:0/747801383 wait complete. 2024-02-11T17:02:08.744 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:02:09.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:02:09 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:02:08.741630+0000 mon.smithi037 (mon.0) 145 : audit [DBG] from='client.? 172.21.15.178:0/747801383' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:02:10.445 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:02:10.446 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:02:10.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:02:10 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:02:09.146414+0000 mgr.smithi037.eofnaq (mgr.14162) 64 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:02:12.367 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:12.364+0000 7f4f49eeb700 1 -- 172.21.15.178:0/3947434570 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f440f5410 msgr2=0x7f4f440f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:02:12.367 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:12.364+0000 7f4f49eeb700 1 --2- 172.21.15.178:0/3947434570 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f440f5410 0x7f4f440f5830 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f4f34009fb0 tx=0x7f4f34009b30).stop 2024-02-11T17:02:12.367 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:12.364+0000 7f4f49eeb700 1 -- 172.21.15.178:0/3947434570 shutdown_connections 2024-02-11T17:02:12.368 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:12.364+0000 7f4f49eeb700 1 --2- 172.21.15.178:0/3947434570 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f440f5410 0x7f4f440f5830 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:02:12.368 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:12.364+0000 7f4f49eeb700 1 -- 172.21.15.178:0/3947434570 >> 172.21.15.178:0/3947434570 conn(0x7f4f440f0b70 msgr2=0x7f4f440f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:02:12.368 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:12.364+0000 7f4f49eeb700 1 -- 172.21.15.178:0/3947434570 shutdown_connections 2024-02-11T17:02:12.368 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:12.365+0000 7f4f49eeb700 1 -- 172.21.15.178:0/3947434570 wait complete. 2024-02-11T17:02:12.368 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:12.365+0000 7f4f49eeb700 1 Processor -- start 2024-02-11T17:02:12.368 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:12.365+0000 7f4f49eeb700 1 -- start start 2024-02-11T17:02:12.368 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:12.365+0000 7f4f49eeb700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f440f5410 0x7f4f44109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:02:12.368 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:12.365+0000 7f4f49eeb700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f44060240 con 0x7f4f440f5410 2024-02-11T17:02:12.368 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:12.366+0000 7f4f437fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f440f5410 0x7f4f44109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:02:12.368 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:12.366+0000 7f4f437fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f440f5410 0x7f4f44109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:42940/0 (socket says 172.21.15.178:42940) 2024-02-11T17:02:12.368 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:12.366+0000 7f4f437fe700 1 -- 172.21.15.178:0/1759702206 learned_addr learned my addr 172.21.15.178:0/1759702206 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:02:12.370 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:12.366+0000 7f4f437fe700 1 -- 172.21.15.178:0/1759702206 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f34005040 con 0x7f4f440f5410 2024-02-11T17:02:12.370 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:12.366+0000 7f4f437fe700 1 --2- 172.21.15.178:0/1759702206 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f440f5410 0x7f4f44109430 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f4f34013510 tx=0x7f4f3400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:02:12.370 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:12.366+0000 7f4f2bfff700 1 -- 172.21.15.178:0/1759702206 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4f34009320 con 0x7f4f440f5410 2024-02-11T17:02:12.370 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:12.366+0000 7f4f2bfff700 1 -- 172.21.15.178:0/1759702206 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4f34013f00 con 0x7f4f440f5410 2024-02-11T17:02:12.370 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:12.366+0000 7f4f49eeb700 1 -- 172.21.15.178:0/1759702206 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f44060440 con 0x7f4f440f5410 2024-02-11T17:02:12.370 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:12.366+0000 7f4f2bfff700 1 -- 172.21.15.178:0/1759702206 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4f34014d90 con 0x7f4f440f5410 2024-02-11T17:02:12.370 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:12.366+0000 7f4f49eeb700 1 -- 172.21.15.178:0/1759702206 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f44109ae0 con 0x7f4f440f5410 2024-02-11T17:02:12.370 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:12.367+0000 7f4f2bfff700 1 -- 172.21.15.178:0/1759702206 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4f340095f0 con 0x7f4f440f5410 2024-02-11T17:02:12.370 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:12.367+0000 7f4f49eeb700 1 -- 172.21.15.178:0/1759702206 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f44045ad0 con 0x7f4f440f5410 2024-02-11T17:02:12.370 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:12.367+0000 7f4f2bfff700 1 --2- 172.21.15.178:0/1759702206 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f4f2c02ff60 0x7f4f2c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:02:12.370 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:12.367+0000 7f4f2bfff700 1 -- 172.21.15.178:0/1759702206 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4f340410c0 con 0x7f4f440f5410 2024-02-11T17:02:12.370 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:12.368+0000 7f4f42ffd700 1 --2- 172.21.15.178:0/1759702206 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f4f2c02ff60 0x7f4f2c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:02:12.371 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:12.368+0000 7f4f42ffd700 1 --2- 172.21.15.178:0/1759702206 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f4f2c02ff60 0x7f4f2c032420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f4f380096a0 tx=0x7f4f38006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:02:12.374 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:12.373+0000 7f4f2bfff700 1 -- 172.21.15.178:0/1759702206 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4f34014420 con 0x7f4f440f5410 2024-02-11T17:02:12.579 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:12.578+0000 7f4f49eeb700 1 -- 172.21.15.178:0/1759702206 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4f440fae60 con 0x7f4f440f5410 2024-02-11T17:02:12.580 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:12.580+0000 7f4f2bfff700 1 -- 172.21.15.178:0/1759702206 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4f3401d7f0 con 0x7f4f440f5410 2024-02-11T17:02:12.581 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:02:12.581 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:02:12.582 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:12.580+0000 7f4f49eeb700 1 -- 172.21.15.178:0/1759702206 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f4f2c02ff60 msgr2=0x7f4f2c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:02:12.582 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:12.580+0000 7f4f49eeb700 1 --2- 172.21.15.178:0/1759702206 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f4f2c02ff60 0x7f4f2c032420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f4f380096a0 tx=0x7f4f38006b40).stop 2024-02-11T17:02:12.582 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:12.580+0000 7f4f49eeb700 1 -- 172.21.15.178:0/1759702206 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f440f5410 msgr2=0x7f4f44109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:02:12.582 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:12.580+0000 7f4f49eeb700 1 --2- 172.21.15.178:0/1759702206 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f440f5410 0x7f4f44109430 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f4f34013510 tx=0x7f4f3400b040).stop 2024-02-11T17:02:12.582 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:12.580+0000 7f4f49eeb700 1 -- 172.21.15.178:0/1759702206 shutdown_connections 2024-02-11T17:02:12.582 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:12.580+0000 7f4f49eeb700 1 --2- 172.21.15.178:0/1759702206 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f4f2c02ff60 0x7f4f2c032420 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:02:12.583 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:12.580+0000 7f4f49eeb700 1 --2- 172.21.15.178:0/1759702206 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f440f5410 0x7f4f44109430 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:02:12.583 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:12.580+0000 7f4f49eeb700 1 -- 172.21.15.178:0/1759702206 >> 172.21.15.178:0/1759702206 conn(0x7f4f440f0b70 msgr2=0x7f4f440f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:02:12.583 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:12.580+0000 7f4f49eeb700 1 -- 172.21.15.178:0/1759702206 shutdown_connections 2024-02-11T17:02:12.583 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:12.580+0000 7f4f49eeb700 1 -- 172.21.15.178:0/1759702206 wait complete. 2024-02-11T17:02:12.583 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:02:12.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:02:12 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:02:11.146760+0000 mgr.smithi037.eofnaq (mgr.14162) 65 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:02:13.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:02:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:02:12.579914+0000 mon.smithi037 (mon.0) 146 : audit [DBG] from='client.? 172.21.15.178:0/1759702206' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:02:14.323 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:02:14.324 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:02:14.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:02:14 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:02:13.147124+0000 mgr.smithi037.eofnaq (mgr.14162) 66 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:02:16.181 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:16.162+0000 7fc2a786d700 1 -- 172.21.15.178:0/1559442635 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc2a00f5410 msgr2=0x7fc2a00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:02:16.181 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:16.162+0000 7fc2a786d700 1 --2- 172.21.15.178:0/1559442635 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc2a00f5410 0x7fc2a00f5830 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7fc290009fb0 tx=0x7fc290009b30).stop 2024-02-11T17:02:16.181 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:16.162+0000 7fc2a786d700 1 -- 172.21.15.178:0/1559442635 shutdown_connections 2024-02-11T17:02:16.181 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:16.162+0000 7fc2a786d700 1 --2- 172.21.15.178:0/1559442635 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc2a00f5410 0x7fc2a00f5830 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:02:16.181 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:16.162+0000 7fc2a786d700 1 -- 172.21.15.178:0/1559442635 >> 172.21.15.178:0/1559442635 conn(0x7fc2a00f0b70 msgr2=0x7fc2a00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:02:16.181 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:16.162+0000 7fc2a786d700 1 -- 172.21.15.178:0/1559442635 shutdown_connections 2024-02-11T17:02:16.181 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:16.162+0000 7fc2a786d700 1 -- 172.21.15.178:0/1559442635 wait complete. 2024-02-11T17:02:16.181 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:16.163+0000 7fc2a786d700 1 Processor -- start 2024-02-11T17:02:16.181 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:16.163+0000 7fc2a786d700 1 -- start start 2024-02-11T17:02:16.181 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:16.163+0000 7fc2a786d700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc2a00f5410 0x7fc2a0105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:02:16.182 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:16.163+0000 7fc2a786d700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a0105b20 con 0x7fc2a00f5410 2024-02-11T17:02:16.182 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:16.163+0000 7fc2a5609700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc2a00f5410 0x7fc2a0105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:02:16.182 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:16.163+0000 7fc2a5609700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc2a00f5410 0x7fc2a0105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:42944/0 (socket says 172.21.15.178:42944) 2024-02-11T17:02:16.182 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:16.163+0000 7fc2a5609700 1 -- 172.21.15.178:0/183335174 learned_addr learned my addr 172.21.15.178:0/183335174 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:02:16.182 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:16.164+0000 7fc2a5609700 1 -- 172.21.15.178:0/183335174 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc290005040 con 0x7fc2a00f5410 2024-02-11T17:02:16.182 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:16.164+0000 7fc2a5609700 1 --2- 172.21.15.178:0/183335174 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc2a00f5410 0x7fc2a0105630 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fc290013510 tx=0x7fc29000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:02:16.182 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:16.164+0000 7fc295ffb700 1 -- 172.21.15.178:0/183335174 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc290009320 con 0x7fc2a00f5410 2024-02-11T17:02:16.182 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:16.164+0000 7fc295ffb700 1 -- 172.21.15.178:0/183335174 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc290013f00 con 0x7fc2a00f5410 2024-02-11T17:02:16.182 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:16.164+0000 7fc2a786d700 1 -- 172.21.15.178:0/183335174 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2a0105d20 con 0x7fc2a00f5410 2024-02-11T17:02:16.182 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:16.164+0000 7fc295ffb700 1 -- 172.21.15.178:0/183335174 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc290014d90 con 0x7fc2a00f5410 2024-02-11T17:02:16.182 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:16.164+0000 7fc2a786d700 1 -- 172.21.15.178:0/183335174 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2a01061b0 con 0x7fc2a00f5410 2024-02-11T17:02:16.182 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:16.165+0000 7fc295ffb700 1 -- 172.21.15.178:0/183335174 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc290014420 con 0x7fc2a00f5410 2024-02-11T17:02:16.182 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:16.165+0000 7fc295ffb700 1 --2- 172.21.15.178:0/183335174 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fc28c02ff60 0x7fc28c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:02:16.182 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:16.165+0000 7fc2a786d700 1 -- 172.21.15.178:0/183335174 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2a0045ad0 con 0x7fc2a00f5410 2024-02-11T17:02:16.183 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:16.165+0000 7fc295ffb700 1 -- 172.21.15.178:0/183335174 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc2900410c0 con 0x7fc2a00f5410 2024-02-11T17:02:16.183 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:16.165+0000 7fc2a4e08700 1 --2- 172.21.15.178:0/183335174 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fc28c02ff60 0x7fc28c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:02:16.183 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:16.166+0000 7fc2a4e08700 1 --2- 172.21.15.178:0/183335174 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fc28c02ff60 0x7fc28c032420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fc29c0096a0 tx=0x7fc29c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:02:16.183 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:16.170+0000 7fc295ffb700 1 -- 172.21.15.178:0/183335174 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc2900095f0 con 0x7fc2a00f5410 2024-02-11T17:02:16.376 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:16.375+0000 7fc2a786d700 1 -- 172.21.15.178:0/183335174 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc2a00faf50 con 0x7fc2a00f5410 2024-02-11T17:02:16.377 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:16.376+0000 7fc295ffb700 1 -- 172.21.15.178:0/183335174 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc29002c090 con 0x7fc2a00f5410 2024-02-11T17:02:16.377 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:02:16.377 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:02:16.379 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:16.378+0000 7fc2a786d700 1 -- 172.21.15.178:0/183335174 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fc28c02ff60 msgr2=0x7fc28c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:02:16.379 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:16.378+0000 7fc2a786d700 1 --2- 172.21.15.178:0/183335174 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fc28c02ff60 0x7fc28c032420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fc29c0096a0 tx=0x7fc29c006b40).stop 2024-02-11T17:02:16.379 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:16.378+0000 7fc2a786d700 1 -- 172.21.15.178:0/183335174 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc2a00f5410 msgr2=0x7fc2a0105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:02:16.379 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:16.378+0000 7fc2a786d700 1 --2- 172.21.15.178:0/183335174 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc2a00f5410 0x7fc2a0105630 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fc290013510 tx=0x7fc29000b040).stop 2024-02-11T17:02:16.379 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:16.378+0000 7fc2a786d700 1 -- 172.21.15.178:0/183335174 shutdown_connections 2024-02-11T17:02:16.379 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:16.378+0000 7fc2a786d700 1 --2- 172.21.15.178:0/183335174 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fc28c02ff60 0x7fc28c032420 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:02:16.379 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:16.378+0000 7fc2a786d700 1 --2- 172.21.15.178:0/183335174 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc2a00f5410 0x7fc2a0105630 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:02:16.380 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:16.378+0000 7fc2a786d700 1 -- 172.21.15.178:0/183335174 >> 172.21.15.178:0/183335174 conn(0x7fc2a00f0b70 msgr2=0x7fc2a00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:02:16.380 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:16.378+0000 7fc2a786d700 1 -- 172.21.15.178:0/183335174 shutdown_connections 2024-02-11T17:02:16.380 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:16.378+0000 7fc2a786d700 1 -- 172.21.15.178:0/183335174 wait complete. 2024-02-11T17:02:16.380 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:02:16.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:02:16 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:02:15.147513+0000 mgr.smithi037.eofnaq (mgr.14162) 67 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:02:17.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:02:17 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:02:16.376508+0000 mon.smithi037 (mon.0) 147 : audit [DBG] from='client.? 172.21.15.178:0/183335174' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:02:17.946 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:02:17.947 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:02:18.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:02:18 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:02:17.147861+0000 mgr.smithi037.eofnaq (mgr.14162) 68 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:02:19.784 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:19.781+0000 7f33efa5b700 1 -- 172.21.15.178:0/2121899398 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f33e80f6480 msgr2=0x7f33e80f68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:02:19.784 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:19.781+0000 7f33efa5b700 1 --2- 172.21.15.178:0/2121899398 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f33e80f6480 0x7f33e80f68a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f33dc009fb0 tx=0x7f33dc009b30).stop 2024-02-11T17:02:19.784 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:19.781+0000 7f33efa5b700 1 -- 172.21.15.178:0/2121899398 shutdown_connections 2024-02-11T17:02:19.785 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:19.781+0000 7f33efa5b700 1 --2- 172.21.15.178:0/2121899398 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f33e80f6480 0x7f33e80f68a0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:02:19.785 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:19.781+0000 7f33efa5b700 1 -- 172.21.15.178:0/2121899398 >> 172.21.15.178:0/2121899398 conn(0x7f33e80f1c00 msgr2=0x7f33e80f4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:02:19.785 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:19.782+0000 7f33efa5b700 1 -- 172.21.15.178:0/2121899398 shutdown_connections 2024-02-11T17:02:19.785 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:19.782+0000 7f33efa5b700 1 -- 172.21.15.178:0/2121899398 wait complete. 2024-02-11T17:02:19.785 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:19.782+0000 7f33efa5b700 1 Processor -- start 2024-02-11T17:02:19.785 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:19.782+0000 7f33efa5b700 1 -- start start 2024-02-11T17:02:19.785 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:19.782+0000 7f33efa5b700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f33e80f6480 0x7f33e806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:02:19.785 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:19.782+0000 7f33efa5b700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33e8069e20 con 0x7f33e80f6480 2024-02-11T17:02:19.785 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:19.783+0000 7f33ed7f7700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f33e80f6480 0x7f33e806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:02:19.785 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:19.783+0000 7f33ed7f7700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f33e80f6480 0x7f33e806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:56672/0 (socket says 172.21.15.178:56672) 2024-02-11T17:02:19.785 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:19.783+0000 7f33ed7f7700 1 -- 172.21.15.178:0/1626944599 learned_addr learned my addr 172.21.15.178:0/1626944599 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:02:19.785 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:19.783+0000 7f33ed7f7700 1 -- 172.21.15.178:0/1626944599 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33dc005040 con 0x7f33e80f6480 2024-02-11T17:02:19.786 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:19.783+0000 7f33ed7f7700 1 --2- 172.21.15.178:0/1626944599 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f33e80f6480 0x7f33e806b9a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f33dc013510 tx=0x7f33dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:02:19.786 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:19.783+0000 7f33d9ffb700 1 -- 172.21.15.178:0/1626944599 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33dc009320 con 0x7f33e80f6480 2024-02-11T17:02:19.786 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:19.783+0000 7f33efa5b700 1 -- 172.21.15.178:0/1626944599 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33e8069fc0 con 0x7f33e80f6480 2024-02-11T17:02:19.786 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:19.783+0000 7f33d9ffb700 1 -- 172.21.15.178:0/1626944599 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f33dc013f00 con 0x7f33e80f6480 2024-02-11T17:02:19.786 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:19.783+0000 7f33d9ffb700 1 -- 172.21.15.178:0/1626944599 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33dc014d90 con 0x7f33e80f6480 2024-02-11T17:02:19.786 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:19.783+0000 7f33efa5b700 1 -- 172.21.15.178:0/1626944599 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33e806a450 con 0x7f33e80f6480 2024-02-11T17:02:19.787 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:19.784+0000 7f33d9ffb700 1 -- 172.21.15.178:0/1626944599 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f33dc0095f0 con 0x7f33e80f6480 2024-02-11T17:02:19.787 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:19.784+0000 7f33efa5b700 1 -- 172.21.15.178:0/1626944599 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33e8059100 con 0x7f33e80f6480 2024-02-11T17:02:19.787 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:19.784+0000 7f33d9ffb700 1 --2- 172.21.15.178:0/1626944599 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f33d402ffb0 0x7f33d4032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:02:19.787 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:19.784+0000 7f33d9ffb700 1 -- 172.21.15.178:0/1626944599 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f33dc041080 con 0x7f33e80f6480 2024-02-11T17:02:19.787 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:19.785+0000 7f33ecff6700 1 --2- 172.21.15.178:0/1626944599 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f33d402ffb0 0x7f33d4032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:02:19.787 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:19.785+0000 7f33ecff6700 1 --2- 172.21.15.178:0/1626944599 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f33d402ffb0 0x7f33d4032470 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f33e40096a0 tx=0x7f33e4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:02:19.789 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:19.789+0000 7f33d9ffb700 1 -- 172.21.15.178:0/1626944599 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f33dc06d2b0 con 0x7f33e80f6480 2024-02-11T17:02:19.992 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:19.991+0000 7f33efa5b700 1 -- 172.21.15.178:0/1626944599 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f33e8045ad0 con 0x7f33e80f6480 2024-02-11T17:02:19.993 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:19.992+0000 7f33d9ffb700 1 -- 172.21.15.178:0/1626944599 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f33dc014420 con 0x7f33e80f6480 2024-02-11T17:02:19.993 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:02:19.993 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:02:19.994 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:19.993+0000 7f33efa5b700 1 -- 172.21.15.178:0/1626944599 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f33d402ffb0 msgr2=0x7f33d4032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:02:19.995 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:19.993+0000 7f33efa5b700 1 --2- 172.21.15.178:0/1626944599 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f33d402ffb0 0x7f33d4032470 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f33e40096a0 tx=0x7f33e4006b40).stop 2024-02-11T17:02:19.995 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:19.993+0000 7f33efa5b700 1 -- 172.21.15.178:0/1626944599 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f33e80f6480 msgr2=0x7f33e806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:02:19.995 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:19.993+0000 7f33efa5b700 1 --2- 172.21.15.178:0/1626944599 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f33e80f6480 0x7f33e806b9a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f33dc013510 tx=0x7f33dc00b040).stop 2024-02-11T17:02:19.995 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:19.993+0000 7f33efa5b700 1 -- 172.21.15.178:0/1626944599 shutdown_connections 2024-02-11T17:02:19.995 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:19.993+0000 7f33efa5b700 1 --2- 172.21.15.178:0/1626944599 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f33d402ffb0 0x7f33d4032470 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:02:19.995 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:19.993+0000 7f33efa5b700 1 --2- 172.21.15.178:0/1626944599 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f33e80f6480 0x7f33e806b9a0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:02:19.995 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:19.993+0000 7f33efa5b700 1 -- 172.21.15.178:0/1626944599 >> 172.21.15.178:0/1626944599 conn(0x7f33e80f1c00 msgr2=0x7f33e80f2720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:02:19.995 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:19.994+0000 7f33efa5b700 1 -- 172.21.15.178:0/1626944599 shutdown_connections 2024-02-11T17:02:19.995 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:19.994+0000 7f33efa5b700 1 -- 172.21.15.178:0/1626944599 wait complete. 2024-02-11T17:02:19.995 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:02:20.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:02:20 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:02:19.148188+0000 mgr.smithi037.eofnaq (mgr.14162) 69 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:02:20.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:02:20 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:02:19.992229+0000 mon.smithi037 (mon.0) 148 : audit [DBG] from='client.? 172.21.15.178:0/1626944599' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:02:21.632 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:02:21.633 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:02:22.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:02:22 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:02:21.148501+0000 mgr.smithi037.eofnaq (mgr.14162) 70 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:02:23.517 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:23.514+0000 7f4626990700 1 -- 172.21.15.178:0/1738256904 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f46200601e0 msgr2=0x7f46200f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:02:23.517 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:23.514+0000 7f4626990700 1 --2- 172.21.15.178:0/1738256904 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f46200601e0 0x7f46200f8f20 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f461c009fb0 tx=0x7f461c009b30).stop 2024-02-11T17:02:23.518 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:23.514+0000 7f4626990700 1 -- 172.21.15.178:0/1738256904 shutdown_connections 2024-02-11T17:02:23.518 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:23.514+0000 7f4626990700 1 --2- 172.21.15.178:0/1738256904 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f46200601e0 0x7f46200f8f20 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:02:23.518 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:23.514+0000 7f4626990700 1 -- 172.21.15.178:0/1738256904 >> 172.21.15.178:0/1738256904 conn(0x7f46200f09a0 msgr2=0x7f46200f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:02:23.518 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:23.514+0000 7f4626990700 1 -- 172.21.15.178:0/1738256904 shutdown_connections 2024-02-11T17:02:23.518 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:23.514+0000 7f4626990700 1 -- 172.21.15.178:0/1738256904 wait complete. 2024-02-11T17:02:23.518 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:23.515+0000 7f4626990700 1 Processor -- start 2024-02-11T17:02:23.518 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:23.515+0000 7f4626990700 1 -- start start 2024-02-11T17:02:23.518 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:23.515+0000 7f4626990700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f46200601e0 0x7f4620106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:02:23.518 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:23.515+0000 7f4626990700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46201069f0 con 0x7f46200601e0 2024-02-11T17:02:23.518 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:23.515+0000 7f462598e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f46200601e0 0x7f4620106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:02:23.519 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:23.515+0000 7f462598e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f46200601e0 0x7f4620106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:56690/0 (socket says 172.21.15.178:56690) 2024-02-11T17:02:23.519 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:23.515+0000 7f462598e700 1 -- 172.21.15.178:0/3635694975 learned_addr learned my addr 172.21.15.178:0/3635694975 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:02:23.519 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:23.515+0000 7f462598e700 1 -- 172.21.15.178:0/3635694975 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f461c005040 con 0x7f46200601e0 2024-02-11T17:02:23.519 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:23.515+0000 7f462598e700 1 --2- 172.21.15.178:0/3635694975 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f46200601e0 0x7f4620106500 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f461c014000 tx=0x7f461c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:02:23.519 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:23.516+0000 7f46167fc700 1 -- 172.21.15.178:0/3635694975 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f461c0095f0 con 0x7f46200601e0 2024-02-11T17:02:23.519 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:23.516+0000 7f46167fc700 1 -- 172.21.15.178:0/3635694975 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f461c007e70 con 0x7f46200601e0 2024-02-11T17:02:23.519 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:23.516+0000 7f4626990700 1 -- 172.21.15.178:0/3635694975 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4620106bf0 con 0x7f46200601e0 2024-02-11T17:02:23.519 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:23.516+0000 7f46167fc700 1 -- 172.21.15.178:0/3635694975 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f461c014ea0 con 0x7f46200601e0 2024-02-11T17:02:23.519 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:23.516+0000 7f4626990700 1 -- 172.21.15.178:0/3635694975 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4620107080 con 0x7f46200601e0 2024-02-11T17:02:23.520 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:23.516+0000 7f46167fc700 1 -- 172.21.15.178:0/3635694975 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f461c025070 con 0x7f46200601e0 2024-02-11T17:02:23.520 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:23.517+0000 7f4626990700 1 -- 172.21.15.178:0/3635694975 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4620045ad0 con 0x7f46200601e0 2024-02-11T17:02:23.520 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:23.517+0000 7f46167fc700 1 --2- 172.21.15.178:0/3635694975 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f460c02ff60 0x7f460c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:02:23.520 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:23.517+0000 7f46167fc700 1 -- 172.21.15.178:0/3635694975 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f461c040a30 con 0x7f46200601e0 2024-02-11T17:02:23.521 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:23.517+0000 7f462518d700 1 --2- 172.21.15.178:0/3635694975 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f460c02ff60 0x7f460c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:02:23.521 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:23.517+0000 7f462518d700 1 --2- 172.21.15.178:0/3635694975 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f460c02ff60 0x7f460c032420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f46100096a0 tx=0x7f4610006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:02:23.523 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:23.521+0000 7f46167fc700 1 -- 172.21.15.178:0/3635694975 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f461c0079c0 con 0x7f46200601e0 2024-02-11T17:02:23.727 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:23.726+0000 7f4626990700 1 -- 172.21.15.178:0/3635694975 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4620107f30 con 0x7f46200601e0 2024-02-11T17:02:23.728 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:23.726+0000 7f46167fc700 1 -- 172.21.15.178:0/3635694975 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f461c00f300 con 0x7f46200601e0 2024-02-11T17:02:23.728 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:02:23.728 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:02:23.730 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:23.728+0000 7f4626990700 1 -- 172.21.15.178:0/3635694975 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f460c02ff60 msgr2=0x7f460c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:02:23.730 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:23.728+0000 7f4626990700 1 --2- 172.21.15.178:0/3635694975 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f460c02ff60 0x7f460c032420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f46100096a0 tx=0x7f4610006b40).stop 2024-02-11T17:02:23.730 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:23.728+0000 7f4626990700 1 -- 172.21.15.178:0/3635694975 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f46200601e0 msgr2=0x7f4620106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:02:23.730 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:23.728+0000 7f4626990700 1 --2- 172.21.15.178:0/3635694975 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f46200601e0 0x7f4620106500 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f461c014000 tx=0x7f461c00b040).stop 2024-02-11T17:02:23.730 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:23.728+0000 7f4626990700 1 -- 172.21.15.178:0/3635694975 shutdown_connections 2024-02-11T17:02:23.730 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:23.728+0000 7f4626990700 1 --2- 172.21.15.178:0/3635694975 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f460c02ff60 0x7f460c032420 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:02:23.730 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:23.728+0000 7f4626990700 1 --2- 172.21.15.178:0/3635694975 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f46200601e0 0x7f4620106500 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:02:23.730 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:23.728+0000 7f4626990700 1 -- 172.21.15.178:0/3635694975 >> 172.21.15.178:0/3635694975 conn(0x7f46200f09a0 msgr2=0x7f46200f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:02:23.730 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:23.728+0000 7f4626990700 1 -- 172.21.15.178:0/3635694975 shutdown_connections 2024-02-11T17:02:23.730 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:23.728+0000 7f4626990700 1 -- 172.21.15.178:0/3635694975 wait complete. 2024-02-11T17:02:23.731 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:02:24.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:02:24 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:02:23.148787+0000 mgr.smithi037.eofnaq (mgr.14162) 71 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:02:24.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:02:24 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:02:23.727860+0000 mon.smithi037 (mon.0) 149 : audit [DBG] from='client.? 172.21.15.178:0/3635694975' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:02:25.469 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:02:25.469 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:02:26.435 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:02:26 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:02:25.149076+0000 mgr.smithi037.eofnaq (mgr.14162) 72 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:02:27.410 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:27.406+0000 7fc3e07ac700 1 -- 172.21.15.178:0/1851384477 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc3d80f6510 msgr2=0x7fc3d80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:02:27.410 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:27.406+0000 7fc3e07ac700 1 --2- 172.21.15.178:0/1851384477 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc3d80f6510 0x7fc3d80f6930 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fc3c8009fb0 tx=0x7fc3c8009b30).stop 2024-02-11T17:02:27.410 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:27.406+0000 7fc3e07ac700 1 -- 172.21.15.178:0/1851384477 shutdown_connections 2024-02-11T17:02:27.410 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:27.406+0000 7fc3e07ac700 1 --2- 172.21.15.178:0/1851384477 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc3d80f6510 0x7fc3d80f6930 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:02:27.410 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:27.406+0000 7fc3e07ac700 1 -- 172.21.15.178:0/1851384477 >> 172.21.15.178:0/1851384477 conn(0x7fc3d80f1c70 msgr2=0x7fc3d80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:02:27.410 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:27.406+0000 7fc3e07ac700 1 -- 172.21.15.178:0/1851384477 shutdown_connections 2024-02-11T17:02:27.410 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:27.406+0000 7fc3e07ac700 1 -- 172.21.15.178:0/1851384477 wait complete. 2024-02-11T17:02:27.410 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:27.407+0000 7fc3e07ac700 1 Processor -- start 2024-02-11T17:02:27.410 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:27.407+0000 7fc3e07ac700 1 -- start start 2024-02-11T17:02:27.410 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:27.407+0000 7fc3e07ac700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc3d80f6510 0x7fc3d8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:02:27.411 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:27.407+0000 7fc3e07ac700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d8106c20 con 0x7fc3d80f6510 2024-02-11T17:02:27.411 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:27.407+0000 7fc3de548700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc3d80f6510 0x7fc3d8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:02:27.411 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:27.407+0000 7fc3de548700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc3d80f6510 0x7fc3d8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:56716/0 (socket says 172.21.15.178:56716) 2024-02-11T17:02:27.411 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:27.407+0000 7fc3de548700 1 -- 172.21.15.178:0/1290126657 learned_addr learned my addr 172.21.15.178:0/1290126657 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:02:27.411 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:27.407+0000 7fc3de548700 1 -- 172.21.15.178:0/1290126657 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3c8005040 con 0x7fc3d80f6510 2024-02-11T17:02:27.411 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:27.408+0000 7fc3de548700 1 --2- 172.21.15.178:0/1290126657 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc3d80f6510 0x7fc3d8106730 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fc3c8013510 tx=0x7fc3c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:02:27.411 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:27.408+0000 7fc3ceffd700 1 -- 172.21.15.178:0/1290126657 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc3c8009320 con 0x7fc3d80f6510 2024-02-11T17:02:27.411 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:27.408+0000 7fc3e07ac700 1 -- 172.21.15.178:0/1290126657 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3d8106e20 con 0x7fc3d80f6510 2024-02-11T17:02:27.411 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:27.408+0000 7fc3ceffd700 1 -- 172.21.15.178:0/1290126657 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc3c8013f00 con 0x7fc3d80f6510 2024-02-11T17:02:27.411 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:27.408+0000 7fc3ceffd700 1 -- 172.21.15.178:0/1290126657 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc3c8014d90 con 0x7fc3d80f6510 2024-02-11T17:02:27.412 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:27.408+0000 7fc3e07ac700 1 -- 172.21.15.178:0/1290126657 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3d81072b0 con 0x7fc3d80f6510 2024-02-11T17:02:27.412 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:27.409+0000 7fc3ceffd700 1 -- 172.21.15.178:0/1290126657 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc3c80095f0 con 0x7fc3d80f6510 2024-02-11T17:02:27.412 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:27.409+0000 7fc3e07ac700 1 -- 172.21.15.178:0/1290126657 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3d8045ad0 con 0x7fc3d80f6510 2024-02-11T17:02:27.412 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:27.409+0000 7fc3ceffd700 1 --2- 172.21.15.178:0/1290126657 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fc3c402ff60 0x7fc3c4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:02:27.412 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:27.409+0000 7fc3ceffd700 1 -- 172.21.15.178:0/1290126657 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc3c80410c0 con 0x7fc3d80f6510 2024-02-11T17:02:27.413 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:27.409+0000 7fc3ddd47700 1 --2- 172.21.15.178:0/1290126657 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fc3c402ff60 0x7fc3c4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:02:27.413 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:27.409+0000 7fc3ddd47700 1 --2- 172.21.15.178:0/1290126657 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fc3c402ff60 0x7fc3c4032420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fc3d40096a0 tx=0x7fc3d4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:02:27.415 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:27.414+0000 7fc3ceffd700 1 -- 172.21.15.178:0/1290126657 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc3c8014420 con 0x7fc3d80f6510 2024-02-11T17:02:27.621 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:27.619+0000 7fc3e07ac700 1 -- 172.21.15.178:0/1290126657 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc3d80fbf60 con 0x7fc3d80f6510 2024-02-11T17:02:27.621 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:27.620+0000 7fc3ceffd700 1 -- 172.21.15.178:0/1290126657 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc3c8010230 con 0x7fc3d80f6510 2024-02-11T17:02:27.622 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:02:27.622 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:02:27.623 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:27.621+0000 7fc3e07ac700 1 -- 172.21.15.178:0/1290126657 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fc3c402ff60 msgr2=0x7fc3c4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:02:27.623 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:27.621+0000 7fc3e07ac700 1 --2- 172.21.15.178:0/1290126657 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fc3c402ff60 0x7fc3c4032420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fc3d40096a0 tx=0x7fc3d4006b40).stop 2024-02-11T17:02:27.623 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:27.621+0000 7fc3e07ac700 1 -- 172.21.15.178:0/1290126657 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc3d80f6510 msgr2=0x7fc3d8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:02:27.624 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:27.621+0000 7fc3e07ac700 1 --2- 172.21.15.178:0/1290126657 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc3d80f6510 0x7fc3d8106730 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fc3c8013510 tx=0x7fc3c800b040).stop 2024-02-11T17:02:27.624 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:27.621+0000 7fc3e07ac700 1 -- 172.21.15.178:0/1290126657 shutdown_connections 2024-02-11T17:02:27.624 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:27.621+0000 7fc3e07ac700 1 --2- 172.21.15.178:0/1290126657 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fc3c402ff60 0x7fc3c4032420 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:02:27.624 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:27.621+0000 7fc3e07ac700 1 --2- 172.21.15.178:0/1290126657 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc3d80f6510 0x7fc3d8106730 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:02:27.624 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:27.621+0000 7fc3e07ac700 1 -- 172.21.15.178:0/1290126657 >> 172.21.15.178:0/1290126657 conn(0x7fc3d80f1c70 msgr2=0x7fc3d80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:02:27.624 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:27.621+0000 7fc3e07ac700 1 -- 172.21.15.178:0/1290126657 shutdown_connections 2024-02-11T17:02:27.624 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:27.622+0000 7fc3e07ac700 1 -- 172.21.15.178:0/1290126657 wait complete. 2024-02-11T17:02:27.625 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:02:28.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:02:28 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:02:27.149385+0000 mgr.smithi037.eofnaq (mgr.14162) 73 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:02:28.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:02:28 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:02:27.620979+0000 mon.smithi037 (mon.0) 150 : audit [DBG] from='client.? 172.21.15.178:0/1290126657' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:02:29.375 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:02:29.376 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:02:30.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:02:30 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:02:29.149682+0000 mgr.smithi037.eofnaq (mgr.14162) 74 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:02:31.255 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:31.251+0000 7f84a72da700 1 -- 172.21.15.178:0/2846751508 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f84a0060140 msgr2=0x7f84a00f8fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:02:31.255 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:31.251+0000 7f84a72da700 1 --2- 172.21.15.178:0/2846751508 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f84a0060140 0x7f84a00f8fc0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f849c009fb0 tx=0x7f849c009b30).stop 2024-02-11T17:02:31.255 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:31.252+0000 7f84a72da700 1 -- 172.21.15.178:0/2846751508 shutdown_connections 2024-02-11T17:02:31.255 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:31.252+0000 7f84a72da700 1 --2- 172.21.15.178:0/2846751508 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f84a0060140 0x7f84a00f8fc0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:02:31.255 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:31.252+0000 7f84a72da700 1 -- 172.21.15.178:0/2846751508 >> 172.21.15.178:0/2846751508 conn(0x7f84a00f09a0 msgr2=0x7f84a00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:02:31.255 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:31.252+0000 7f84a72da700 1 -- 172.21.15.178:0/2846751508 shutdown_connections 2024-02-11T17:02:31.256 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:31.252+0000 7f84a72da700 1 -- 172.21.15.178:0/2846751508 wait complete. 2024-02-11T17:02:31.256 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:31.252+0000 7f84a72da700 1 Processor -- start 2024-02-11T17:02:31.256 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:31.252+0000 7f84a72da700 1 -- start start 2024-02-11T17:02:31.256 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:31.252+0000 7f84a72da700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f84a0060140 0x7f84a0106520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:02:31.256 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:31.252+0000 7f84a72da700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a0106a10 con 0x7f84a0060140 2024-02-11T17:02:31.256 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:31.253+0000 7f84a62d8700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f84a0060140 0x7f84a0106520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:02:31.256 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:31.253+0000 7f84a62d8700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f84a0060140 0x7f84a0106520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:44526/0 (socket says 172.21.15.178:44526) 2024-02-11T17:02:31.256 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:31.253+0000 7f84a62d8700 1 -- 172.21.15.178:0/1885286153 learned_addr learned my addr 172.21.15.178:0/1885286153 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:02:31.256 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:31.253+0000 7f84a62d8700 1 -- 172.21.15.178:0/1885286153 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f849c005040 con 0x7f84a0060140 2024-02-11T17:02:31.256 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:31.253+0000 7f84a62d8700 1 --2- 172.21.15.178:0/1885286153 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f84a0060140 0x7f84a0106520 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f849c013510 tx=0x7f849c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:02:31.257 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:31.253+0000 7f8496ffd700 1 -- 172.21.15.178:0/1885286153 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f849c009320 con 0x7f84a0060140 2024-02-11T17:02:31.257 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:31.253+0000 7f84a72da700 1 -- 172.21.15.178:0/1885286153 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84a0106c10 con 0x7f84a0060140 2024-02-11T17:02:31.257 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:31.253+0000 7f8496ffd700 1 -- 172.21.15.178:0/1885286153 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f849c013f00 con 0x7f84a0060140 2024-02-11T17:02:31.257 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:31.253+0000 7f8496ffd700 1 -- 172.21.15.178:0/1885286153 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f849c014d90 con 0x7f84a0060140 2024-02-11T17:02:31.257 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:31.253+0000 7f84a72da700 1 -- 172.21.15.178:0/1885286153 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84a01070a0 con 0x7f84a0060140 2024-02-11T17:02:31.258 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:31.254+0000 7f8496ffd700 1 -- 172.21.15.178:0/1885286153 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f849c0095f0 con 0x7f84a0060140 2024-02-11T17:02:31.258 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:31.254+0000 7f84a72da700 1 -- 172.21.15.178:0/1885286153 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84a0045ad0 con 0x7f84a0060140 2024-02-11T17:02:31.258 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:31.254+0000 7f8496ffd700 1 --2- 172.21.15.178:0/1885286153 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f848c02ff60 0x7f848c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:02:31.258 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:31.254+0000 7f8496ffd700 1 -- 172.21.15.178:0/1885286153 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f849c0410a0 con 0x7f84a0060140 2024-02-11T17:02:31.258 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:31.255+0000 7f84a5ad7700 1 --2- 172.21.15.178:0/1885286153 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f848c02ff60 0x7f848c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:02:31.258 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:31.255+0000 7f84a5ad7700 1 --2- 172.21.15.178:0/1885286153 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f848c02ff60 0x7f848c032420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f84900096a0 tx=0x7f8490006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:02:31.261 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:31.259+0000 7f8496ffd700 1 -- 172.21.15.178:0/1885286153 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f849c01d7f0 con 0x7f84a0060140 2024-02-11T17:02:31.465 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:31.464+0000 7f84a72da700 1 -- 172.21.15.178:0/1885286153 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f84a0107f50 con 0x7f84a0060140 2024-02-11T17:02:31.466 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:31.464+0000 7f8496ffd700 1 -- 172.21.15.178:0/1885286153 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f849c010230 con 0x7f84a0060140 2024-02-11T17:02:31.466 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:02:31.466 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:02:31.468 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:31.466+0000 7f84a72da700 1 -- 172.21.15.178:0/1885286153 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f848c02ff60 msgr2=0x7f848c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:02:31.468 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:31.466+0000 7f84a72da700 1 --2- 172.21.15.178:0/1885286153 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f848c02ff60 0x7f848c032420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f84900096a0 tx=0x7f8490006b40).stop 2024-02-11T17:02:31.468 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:31.466+0000 7f84a72da700 1 -- 172.21.15.178:0/1885286153 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f84a0060140 msgr2=0x7f84a0106520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:02:31.468 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:31.466+0000 7f84a72da700 1 --2- 172.21.15.178:0/1885286153 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f84a0060140 0x7f84a0106520 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f849c013510 tx=0x7f849c00b040).stop 2024-02-11T17:02:31.468 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:31.466+0000 7f84a72da700 1 -- 172.21.15.178:0/1885286153 shutdown_connections 2024-02-11T17:02:31.468 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:31.466+0000 7f84a72da700 1 --2- 172.21.15.178:0/1885286153 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f848c02ff60 0x7f848c032420 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:02:31.468 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:31.466+0000 7f84a72da700 1 --2- 172.21.15.178:0/1885286153 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f84a0060140 0x7f84a0106520 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:02:31.468 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:31.466+0000 7f84a72da700 1 -- 172.21.15.178:0/1885286153 >> 172.21.15.178:0/1885286153 conn(0x7f84a00f09a0 msgr2=0x7f84a00f14d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:02:31.468 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:31.466+0000 7f84a72da700 1 -- 172.21.15.178:0/1885286153 shutdown_connections 2024-02-11T17:02:31.469 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:31.466+0000 7f84a72da700 1 -- 172.21.15.178:0/1885286153 wait complete. 2024-02-11T17:02:31.469 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:02:32.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:02:32 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:02:31.149979+0000 mgr.smithi037.eofnaq (mgr.14162) 75 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:02:32.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:02:32 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:02:31.465528+0000 mon.smithi037 (mon.0) 151 : audit [DBG] from='client.? 172.21.15.178:0/1885286153' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:02:33.151 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:02:33.151 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:02:34.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:02:34 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:02:33.150303+0000 mgr.smithi037.eofnaq (mgr.14162) 76 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:02:35.059 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:35.055+0000 7f0230974700 1 -- 172.21.15.178:0/681334950 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f022c0f5410 msgr2=0x7f022c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:02:35.059 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:35.055+0000 7f0230974700 1 --2- 172.21.15.178:0/681334950 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f022c0f5410 0x7f022c0f5830 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f0220009fb0 tx=0x7f0220009b30).stop 2024-02-11T17:02:35.059 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:35.055+0000 7f0230974700 1 -- 172.21.15.178:0/681334950 shutdown_connections 2024-02-11T17:02:35.059 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:35.055+0000 7f0230974700 1 --2- 172.21.15.178:0/681334950 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f022c0f5410 0x7f022c0f5830 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:02:35.059 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:35.055+0000 7f0230974700 1 -- 172.21.15.178:0/681334950 >> 172.21.15.178:0/681334950 conn(0x7f022c0f0b70 msgr2=0x7f022c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:02:35.059 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:35.055+0000 7f0230974700 1 -- 172.21.15.178:0/681334950 shutdown_connections 2024-02-11T17:02:35.059 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:35.055+0000 7f0230974700 1 -- 172.21.15.178:0/681334950 wait complete. 2024-02-11T17:02:35.059 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:35.056+0000 7f0230974700 1 Processor -- start 2024-02-11T17:02:35.059 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:35.056+0000 7f0230974700 1 -- start start 2024-02-11T17:02:35.060 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:35.056+0000 7f0230974700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f022c0f5410 0x7f022c109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:02:35.060 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:35.056+0000 7f0230974700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f022c060220 con 0x7f022c0f5410 2024-02-11T17:02:35.060 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:35.056+0000 7f022a59c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f022c0f5410 0x7f022c109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:02:35.060 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:35.056+0000 7f022a59c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f022c0f5410 0x7f022c109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:44552/0 (socket says 172.21.15.178:44552) 2024-02-11T17:02:35.060 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:35.056+0000 7f022a59c700 1 -- 172.21.15.178:0/4009899634 learned_addr learned my addr 172.21.15.178:0/4009899634 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:02:35.060 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:35.057+0000 7f022a59c700 1 -- 172.21.15.178:0/4009899634 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0220005040 con 0x7f022c0f5410 2024-02-11T17:02:35.060 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:35.057+0000 7f022a59c700 1 --2- 172.21.15.178:0/4009899634 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f022c0f5410 0x7f022c109450 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f0220013510 tx=0x7f022000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:02:35.060 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:35.057+0000 7f021affd700 1 -- 172.21.15.178:0/4009899634 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0220009320 con 0x7f022c0f5410 2024-02-11T17:02:35.060 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:35.057+0000 7f0230974700 1 -- 172.21.15.178:0/4009899634 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f022c060420 con 0x7f022c0f5410 2024-02-11T17:02:35.060 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:35.057+0000 7f021affd700 1 -- 172.21.15.178:0/4009899634 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0220013f00 con 0x7f022c0f5410 2024-02-11T17:02:35.060 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:35.057+0000 7f021affd700 1 -- 172.21.15.178:0/4009899634 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0220014d90 con 0x7f022c0f5410 2024-02-11T17:02:35.061 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:35.057+0000 7f0230974700 1 -- 172.21.15.178:0/4009899634 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f022c109b00 con 0x7f022c0f5410 2024-02-11T17:02:35.061 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:35.058+0000 7f0230974700 1 -- 172.21.15.178:0/4009899634 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f022c059100 con 0x7f022c0f5410 2024-02-11T17:02:35.062 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:35.058+0000 7f021affd700 1 -- 172.21.15.178:0/4009899634 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f02200095f0 con 0x7f022c0f5410 2024-02-11T17:02:35.062 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:35.058+0000 7f021affd700 1 --2- 172.21.15.178:0/4009899634 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f021402ff60 0x7f0214032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:02:35.062 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:35.058+0000 7f021affd700 1 -- 172.21.15.178:0/4009899634 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0220041080 con 0x7f022c0f5410 2024-02-11T17:02:35.062 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:35.058+0000 7f0229d9b700 1 --2- 172.21.15.178:0/4009899634 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f021402ff60 0x7f0214032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:02:35.062 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:35.059+0000 7f0229d9b700 1 --2- 172.21.15.178:0/4009899634 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f021402ff60 0x7f0214032420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f021c0096a0 tx=0x7f021c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:02:35.064 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:35.063+0000 7f021affd700 1 -- 172.21.15.178:0/4009899634 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0220014420 con 0x7f022c0f5410 2024-02-11T17:02:35.268 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:35.266+0000 7f0230974700 1 -- 172.21.15.178:0/4009899634 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f022c045ad0 con 0x7f022c0f5410 2024-02-11T17:02:35.269 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:35.267+0000 7f021affd700 1 -- 172.21.15.178:0/4009899634 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0220010230 con 0x7f022c0f5410 2024-02-11T17:02:35.269 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:02:35.269 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:02:35.270 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:35.269+0000 7f0230974700 1 -- 172.21.15.178:0/4009899634 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f021402ff60 msgr2=0x7f0214032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:02:35.271 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:35.269+0000 7f0230974700 1 --2- 172.21.15.178:0/4009899634 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f021402ff60 0x7f0214032420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f021c0096a0 tx=0x7f021c006b40).stop 2024-02-11T17:02:35.271 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:35.269+0000 7f0230974700 1 -- 172.21.15.178:0/4009899634 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f022c0f5410 msgr2=0x7f022c109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:02:35.271 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:35.269+0000 7f0230974700 1 --2- 172.21.15.178:0/4009899634 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f022c0f5410 0x7f022c109450 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f0220013510 tx=0x7f022000b040).stop 2024-02-11T17:02:35.271 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:35.269+0000 7f0230974700 1 -- 172.21.15.178:0/4009899634 shutdown_connections 2024-02-11T17:02:35.271 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:35.269+0000 7f0230974700 1 --2- 172.21.15.178:0/4009899634 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f021402ff60 0x7f0214032420 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:02:35.271 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:35.269+0000 7f0230974700 1 --2- 172.21.15.178:0/4009899634 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f022c0f5410 0x7f022c109450 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:02:35.271 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:35.269+0000 7f0230974700 1 -- 172.21.15.178:0/4009899634 >> 172.21.15.178:0/4009899634 conn(0x7f022c0f0b70 msgr2=0x7f022c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:02:35.271 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:35.269+0000 7f0230974700 1 -- 172.21.15.178:0/4009899634 shutdown_connections 2024-02-11T17:02:35.271 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:35.269+0000 7f0230974700 1 -- 172.21.15.178:0/4009899634 wait complete. 2024-02-11T17:02:35.271 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:02:36.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:02:36 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:02:35.150628+0000 mgr.smithi037.eofnaq (mgr.14162) 77 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:02:36.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:02:36 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:02:35.268240+0000 mon.smithi037 (mon.0) 152 : audit [DBG] from='client.? 172.21.15.178:0/4009899634' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:02:36.939 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:02:36.940 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:02:38.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:02:38 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:02:37.150922+0000 mgr.smithi037.eofnaq (mgr.14162) 78 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:02:38.845 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:38.842+0000 7fdc4e39b700 1 -- 172.21.15.178:0/3133478489 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdc480f5410 msgr2=0x7fdc480f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:02:38.846 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:38.842+0000 7fdc4e39b700 1 --2- 172.21.15.178:0/3133478489 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdc480f5410 0x7fdc480f5830 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7fdc38009fb0 tx=0x7fdc38009b30).stop 2024-02-11T17:02:38.846 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:38.842+0000 7fdc4e39b700 1 -- 172.21.15.178:0/3133478489 shutdown_connections 2024-02-11T17:02:38.846 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:38.842+0000 7fdc4e39b700 1 --2- 172.21.15.178:0/3133478489 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdc480f5410 0x7fdc480f5830 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:02:38.846 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:38.842+0000 7fdc4e39b700 1 -- 172.21.15.178:0/3133478489 >> 172.21.15.178:0/3133478489 conn(0x7fdc480f0b70 msgr2=0x7fdc480f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:02:38.846 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:38.842+0000 7fdc4e39b700 1 -- 172.21.15.178:0/3133478489 shutdown_connections 2024-02-11T17:02:38.846 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:38.842+0000 7fdc4e39b700 1 -- 172.21.15.178:0/3133478489 wait complete. 2024-02-11T17:02:38.846 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:38.843+0000 7fdc4e39b700 1 Processor -- start 2024-02-11T17:02:38.846 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:38.843+0000 7fdc4e39b700 1 -- start start 2024-02-11T17:02:38.846 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:38.843+0000 7fdc4e39b700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdc480f5410 0x7fdc48109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:02:38.846 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:38.843+0000 7fdc4e39b700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc48060220 con 0x7fdc480f5410 2024-02-11T17:02:38.846 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:38.844+0000 7fdc47fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdc480f5410 0x7fdc48109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:02:38.847 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:38.844+0000 7fdc47fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdc480f5410 0x7fdc48109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:44570/0 (socket says 172.21.15.178:44570) 2024-02-11T17:02:38.847 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:38.844+0000 7fdc47fff700 1 -- 172.21.15.178:0/2110576198 learned_addr learned my addr 172.21.15.178:0/2110576198 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:02:38.847 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:38.844+0000 7fdc47fff700 1 -- 172.21.15.178:0/2110576198 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc38005040 con 0x7fdc480f5410 2024-02-11T17:02:38.847 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:38.844+0000 7fdc47fff700 1 --2- 172.21.15.178:0/2110576198 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdc480f5410 0x7fdc48109450 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fdc38013510 tx=0x7fdc3800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:02:38.848 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:38.844+0000 7fdc44ff9700 1 -- 172.21.15.178:0/2110576198 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc38009320 con 0x7fdc480f5410 2024-02-11T17:02:38.848 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:38.844+0000 7fdc4e39b700 1 -- 172.21.15.178:0/2110576198 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc48060420 con 0x7fdc480f5410 2024-02-11T17:02:38.848 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:38.844+0000 7fdc44ff9700 1 -- 172.21.15.178:0/2110576198 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdc38013f00 con 0x7fdc480f5410 2024-02-11T17:02:38.848 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:38.844+0000 7fdc44ff9700 1 -- 172.21.15.178:0/2110576198 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc38014d90 con 0x7fdc480f5410 2024-02-11T17:02:38.848 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:38.844+0000 7fdc4e39b700 1 -- 172.21.15.178:0/2110576198 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc48109b00 con 0x7fdc480f5410 2024-02-11T17:02:38.848 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:38.845+0000 7fdc44ff9700 1 -- 172.21.15.178:0/2110576198 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdc380263a0 con 0x7fdc480f5410 2024-02-11T17:02:38.848 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:38.845+0000 7fdc44ff9700 1 --2- 172.21.15.178:0/2110576198 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fdc3002ff60 0x7fdc30032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:02:38.848 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:38.845+0000 7fdc44ff9700 1 -- 172.21.15.178:0/2110576198 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdc380410b0 con 0x7fdc480f5410 2024-02-11T17:02:38.849 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:38.845+0000 7fdc4e39b700 1 -- 172.21.15.178:0/2110576198 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc48059100 con 0x7fdc480f5410 2024-02-11T17:02:38.849 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:38.846+0000 7fdc477fe700 1 --2- 172.21.15.178:0/2110576198 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fdc3002ff60 0x7fdc30032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:02:38.849 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:38.846+0000 7fdc477fe700 1 --2- 172.21.15.178:0/2110576198 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fdc3002ff60 0x7fdc30032420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fdc3c0096a0 tx=0x7fdc3c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:02:38.852 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:38.850+0000 7fdc44ff9700 1 -- 172.21.15.178:0/2110576198 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdc38012920 con 0x7fdc480f5410 2024-02-11T17:02:39.055 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:39.054+0000 7fdc4e39b700 1 -- 172.21.15.178:0/2110576198 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdc4810a1d0 con 0x7fdc480f5410 2024-02-11T17:02:39.056 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:39.055+0000 7fdc44ff9700 1 -- 172.21.15.178:0/2110576198 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdc3802b020 con 0x7fdc480f5410 2024-02-11T17:02:39.056 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:02:39.056 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:02:39.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:39.056+0000 7fdc4e39b700 1 -- 172.21.15.178:0/2110576198 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fdc3002ff60 msgr2=0x7fdc30032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:02:39.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:39.056+0000 7fdc4e39b700 1 --2- 172.21.15.178:0/2110576198 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fdc3002ff60 0x7fdc30032420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fdc3c0096a0 tx=0x7fdc3c006b40).stop 2024-02-11T17:02:39.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:39.056+0000 7fdc4e39b700 1 -- 172.21.15.178:0/2110576198 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdc480f5410 msgr2=0x7fdc48109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:02:39.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:39.056+0000 7fdc4e39b700 1 --2- 172.21.15.178:0/2110576198 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdc480f5410 0x7fdc48109450 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fdc38013510 tx=0x7fdc3800b040).stop 2024-02-11T17:02:39.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:39.056+0000 7fdc4e39b700 1 -- 172.21.15.178:0/2110576198 shutdown_connections 2024-02-11T17:02:39.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:39.056+0000 7fdc4e39b700 1 --2- 172.21.15.178:0/2110576198 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fdc3002ff60 0x7fdc30032420 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:02:39.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:39.056+0000 7fdc4e39b700 1 --2- 172.21.15.178:0/2110576198 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdc480f5410 0x7fdc48109450 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:02:39.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:39.056+0000 7fdc4e39b700 1 -- 172.21.15.178:0/2110576198 >> 172.21.15.178:0/2110576198 conn(0x7fdc480f0b70 msgr2=0x7fdc480f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:02:39.059 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:39.056+0000 7fdc4e39b700 1 -- 172.21.15.178:0/2110576198 shutdown_connections 2024-02-11T17:02:39.059 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:39.056+0000 7fdc4e39b700 1 -- 172.21.15.178:0/2110576198 wait complete. 2024-02-11T17:02:39.059 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:02:39.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:02:39 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:02:39.055952+0000 mon.smithi037 (mon.0) 153 : audit [DBG] from='client.? 172.21.15.178:0/2110576198' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:02:40.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:02:40 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:02:39.151194+0000 mgr.smithi037.eofnaq (mgr.14162) 79 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:02:40.741 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:02:40.742 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:02:42.642 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:42.639+0000 7f64becd7700 1 -- 172.21.15.178:0/2010890132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f64b80f5410 msgr2=0x7f64b80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:02:42.642 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:42.639+0000 7f64becd7700 1 --2- 172.21.15.178:0/2010890132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f64b80f5410 0x7f64b80f5830 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f64ac009fb0 tx=0x7f64ac009b30).stop 2024-02-11T17:02:42.642 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:42.639+0000 7f64becd7700 1 -- 172.21.15.178:0/2010890132 shutdown_connections 2024-02-11T17:02:42.642 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:42.639+0000 7f64becd7700 1 --2- 172.21.15.178:0/2010890132 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f64b80f5410 0x7f64b80f5830 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:02:42.642 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:42.639+0000 7f64becd7700 1 -- 172.21.15.178:0/2010890132 >> 172.21.15.178:0/2010890132 conn(0x7f64b80f0b70 msgr2=0x7f64b80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:02:42.642 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:42.639+0000 7f64becd7700 1 -- 172.21.15.178:0/2010890132 shutdown_connections 2024-02-11T17:02:42.642 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:42.639+0000 7f64becd7700 1 -- 172.21.15.178:0/2010890132 wait complete. 2024-02-11T17:02:42.642 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:42.639+0000 7f64becd7700 1 Processor -- start 2024-02-11T17:02:42.642 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:42.640+0000 7f64becd7700 1 -- start start 2024-02-11T17:02:42.642 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:42.640+0000 7f64becd7700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f64b80f5410 0x7f64b8106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:02:42.643 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:42.640+0000 7f64becd7700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64b8106bf0 con 0x7f64b80f5410 2024-02-11T17:02:42.643 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:42.640+0000 7f64bca73700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f64b80f5410 0x7f64b8106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:02:42.643 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:42.640+0000 7f64bca73700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f64b80f5410 0x7f64b8106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:56512/0 (socket says 172.21.15.178:56512) 2024-02-11T17:02:42.643 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:42.640+0000 7f64bca73700 1 -- 172.21.15.178:0/361698275 learned_addr learned my addr 172.21.15.178:0/361698275 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:02:42.643 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:42.640+0000 7f64bca73700 1 -- 172.21.15.178:0/361698275 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64ac005040 con 0x7f64b80f5410 2024-02-11T17:02:42.644 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:42.640+0000 7f64bca73700 1 --2- 172.21.15.178:0/361698275 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f64b80f5410 0x7f64b8106700 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f64ac013510 tx=0x7f64ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:02:42.644 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:42.641+0000 7f64b57fa700 1 -- 172.21.15.178:0/361698275 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64ac009320 con 0x7f64b80f5410 2024-02-11T17:02:42.644 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:42.641+0000 7f64b57fa700 1 -- 172.21.15.178:0/361698275 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f64ac013f00 con 0x7f64b80f5410 2024-02-11T17:02:42.644 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:42.641+0000 7f64becd7700 1 -- 172.21.15.178:0/361698275 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64b8106df0 con 0x7f64b80f5410 2024-02-11T17:02:42.644 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:42.641+0000 7f64b57fa700 1 -- 172.21.15.178:0/361698275 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64ac014d90 con 0x7f64b80f5410 2024-02-11T17:02:42.644 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:42.641+0000 7f64becd7700 1 -- 172.21.15.178:0/361698275 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64b8107280 con 0x7f64b80f5410 2024-02-11T17:02:42.644 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:42.642+0000 7f64b57fa700 1 -- 172.21.15.178:0/361698275 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f64ac0095f0 con 0x7f64b80f5410 2024-02-11T17:02:42.644 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:42.642+0000 7f64becd7700 1 -- 172.21.15.178:0/361698275 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64b8045ad0 con 0x7f64b80f5410 2024-02-11T17:02:42.644 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:42.642+0000 7f64b57fa700 1 --2- 172.21.15.178:0/361698275 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f64a802ff60 0x7f64a8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:02:42.644 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:42.642+0000 7f64b57fa700 1 -- 172.21.15.178:0/361698275 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f64ac041000 con 0x7f64b80f5410 2024-02-11T17:02:42.645 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:42.642+0000 7f64b7fff700 1 --2- 172.21.15.178:0/361698275 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f64a802ff60 0x7f64a8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:02:42.645 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:42.642+0000 7f64b7fff700 1 --2- 172.21.15.178:0/361698275 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f64a802ff60 0x7f64a8032420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f64a40096a0 tx=0x7f64a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:02:42.648 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:42.647+0000 7f64b57fa700 1 -- 172.21.15.178:0/361698275 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f64ac014420 con 0x7f64b80f5410 2024-02-11T17:02:42.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:02:42 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:02:41.151521+0000 mgr.smithi037.eofnaq (mgr.14162) 80 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:02:42.853 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:42.852+0000 7f64becd7700 1 -- 172.21.15.178:0/361698275 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f64b80fae60 con 0x7f64b80f5410 2024-02-11T17:02:42.854 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:42.852+0000 7f64b57fa700 1 -- 172.21.15.178:0/361698275 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f64ac01d7f0 con 0x7f64b80f5410 2024-02-11T17:02:42.854 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:02:42.854 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:02:42.856 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:42.854+0000 7f64becd7700 1 -- 172.21.15.178:0/361698275 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f64a802ff60 msgr2=0x7f64a8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:02:42.856 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:42.854+0000 7f64becd7700 1 --2- 172.21.15.178:0/361698275 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f64a802ff60 0x7f64a8032420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f64a40096a0 tx=0x7f64a4006b40).stop 2024-02-11T17:02:42.856 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:42.854+0000 7f64becd7700 1 -- 172.21.15.178:0/361698275 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f64b80f5410 msgr2=0x7f64b8106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:02:42.856 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:42.854+0000 7f64becd7700 1 --2- 172.21.15.178:0/361698275 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f64b80f5410 0x7f64b8106700 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f64ac013510 tx=0x7f64ac00b040).stop 2024-02-11T17:02:42.856 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:42.854+0000 7f64becd7700 1 -- 172.21.15.178:0/361698275 shutdown_connections 2024-02-11T17:02:42.856 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:42.854+0000 7f64becd7700 1 --2- 172.21.15.178:0/361698275 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f64a802ff60 0x7f64a8032420 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:02:42.856 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:42.854+0000 7f64becd7700 1 --2- 172.21.15.178:0/361698275 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f64b80f5410 0x7f64b8106700 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:02:42.856 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:42.854+0000 7f64becd7700 1 -- 172.21.15.178:0/361698275 >> 172.21.15.178:0/361698275 conn(0x7f64b80f0b70 msgr2=0x7f64b80f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:02:42.856 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:42.854+0000 7f64becd7700 1 -- 172.21.15.178:0/361698275 shutdown_connections 2024-02-11T17:02:42.856 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:42.854+0000 7f64becd7700 1 -- 172.21.15.178:0/361698275 wait complete. 2024-02-11T17:02:42.857 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:02:43.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:02:43 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:02:42.853809+0000 mon.smithi037 (mon.0) 154 : audit [DBG] from='client.? 172.21.15.178:0/361698275' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:02:44.558 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:02:44.559 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:02:44.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:02:44 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:02:43.151931+0000 mgr.smithi037.eofnaq (mgr.14162) 81 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:02:46.418 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:46.415+0000 7fbad2aa8700 1 -- 172.21.15.178:0/2486236883 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbacc0f5410 msgr2=0x7fbacc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:02:46.418 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:46.415+0000 7fbad2aa8700 1 --2- 172.21.15.178:0/2486236883 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbacc0f5410 0x7fbacc0f5830 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fbac0009fb0 tx=0x7fbac0009b30).stop 2024-02-11T17:02:46.418 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:46.415+0000 7fbad2aa8700 1 -- 172.21.15.178:0/2486236883 shutdown_connections 2024-02-11T17:02:46.418 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:46.416+0000 7fbad2aa8700 1 --2- 172.21.15.178:0/2486236883 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbacc0f5410 0x7fbacc0f5830 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:02:46.418 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:46.416+0000 7fbad2aa8700 1 -- 172.21.15.178:0/2486236883 >> 172.21.15.178:0/2486236883 conn(0x7fbacc0f0b70 msgr2=0x7fbacc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:02:46.418 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:46.416+0000 7fbad2aa8700 1 -- 172.21.15.178:0/2486236883 shutdown_connections 2024-02-11T17:02:46.418 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:46.416+0000 7fbad2aa8700 1 -- 172.21.15.178:0/2486236883 wait complete. 2024-02-11T17:02:46.418 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:46.416+0000 7fbad2aa8700 1 Processor -- start 2024-02-11T17:02:46.419 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:46.416+0000 7fbad2aa8700 1 -- start start 2024-02-11T17:02:46.419 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:46.416+0000 7fbad2aa8700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbacc0f5410 0x7fbacc106760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:02:46.419 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:46.416+0000 7fbad2aa8700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbacc106c50 con 0x7fbacc0f5410 2024-02-11T17:02:46.421 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:46.417+0000 7fbad0844700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbacc0f5410 0x7fbacc106760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:02:46.421 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:46.417+0000 7fbad0844700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbacc0f5410 0x7fbacc106760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:56522/0 (socket says 172.21.15.178:56522) 2024-02-11T17:02:46.421 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:46.417+0000 7fbad0844700 1 -- 172.21.15.178:0/3122260030 learned_addr learned my addr 172.21.15.178:0/3122260030 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:02:46.421 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:46.417+0000 7fbad0844700 1 -- 172.21.15.178:0/3122260030 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbac0005040 con 0x7fbacc0f5410 2024-02-11T17:02:46.421 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:46.417+0000 7fbad0844700 1 --2- 172.21.15.178:0/3122260030 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbacc0f5410 0x7fbacc106760 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fbac0009f80 tx=0x7fbac000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:02:46.421 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:46.417+0000 7fbac97fa700 1 -- 172.21.15.178:0/3122260030 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbac0009320 con 0x7fbacc0f5410 2024-02-11T17:02:46.421 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:46.417+0000 7fbac97fa700 1 -- 172.21.15.178:0/3122260030 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbac0013ec0 con 0x7fbacc0f5410 2024-02-11T17:02:46.421 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:46.417+0000 7fbad2aa8700 1 -- 172.21.15.178:0/3122260030 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbacc106e50 con 0x7fbacc0f5410 2024-02-11T17:02:46.421 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:46.417+0000 7fbac97fa700 1 -- 172.21.15.178:0/3122260030 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbac001ece0 con 0x7fbacc0f5410 2024-02-11T17:02:46.421 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:46.417+0000 7fbad2aa8700 1 -- 172.21.15.178:0/3122260030 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbacc1072e0 con 0x7fbacc0f5410 2024-02-11T17:02:46.421 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:46.418+0000 7fbac97fa700 1 -- 172.21.15.178:0/3122260030 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbac001e420 con 0x7fbacc0f5410 2024-02-11T17:02:46.422 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:46.418+0000 7fbad2aa8700 1 -- 172.21.15.178:0/3122260030 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbacc045ad0 con 0x7fbacc0f5410 2024-02-11T17:02:46.422 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:46.418+0000 7fbac97fa700 1 --2- 172.21.15.178:0/3122260030 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fbabc02ff60 0x7fbabc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:02:46.422 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:46.418+0000 7fbac97fa700 1 -- 172.21.15.178:0/3122260030 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbac0040f00 con 0x7fbacc0f5410 2024-02-11T17:02:46.422 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:46.418+0000 7fbacbfff700 1 --2- 172.21.15.178:0/3122260030 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fbabc02ff60 0x7fbabc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:02:46.422 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:46.419+0000 7fbacbfff700 1 --2- 172.21.15.178:0/3122260030 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fbabc02ff60 0x7fbabc032420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fbab80096a0 tx=0x7fbab8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:02:46.425 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:46.423+0000 7fbac97fa700 1 -- 172.21.15.178:0/3122260030 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbac001e730 con 0x7fbacc0f5410 2024-02-11T17:02:46.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:02:46 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:02:45.152225+0000 mgr.smithi037.eofnaq (mgr.14162) 82 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:02:46.629 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:46.627+0000 7fbad2aa8700 1 -- 172.21.15.178:0/3122260030 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbacc108190 con 0x7fbacc0f5410 2024-02-11T17:02:46.629 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:46.628+0000 7fbac97fa700 1 -- 172.21.15.178:0/3122260030 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbac0004490 con 0x7fbacc0f5410 2024-02-11T17:02:46.630 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:02:46.630 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:02:46.631 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:46.629+0000 7fbad2aa8700 1 -- 172.21.15.178:0/3122260030 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fbabc02ff60 msgr2=0x7fbabc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:02:46.631 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:46.629+0000 7fbad2aa8700 1 --2- 172.21.15.178:0/3122260030 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fbabc02ff60 0x7fbabc032420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fbab80096a0 tx=0x7fbab8006b40).stop 2024-02-11T17:02:46.631 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:46.629+0000 7fbad2aa8700 1 -- 172.21.15.178:0/3122260030 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbacc0f5410 msgr2=0x7fbacc106760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:02:46.631 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:46.629+0000 7fbad2aa8700 1 --2- 172.21.15.178:0/3122260030 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbacc0f5410 0x7fbacc106760 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fbac0009f80 tx=0x7fbac000b040).stop 2024-02-11T17:02:46.632 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:46.629+0000 7fbad2aa8700 1 -- 172.21.15.178:0/3122260030 shutdown_connections 2024-02-11T17:02:46.632 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:46.629+0000 7fbad2aa8700 1 --2- 172.21.15.178:0/3122260030 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fbabc02ff60 0x7fbabc032420 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:02:46.632 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:46.629+0000 7fbad2aa8700 1 --2- 172.21.15.178:0/3122260030 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbacc0f5410 0x7fbacc106760 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:02:46.632 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:46.629+0000 7fbad2aa8700 1 -- 172.21.15.178:0/3122260030 >> 172.21.15.178:0/3122260030 conn(0x7fbacc0f0b70 msgr2=0x7fbacc05fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:02:46.632 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:46.629+0000 7fbad2aa8700 1 -- 172.21.15.178:0/3122260030 shutdown_connections 2024-02-11T17:02:46.632 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:46.629+0000 7fbad2aa8700 1 -- 172.21.15.178:0/3122260030 wait complete. 2024-02-11T17:02:46.632 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:02:47.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:02:47 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:02:46.629104+0000 mon.smithi037 (mon.0) 155 : audit [DBG] from='client.? 172.21.15.178:0/3122260030' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:02:48.292 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:02:48.292 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:02:48.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:02:48 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:02:47.152504+0000 mgr.smithi037.eofnaq (mgr.14162) 83 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:02:50.069 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:50.067+0000 7f81c23d3700 1 -- 172.21.15.178:0/1306102444 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f81bc0f5410 msgr2=0x7f81bc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:02:50.069 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:50.067+0000 7f81c23d3700 1 --2- 172.21.15.178:0/1306102444 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f81bc0f5410 0x7f81bc0f5830 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f81ac009fb0 tx=0x7f81ac009b30).stop 2024-02-11T17:02:50.070 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:50.067+0000 7f81c23d3700 1 -- 172.21.15.178:0/1306102444 shutdown_connections 2024-02-11T17:02:50.070 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:50.067+0000 7f81c23d3700 1 --2- 172.21.15.178:0/1306102444 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f81bc0f5410 0x7f81bc0f5830 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:02:50.070 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:50.067+0000 7f81c23d3700 1 -- 172.21.15.178:0/1306102444 >> 172.21.15.178:0/1306102444 conn(0x7f81bc0f0b70 msgr2=0x7f81bc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:02:50.070 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:50.067+0000 7f81c23d3700 1 -- 172.21.15.178:0/1306102444 shutdown_connections 2024-02-11T17:02:50.070 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:50.067+0000 7f81c23d3700 1 -- 172.21.15.178:0/1306102444 wait complete. 2024-02-11T17:02:50.070 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:50.068+0000 7f81c23d3700 1 Processor -- start 2024-02-11T17:02:50.070 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:50.068+0000 7f81c23d3700 1 -- start start 2024-02-11T17:02:50.072 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:50.068+0000 7f81c23d3700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f81bc0f5410 0x7f81bc105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:02:50.072 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:50.068+0000 7f81c23d3700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81bc105b20 con 0x7f81bc0f5410 2024-02-11T17:02:50.072 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:50.068+0000 7f81bbfff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f81bc0f5410 0x7f81bc105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:02:50.072 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:50.069+0000 7f81bbfff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f81bc0f5410 0x7f81bc105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:51146/0 (socket says 172.21.15.178:51146) 2024-02-11T17:02:50.072 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:50.069+0000 7f81bbfff700 1 -- 172.21.15.178:0/2443389282 learned_addr learned my addr 172.21.15.178:0/2443389282 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:02:50.072 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:50.069+0000 7f81bbfff700 1 -- 172.21.15.178:0/2443389282 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81ac005040 con 0x7f81bc0f5410 2024-02-11T17:02:50.072 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:50.069+0000 7f81bbfff700 1 --2- 172.21.15.178:0/2443389282 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f81bc0f5410 0x7f81bc105630 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f81ac013510 tx=0x7f81ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:02:50.072 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:50.069+0000 7f81b8ff9700 1 -- 172.21.15.178:0/2443389282 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f81ac009320 con 0x7f81bc0f5410 2024-02-11T17:02:50.072 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:50.069+0000 7f81b8ff9700 1 -- 172.21.15.178:0/2443389282 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f81ac013f00 con 0x7f81bc0f5410 2024-02-11T17:02:50.072 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:50.069+0000 7f81b8ff9700 1 -- 172.21.15.178:0/2443389282 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f81ac014d90 con 0x7f81bc0f5410 2024-02-11T17:02:50.072 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:50.069+0000 7f81c23d3700 1 -- 172.21.15.178:0/2443389282 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81bc105d20 con 0x7f81bc0f5410 2024-02-11T17:02:50.072 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:50.069+0000 7f81c23d3700 1 -- 172.21.15.178:0/2443389282 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81bc1061b0 con 0x7f81bc0f5410 2024-02-11T17:02:50.073 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:50.070+0000 7f81b8ff9700 1 -- 172.21.15.178:0/2443389282 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f81ac0095f0 con 0x7f81bc0f5410 2024-02-11T17:02:50.073 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:50.070+0000 7f81c23d3700 1 -- 172.21.15.178:0/2443389282 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81bc045ad0 con 0x7f81bc0f5410 2024-02-11T17:02:50.073 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:50.070+0000 7f81b8ff9700 1 --2- 172.21.15.178:0/2443389282 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f81a402ff60 0x7f81a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:02:50.073 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:50.070+0000 7f81b8ff9700 1 -- 172.21.15.178:0/2443389282 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f81ac0410c0 con 0x7f81bc0f5410 2024-02-11T17:02:50.073 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:50.070+0000 7f81bb7fe700 1 --2- 172.21.15.178:0/2443389282 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f81a402ff60 0x7f81a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:02:50.073 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:50.071+0000 7f81bb7fe700 1 --2- 172.21.15.178:0/2443389282 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f81a402ff60 0x7f81a4032420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f81b00096a0 tx=0x7f81b0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:02:50.076 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:50.075+0000 7f81b8ff9700 1 -- 172.21.15.178:0/2443389282 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f81ac0263a0 con 0x7f81bc0f5410 2024-02-11T17:02:50.281 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:50.279+0000 7f81c23d3700 1 -- 172.21.15.178:0/2443389282 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f81bc0fae60 con 0x7f81bc0f5410 2024-02-11T17:02:50.281 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:50.280+0000 7f81b8ff9700 1 -- 172.21.15.178:0/2443389282 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f81ac010230 con 0x7f81bc0f5410 2024-02-11T17:02:50.281 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:02:50.282 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:02:50.283 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:50.281+0000 7f81c23d3700 1 -- 172.21.15.178:0/2443389282 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f81a402ff60 msgr2=0x7f81a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:02:50.283 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:50.281+0000 7f81c23d3700 1 --2- 172.21.15.178:0/2443389282 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f81a402ff60 0x7f81a4032420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f81b00096a0 tx=0x7f81b0006b40).stop 2024-02-11T17:02:50.283 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:50.281+0000 7f81c23d3700 1 -- 172.21.15.178:0/2443389282 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f81bc0f5410 msgr2=0x7f81bc105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:02:50.283 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:50.281+0000 7f81c23d3700 1 --2- 172.21.15.178:0/2443389282 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f81bc0f5410 0x7f81bc105630 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f81ac013510 tx=0x7f81ac00b040).stop 2024-02-11T17:02:50.283 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:50.281+0000 7f81c23d3700 1 -- 172.21.15.178:0/2443389282 shutdown_connections 2024-02-11T17:02:50.283 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:50.281+0000 7f81c23d3700 1 --2- 172.21.15.178:0/2443389282 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f81a402ff60 0x7f81a4032420 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:02:50.283 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:50.281+0000 7f81c23d3700 1 --2- 172.21.15.178:0/2443389282 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f81bc0f5410 0x7f81bc105630 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:02:50.283 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:50.281+0000 7f81c23d3700 1 -- 172.21.15.178:0/2443389282 >> 172.21.15.178:0/2443389282 conn(0x7f81bc0f0b70 msgr2=0x7f81bc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:02:50.284 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:50.281+0000 7f81c23d3700 1 -- 172.21.15.178:0/2443389282 shutdown_connections 2024-02-11T17:02:50.284 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:50.281+0000 7f81c23d3700 1 -- 172.21.15.178:0/2443389282 wait complete. 2024-02-11T17:02:50.284 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:02:50.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:02:50 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:02:49.152927+0000 mgr.smithi037.eofnaq (mgr.14162) 84 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:02:51.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:02:51 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:02:50.281059+0000 mon.smithi037 (mon.0) 156 : audit [DBG] from='client.? 172.21.15.178:0/2443389282' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:02:51.965 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:02:51.965 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:02:52.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:02:52 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:02:51.153330+0000 mgr.smithi037.eofnaq (mgr.14162) 85 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:02:53.770 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:53.768+0000 7fd4409db700 1 -- 172.21.15.178:0/2624987801 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd43c0f5410 msgr2=0x7fd43c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:02:53.770 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:53.768+0000 7fd4409db700 1 --2- 172.21.15.178:0/2624987801 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd43c0f5410 0x7fd43c0f5830 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fd42c009fb0 tx=0x7fd42c009b30).stop 2024-02-11T17:02:53.770 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:53.768+0000 7fd4409db700 1 -- 172.21.15.178:0/2624987801 shutdown_connections 2024-02-11T17:02:53.770 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:53.768+0000 7fd4409db700 1 --2- 172.21.15.178:0/2624987801 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd43c0f5410 0x7fd43c0f5830 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:02:53.770 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:53.768+0000 7fd4409db700 1 -- 172.21.15.178:0/2624987801 >> 172.21.15.178:0/2624987801 conn(0x7fd43c0f0b70 msgr2=0x7fd43c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:02:53.770 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:53.768+0000 7fd4409db700 1 -- 172.21.15.178:0/2624987801 shutdown_connections 2024-02-11T17:02:53.771 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:53.768+0000 7fd4409db700 1 -- 172.21.15.178:0/2624987801 wait complete. 2024-02-11T17:02:53.771 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:53.768+0000 7fd4409db700 1 Processor -- start 2024-02-11T17:02:53.771 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:53.769+0000 7fd4409db700 1 -- start start 2024-02-11T17:02:53.772 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:53.769+0000 7fd4409db700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd43c0f5410 0x7fd43c105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:02:53.772 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:53.769+0000 7fd4409db700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd43c105b20 con 0x7fd43c0f5410 2024-02-11T17:02:53.772 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:53.769+0000 7fd43a59c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd43c0f5410 0x7fd43c105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:02:53.772 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:53.769+0000 7fd43a59c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd43c0f5410 0x7fd43c105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:51154/0 (socket says 172.21.15.178:51154) 2024-02-11T17:02:53.772 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:53.769+0000 7fd43a59c700 1 -- 172.21.15.178:0/1399606673 learned_addr learned my addr 172.21.15.178:0/1399606673 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:02:53.773 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:53.769+0000 7fd43a59c700 1 -- 172.21.15.178:0/1399606673 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd42c005040 con 0x7fd43c0f5410 2024-02-11T17:02:53.773 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:53.769+0000 7fd43a59c700 1 --2- 172.21.15.178:0/1399606673 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd43c0f5410 0x7fd43c105630 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fd42c013510 tx=0x7fd42c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:02:53.773 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:53.770+0000 7fd42affd700 1 -- 172.21.15.178:0/1399606673 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd42c009320 con 0x7fd43c0f5410 2024-02-11T17:02:53.773 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:53.770+0000 7fd4409db700 1 -- 172.21.15.178:0/1399606673 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd43c105d20 con 0x7fd43c0f5410 2024-02-11T17:02:53.773 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:53.770+0000 7fd42affd700 1 -- 172.21.15.178:0/1399606673 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd42c013f00 con 0x7fd43c0f5410 2024-02-11T17:02:53.773 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:53.770+0000 7fd42affd700 1 -- 172.21.15.178:0/1399606673 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd42c014d90 con 0x7fd43c0f5410 2024-02-11T17:02:53.773 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:53.770+0000 7fd4409db700 1 -- 172.21.15.178:0/1399606673 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd43c1061b0 con 0x7fd43c0f5410 2024-02-11T17:02:53.773 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:53.770+0000 7fd42affd700 1 -- 172.21.15.178:0/1399606673 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd42c00f410 con 0x7fd43c0f5410 2024-02-11T17:02:53.773 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:53.771+0000 7fd42affd700 1 --2- 172.21.15.178:0/1399606673 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fd42402ff60 0x7fd424032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:02:53.773 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:53.771+0000 7fd42affd700 1 -- 172.21.15.178:0/1399606673 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd42c042e50 con 0x7fd43c0f5410 2024-02-11T17:02:53.773 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:53.771+0000 7fd4409db700 1 -- 172.21.15.178:0/1399606673 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd43c045ad0 con 0x7fd43c0f5410 2024-02-11T17:02:53.773 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:53.771+0000 7fd439d9b700 1 --2- 172.21.15.178:0/1399606673 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fd42402ff60 0x7fd424032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:02:53.774 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:53.771+0000 7fd439d9b700 1 --2- 172.21.15.178:0/1399606673 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fd42402ff60 0x7fd424032420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fd4300096a0 tx=0x7fd430006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:02:53.777 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:53.775+0000 7fd42affd700 1 -- 172.21.15.178:0/1399606673 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd42c012920 con 0x7fd43c0f5410 2024-02-11T17:02:53.981 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:53.979+0000 7fd4409db700 1 -- 172.21.15.178:0/1399606673 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd43c0faf50 con 0x7fd43c0f5410 2024-02-11T17:02:53.981 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:53.980+0000 7fd42affd700 1 -- 172.21.15.178:0/1399606673 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd42c012330 con 0x7fd43c0f5410 2024-02-11T17:02:53.982 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:02:53.982 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:02:53.983 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:53.981+0000 7fd4409db700 1 -- 172.21.15.178:0/1399606673 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fd42402ff60 msgr2=0x7fd424032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:02:53.983 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:53.981+0000 7fd4409db700 1 --2- 172.21.15.178:0/1399606673 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fd42402ff60 0x7fd424032420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fd4300096a0 tx=0x7fd430006b40).stop 2024-02-11T17:02:53.983 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:53.981+0000 7fd4409db700 1 -- 172.21.15.178:0/1399606673 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd43c0f5410 msgr2=0x7fd43c105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:02:53.983 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:53.981+0000 7fd4409db700 1 --2- 172.21.15.178:0/1399606673 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd43c0f5410 0x7fd43c105630 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fd42c013510 tx=0x7fd42c00b040).stop 2024-02-11T17:02:53.983 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:53.981+0000 7fd4409db700 1 -- 172.21.15.178:0/1399606673 shutdown_connections 2024-02-11T17:02:53.984 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:53.981+0000 7fd4409db700 1 --2- 172.21.15.178:0/1399606673 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fd42402ff60 0x7fd424032420 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:02:53.984 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:53.981+0000 7fd4409db700 1 --2- 172.21.15.178:0/1399606673 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd43c0f5410 0x7fd43c105630 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:02:53.984 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:53.981+0000 7fd4409db700 1 -- 172.21.15.178:0/1399606673 >> 172.21.15.178:0/1399606673 conn(0x7fd43c0f0b70 msgr2=0x7fd43c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:02:53.984 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:53.982+0000 7fd4409db700 1 -- 172.21.15.178:0/1399606673 shutdown_connections 2024-02-11T17:02:53.984 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:53.982+0000 7fd4409db700 1 -- 172.21.15.178:0/1399606673 wait complete. 2024-02-11T17:02:53.984 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:02:54.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:02:54 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:02:53.153735+0000 mgr.smithi037.eofnaq (mgr.14162) 86 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:02:54.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:02:54 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:02:53.981237+0000 mon.smithi037 (mon.0) 157 : audit [DBG] from='client.? 172.21.15.178:0/1399606673' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:02:55.762 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:02:55.763 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:02:56.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:02:56 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:02:55.154124+0000 mgr.smithi037.eofnaq (mgr.14162) 87 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:02:57.652 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:57.649+0000 7f042a87c700 1 -- 172.21.15.178:0/137831508 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f04240f5410 msgr2=0x7f04240f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:02:57.652 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:57.649+0000 7f042a87c700 1 --2- 172.21.15.178:0/137831508 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f04240f5410 0x7f04240f5830 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f0414009fb0 tx=0x7f0414009b30).stop 2024-02-11T17:02:57.652 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:57.649+0000 7f042a87c700 1 -- 172.21.15.178:0/137831508 shutdown_connections 2024-02-11T17:02:57.652 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:57.649+0000 7f042a87c700 1 --2- 172.21.15.178:0/137831508 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f04240f5410 0x7f04240f5830 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:02:57.652 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:57.649+0000 7f042a87c700 1 -- 172.21.15.178:0/137831508 >> 172.21.15.178:0/137831508 conn(0x7f04240f0b70 msgr2=0x7f04240f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:02:57.652 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:57.649+0000 7f042a87c700 1 -- 172.21.15.178:0/137831508 shutdown_connections 2024-02-11T17:02:57.652 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:57.649+0000 7f042a87c700 1 -- 172.21.15.178:0/137831508 wait complete. 2024-02-11T17:02:57.652 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:57.649+0000 7f042a87c700 1 Processor -- start 2024-02-11T17:02:57.652 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:57.649+0000 7f042a87c700 1 -- start start 2024-02-11T17:02:57.653 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:57.649+0000 7f042a87c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f04240f5410 0x7f0424106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:02:57.653 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:57.649+0000 7f042a87c700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0424106c00 con 0x7f04240f5410 2024-02-11T17:02:57.653 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:57.650+0000 7f0423fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f04240f5410 0x7f0424106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:02:57.653 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:57.650+0000 7f0423fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f04240f5410 0x7f0424106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:51172/0 (socket says 172.21.15.178:51172) 2024-02-11T17:02:57.653 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:57.650+0000 7f0423fff700 1 -- 172.21.15.178:0/2695869322 learned_addr learned my addr 172.21.15.178:0/2695869322 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:02:57.653 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:57.650+0000 7f0423fff700 1 -- 172.21.15.178:0/2695869322 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0414005040 con 0x7f04240f5410 2024-02-11T17:02:57.653 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:57.650+0000 7f0423fff700 1 --2- 172.21.15.178:0/2695869322 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f04240f5410 0x7f0424106710 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f0414013510 tx=0x7f041400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:02:57.653 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:57.650+0000 7f0420ff9700 1 -- 172.21.15.178:0/2695869322 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0414009320 con 0x7f04240f5410 2024-02-11T17:02:57.653 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:57.650+0000 7f0420ff9700 1 -- 172.21.15.178:0/2695869322 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0414013f00 con 0x7f04240f5410 2024-02-11T17:02:57.653 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:57.650+0000 7f042a87c700 1 -- 172.21.15.178:0/2695869322 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0424106e00 con 0x7f04240f5410 2024-02-11T17:02:57.653 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:57.650+0000 7f0420ff9700 1 -- 172.21.15.178:0/2695869322 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0414014d90 con 0x7f04240f5410 2024-02-11T17:02:57.653 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:57.650+0000 7f042a87c700 1 -- 172.21.15.178:0/2695869322 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0424107290 con 0x7f04240f5410 2024-02-11T17:02:57.654 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:57.651+0000 7f0420ff9700 1 -- 172.21.15.178:0/2695869322 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0414014420 con 0x7f04240f5410 2024-02-11T17:02:57.654 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:57.651+0000 7f042a87c700 1 -- 172.21.15.178:0/2695869322 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0424045ad0 con 0x7f04240f5410 2024-02-11T17:02:57.654 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:57.651+0000 7f0420ff9700 1 --2- 172.21.15.178:0/2695869322 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f040c02ff60 0x7f040c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:02:57.654 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:57.651+0000 7f0420ff9700 1 -- 172.21.15.178:0/2695869322 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0414041000 con 0x7f04240f5410 2024-02-11T17:02:57.654 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:57.652+0000 7f04237fe700 1 --2- 172.21.15.178:0/2695869322 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f040c02ff60 0x7f040c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:02:57.655 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:57.652+0000 7f04237fe700 1 --2- 172.21.15.178:0/2695869322 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f040c02ff60 0x7f040c032420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f04180096a0 tx=0x7f0418006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:02:57.658 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:57.656+0000 7f0420ff9700 1 -- 172.21.15.178:0/2695869322 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f04140095f0 con 0x7f04240f5410 2024-02-11T17:02:57.860 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:57.859+0000 7f042a87c700 1 -- 172.21.15.178:0/2695869322 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f04240fae60 con 0x7f04240f5410 2024-02-11T17:02:57.861 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:57.859+0000 7f0420ff9700 1 -- 172.21.15.178:0/2695869322 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f041401db70 con 0x7f04240f5410 2024-02-11T17:02:57.861 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:02:57.861 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:02:57.863 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:57.861+0000 7f042a87c700 1 -- 172.21.15.178:0/2695869322 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f040c02ff60 msgr2=0x7f040c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:02:57.863 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:57.861+0000 7f042a87c700 1 --2- 172.21.15.178:0/2695869322 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f040c02ff60 0x7f040c032420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f04180096a0 tx=0x7f0418006b40).stop 2024-02-11T17:02:57.863 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:57.861+0000 7f042a87c700 1 -- 172.21.15.178:0/2695869322 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f04240f5410 msgr2=0x7f0424106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:02:57.863 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:57.861+0000 7f042a87c700 1 --2- 172.21.15.178:0/2695869322 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f04240f5410 0x7f0424106710 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f0414013510 tx=0x7f041400b040).stop 2024-02-11T17:02:57.863 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:57.861+0000 7f042a87c700 1 -- 172.21.15.178:0/2695869322 shutdown_connections 2024-02-11T17:02:57.863 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:57.861+0000 7f042a87c700 1 --2- 172.21.15.178:0/2695869322 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f040c02ff60 0x7f040c032420 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:02:57.863 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:57.861+0000 7f042a87c700 1 --2- 172.21.15.178:0/2695869322 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f04240f5410 0x7f0424106710 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:02:57.863 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:57.861+0000 7f042a87c700 1 -- 172.21.15.178:0/2695869322 >> 172.21.15.178:0/2695869322 conn(0x7f04240f0b70 msgr2=0x7f042405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:02:57.863 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:57.861+0000 7f042a87c700 1 -- 172.21.15.178:0/2695869322 shutdown_connections 2024-02-11T17:02:57.863 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:02:57.861+0000 7f042a87c700 1 -- 172.21.15.178:0/2695869322 wait complete. 2024-02-11T17:02:57.864 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:02:58.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:02:58 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:02:57.154509+0000 mgr.smithi037.eofnaq (mgr.14162) 88 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:02:58.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:02:58 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:02:57.860697+0000 mon.smithi037 (mon.0) 158 : audit [DBG] from='client.? 172.21.15.178:0/2695869322' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:02:59.559 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:02:59.560 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:03:00.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:03:00 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:02:59.154932+0000 mgr.smithi037.eofnaq (mgr.14162) 89 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:03:01.545 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:01.541+0000 7f324a7e5700 1 -- 172.21.15.178:0/757344384 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3244060280 msgr2=0x7f32440f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:03:01.545 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:01.541+0000 7f324a7e5700 1 --2- 172.21.15.178:0/757344384 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3244060280 0x7f32440f90f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f3234009fb0 tx=0x7f3234009b30).stop 2024-02-11T17:03:01.545 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:01.541+0000 7f324a7e5700 1 -- 172.21.15.178:0/757344384 shutdown_connections 2024-02-11T17:03:01.545 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:01.541+0000 7f324a7e5700 1 --2- 172.21.15.178:0/757344384 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3244060280 0x7f32440f90f0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:03:01.545 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:01.541+0000 7f324a7e5700 1 -- 172.21.15.178:0/757344384 >> 172.21.15.178:0/757344384 conn(0x7f32440f0b70 msgr2=0x7f32440f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:03:01.545 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:01.541+0000 7f324a7e5700 1 -- 172.21.15.178:0/757344384 shutdown_connections 2024-02-11T17:03:01.545 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:01.541+0000 7f324a7e5700 1 -- 172.21.15.178:0/757344384 wait complete. 2024-02-11T17:03:01.545 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:01.542+0000 7f324a7e5700 1 Processor -- start 2024-02-11T17:03:01.545 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:01.542+0000 7f324a7e5700 1 -- start start 2024-02-11T17:03:01.545 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:01.542+0000 7f324a7e5700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3244060280 0x7f32441066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:03:01.545 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:01.542+0000 7f324a7e5700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3244106bd0 con 0x7f3244060280 2024-02-11T17:03:01.546 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:01.542+0000 7f3243fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3244060280 0x7f32441066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:03:01.546 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:01.542+0000 7f3243fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3244060280 0x7f32441066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:47890/0 (socket says 172.21.15.178:47890) 2024-02-11T17:03:01.546 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:01.542+0000 7f3243fff700 1 -- 172.21.15.178:0/2226291214 learned_addr learned my addr 172.21.15.178:0/2226291214 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:03:01.546 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:01.543+0000 7f3243fff700 1 -- 172.21.15.178:0/2226291214 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3234005040 con 0x7f3244060280 2024-02-11T17:03:01.546 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:01.543+0000 7f3243fff700 1 --2- 172.21.15.178:0/2226291214 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3244060280 0x7f32441066e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f3234013510 tx=0x7f323400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:03:01.546 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:01.543+0000 7f3240ff9700 1 -- 172.21.15.178:0/2226291214 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3234009320 con 0x7f3244060280 2024-02-11T17:03:01.546 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:01.543+0000 7f3240ff9700 1 -- 172.21.15.178:0/2226291214 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3234013f00 con 0x7f3244060280 2024-02-11T17:03:01.546 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:01.543+0000 7f324a7e5700 1 -- 172.21.15.178:0/2226291214 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3244106dd0 con 0x7f3244060280 2024-02-11T17:03:01.546 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:01.543+0000 7f3240ff9700 1 -- 172.21.15.178:0/2226291214 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3234014dc0 con 0x7f3244060280 2024-02-11T17:03:01.546 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:01.543+0000 7f324a7e5700 1 -- 172.21.15.178:0/2226291214 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3244107260 con 0x7f3244060280 2024-02-11T17:03:01.547 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:01.544+0000 7f3240ff9700 1 -- 172.21.15.178:0/2226291214 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f323401d3d0 con 0x7f3244060280 2024-02-11T17:03:01.547 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:01.544+0000 7f324a7e5700 1 -- 172.21.15.178:0/2226291214 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3244045ad0 con 0x7f3244060280 2024-02-11T17:03:01.547 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:01.544+0000 7f3240ff9700 1 --2- 172.21.15.178:0/2226291214 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f322c02ff60 0x7f322c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:03:01.547 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:01.544+0000 7f3240ff9700 1 -- 172.21.15.178:0/2226291214 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3234041120 con 0x7f3244060280 2024-02-11T17:03:01.547 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:01.544+0000 7f32437fe700 1 --2- 172.21.15.178:0/2226291214 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f322c02ff60 0x7f322c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:03:01.547 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:01.545+0000 7f32437fe700 1 --2- 172.21.15.178:0/2226291214 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f322c02ff60 0x7f322c032420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f32380096a0 tx=0x7f3238006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:03:01.550 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:01.549+0000 7f3240ff9700 1 -- 172.21.15.178:0/2226291214 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f32340126f0 con 0x7f3244060280 2024-02-11T17:03:01.755 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:01.753+0000 7f324a7e5700 1 -- 172.21.15.178:0/2226291214 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3244108110 con 0x7f3244060280 2024-02-11T17:03:01.755 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:01.755+0000 7f3240ff9700 1 -- 172.21.15.178:0/2226291214 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3234012330 con 0x7f3244060280 2024-02-11T17:03:01.755 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:03:01.756 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:03:01.757 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:01.755+0000 7f324a7e5700 1 -- 172.21.15.178:0/2226291214 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f322c02ff60 msgr2=0x7f322c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:03:01.757 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:01.755+0000 7f324a7e5700 1 --2- 172.21.15.178:0/2226291214 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f322c02ff60 0x7f322c032420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f32380096a0 tx=0x7f3238006b40).stop 2024-02-11T17:03:01.757 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:01.755+0000 7f324a7e5700 1 -- 172.21.15.178:0/2226291214 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3244060280 msgr2=0x7f32441066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:03:01.757 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:01.755+0000 7f324a7e5700 1 --2- 172.21.15.178:0/2226291214 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3244060280 0x7f32441066e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f3234013510 tx=0x7f323400b040).stop 2024-02-11T17:03:01.757 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:01.755+0000 7f324a7e5700 1 -- 172.21.15.178:0/2226291214 shutdown_connections 2024-02-11T17:03:01.757 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:01.755+0000 7f324a7e5700 1 --2- 172.21.15.178:0/2226291214 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f322c02ff60 0x7f322c032420 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:03:01.757 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:01.755+0000 7f324a7e5700 1 --2- 172.21.15.178:0/2226291214 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3244060280 0x7f32441066e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:03:01.757 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:01.755+0000 7f324a7e5700 1 -- 172.21.15.178:0/2226291214 >> 172.21.15.178:0/2226291214 conn(0x7f32440f0b70 msgr2=0x7f32440f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:03:01.758 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:01.755+0000 7f324a7e5700 1 -- 172.21.15.178:0/2226291214 shutdown_connections 2024-02-11T17:03:01.758 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:01.756+0000 7f324a7e5700 1 -- 172.21.15.178:0/2226291214 wait complete. 2024-02-11T17:03:01.758 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:03:02.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:03:02 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:03:01.155260+0000 mgr.smithi037.eofnaq (mgr.14162) 90 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:03:02.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:03:02 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:03:01.755097+0000 mon.smithi037 (mon.0) 159 : audit [DBG] from='client.? 172.21.15.178:0/2226291214' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:03:03.426 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:03:03.427 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:03:04.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:03:04 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:03:03.155629+0000 mgr.smithi037.eofnaq (mgr.14162) 91 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:03:04.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:03:04 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:03:03.272783+0000 mon.smithi037 (mon.0) 160 : audit [INF] from='mgr.14162 172.21.15.37:0/2745465012' entity='mgr.smithi037.eofnaq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi037.eofnaq/mirror_snapshot_schedule"}]: dispatch 2024-02-11T17:03:04.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:03:04 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:03:03.276399+0000 mon.smithi037 (mon.0) 161 : audit [INF] from='mgr.14162 172.21.15.37:0/2745465012' entity='mgr.smithi037.eofnaq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi037.eofnaq/trash_purge_schedule"}]: dispatch 2024-02-11T17:03:05.256 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:05.244+0000 7f438d91e700 1 -- 172.21.15.178:0/2062661084 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f43880f5410 msgr2=0x7f43880f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:03:05.256 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:05.244+0000 7f438d91e700 1 --2- 172.21.15.178:0/2062661084 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f43880f5410 0x7f43880f5830 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f437c009fb0 tx=0x7f437c009b30).stop 2024-02-11T17:03:05.256 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:05.244+0000 7f438d91e700 1 -- 172.21.15.178:0/2062661084 shutdown_connections 2024-02-11T17:03:05.256 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:05.244+0000 7f438d91e700 1 --2- 172.21.15.178:0/2062661084 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f43880f5410 0x7f43880f5830 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:03:05.256 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:05.244+0000 7f438d91e700 1 -- 172.21.15.178:0/2062661084 >> 172.21.15.178:0/2062661084 conn(0x7f43880f0b70 msgr2=0x7f43880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:03:05.256 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:05.244+0000 7f438d91e700 1 -- 172.21.15.178:0/2062661084 shutdown_connections 2024-02-11T17:03:05.256 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:05.244+0000 7f438d91e700 1 -- 172.21.15.178:0/2062661084 wait complete. 2024-02-11T17:03:05.256 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:05.245+0000 7f438d91e700 1 Processor -- start 2024-02-11T17:03:05.257 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:05.245+0000 7f438d91e700 1 -- start start 2024-02-11T17:03:05.257 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:05.245+0000 7f438d91e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f43880f5410 0x7f4388109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:03:05.257 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:05.245+0000 7f438d91e700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4388060240 con 0x7f43880f5410 2024-02-11T17:03:05.257 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:05.246+0000 7f4386ffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f43880f5410 0x7f4388109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:03:05.257 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:05.246+0000 7f4386ffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f43880f5410 0x7f4388109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:47902/0 (socket says 172.21.15.178:47902) 2024-02-11T17:03:05.257 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:05.246+0000 7f4386ffd700 1 -- 172.21.15.178:0/1245176973 learned_addr learned my addr 172.21.15.178:0/1245176973 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:03:05.257 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:05.246+0000 7f4386ffd700 1 -- 172.21.15.178:0/1245176973 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f437c005040 con 0x7f43880f5410 2024-02-11T17:03:05.257 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:05.246+0000 7f4386ffd700 1 --2- 172.21.15.178:0/1245176973 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f43880f5410 0x7f4388109430 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f437c013510 tx=0x7f437c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:03:05.257 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:05.246+0000 7f436f7fe700 1 -- 172.21.15.178:0/1245176973 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f437c009320 con 0x7f43880f5410 2024-02-11T17:03:05.257 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:05.246+0000 7f436f7fe700 1 -- 172.21.15.178:0/1245176973 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f437c013f00 con 0x7f43880f5410 2024-02-11T17:03:05.257 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:05.246+0000 7f438d91e700 1 -- 172.21.15.178:0/1245176973 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4388060440 con 0x7f43880f5410 2024-02-11T17:03:05.257 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:05.246+0000 7f436f7fe700 1 -- 172.21.15.178:0/1245176973 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f437c014d90 con 0x7f43880f5410 2024-02-11T17:03:05.257 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:05.246+0000 7f438d91e700 1 -- 172.21.15.178:0/1245176973 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4388109ae0 con 0x7f43880f5410 2024-02-11T17:03:05.257 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:05.247+0000 7f436f7fe700 1 -- 172.21.15.178:0/1245176973 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f437c0095f0 con 0x7f43880f5410 2024-02-11T17:03:05.258 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:05.247+0000 7f438d91e700 1 -- 172.21.15.178:0/1245176973 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4388045ad0 con 0x7f43880f5410 2024-02-11T17:03:05.258 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:05.247+0000 7f436f7fe700 1 --2- 172.21.15.178:0/1245176973 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f437002ffb0 0x7f4370032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:03:05.258 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:05.247+0000 7f436f7fe700 1 -- 172.21.15.178:0/1245176973 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f437c0410c0 con 0x7f43880f5410 2024-02-11T17:03:05.258 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:05.248+0000 7f43867fc700 1 --2- 172.21.15.178:0/1245176973 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f437002ffb0 0x7f4370032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:03:05.258 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:05.248+0000 7f43867fc700 1 --2- 172.21.15.178:0/1245176973 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f437002ffb0 0x7f4370032470 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f43780096a0 tx=0x7f4378006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:03:05.258 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:05.252+0000 7f436f7fe700 1 -- 172.21.15.178:0/1245176973 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f437c014420 con 0x7f43880f5410 2024-02-11T17:03:05.460 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:05.459+0000 7f438d91e700 1 -- 172.21.15.178:0/1245176973 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f43880fae60 con 0x7f43880f5410 2024-02-11T17:03:05.461 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:05.459+0000 7f436f7fe700 1 -- 172.21.15.178:0/1245176973 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f437c01d7f0 con 0x7f43880f5410 2024-02-11T17:03:05.461 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:03:05.461 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:03:05.462 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:05.461+0000 7f438d91e700 1 -- 172.21.15.178:0/1245176973 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f437002ffb0 msgr2=0x7f4370032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:03:05.462 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:05.461+0000 7f438d91e700 1 --2- 172.21.15.178:0/1245176973 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f437002ffb0 0x7f4370032470 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f43780096a0 tx=0x7f4378006b40).stop 2024-02-11T17:03:05.463 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:05.461+0000 7f438d91e700 1 -- 172.21.15.178:0/1245176973 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f43880f5410 msgr2=0x7f4388109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:03:05.463 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:05.461+0000 7f438d91e700 1 --2- 172.21.15.178:0/1245176973 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f43880f5410 0x7f4388109430 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f437c013510 tx=0x7f437c00b040).stop 2024-02-11T17:03:05.463 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:05.461+0000 7f438d91e700 1 -- 172.21.15.178:0/1245176973 shutdown_connections 2024-02-11T17:03:05.463 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:05.461+0000 7f438d91e700 1 --2- 172.21.15.178:0/1245176973 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f437002ffb0 0x7f4370032470 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:03:05.463 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:05.461+0000 7f438d91e700 1 --2- 172.21.15.178:0/1245176973 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f43880f5410 0x7f4388109430 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:03:05.463 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:05.461+0000 7f438d91e700 1 -- 172.21.15.178:0/1245176973 >> 172.21.15.178:0/1245176973 conn(0x7f43880f0b70 msgr2=0x7f43880f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:03:05.463 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:05.461+0000 7f438d91e700 1 -- 172.21.15.178:0/1245176973 shutdown_connections 2024-02-11T17:03:05.463 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:05.461+0000 7f438d91e700 1 -- 172.21.15.178:0/1245176973 wait complete. 2024-02-11T17:03:05.463 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:03:06.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:03:06 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:03:05.155926+0000 mgr.smithi037.eofnaq (mgr.14162) 92 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:03:06.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:03:06 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:03:05.460522+0000 mon.smithi037 (mon.0) 162 : audit [DBG] from='client.? 172.21.15.178:0/1245176973' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:03:07.094 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:03:07.094 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:03:08.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:03:08 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:03:07.156240+0000 mgr.smithi037.eofnaq (mgr.14162) 93 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:03:08.959 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:08.957+0000 7ff915308700 1 -- 172.21.15.178:0/4003000724 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff9100f7b60 msgr2=0x7ff9100f7f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:03:08.959 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:08.957+0000 7ff915308700 1 --2- 172.21.15.178:0/4003000724 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff9100f7b60 0x7ff9100f7f80 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7ff900009fb0 tx=0x7ff900009b30).stop 2024-02-11T17:03:08.959 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:08.957+0000 7ff915308700 1 -- 172.21.15.178:0/4003000724 shutdown_connections 2024-02-11T17:03:08.959 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:08.957+0000 7ff915308700 1 --2- 172.21.15.178:0/4003000724 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff9100f7b60 0x7ff9100f7f80 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:03:08.960 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:08.957+0000 7ff915308700 1 -- 172.21.15.178:0/4003000724 >> 172.21.15.178:0/4003000724 conn(0x7ff9100f02e0 msgr2=0x7ff9100f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:03:08.960 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:08.958+0000 7ff915308700 1 -- 172.21.15.178:0/4003000724 shutdown_connections 2024-02-11T17:03:08.960 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:08.958+0000 7ff915308700 1 -- 172.21.15.178:0/4003000724 wait complete. 2024-02-11T17:03:08.960 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:08.958+0000 7ff915308700 1 Processor -- start 2024-02-11T17:03:08.960 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:08.958+0000 7ff915308700 1 -- start start 2024-02-11T17:03:08.960 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:08.958+0000 7ff915308700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff9100f7b60 0x7ff9100f7430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:03:08.960 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:08.958+0000 7ff915308700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9100f58b0 con 0x7ff9100f7b60 2024-02-11T17:03:08.960 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:08.959+0000 7ff90effd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff9100f7b60 0x7ff9100f7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:03:08.960 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:08.959+0000 7ff90effd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff9100f7b60 0x7ff9100f7430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:47924/0 (socket says 172.21.15.178:47924) 2024-02-11T17:03:08.960 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:08.959+0000 7ff90effd700 1 -- 172.21.15.178:0/1632363959 learned_addr learned my addr 172.21.15.178:0/1632363959 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:03:08.960 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:08.959+0000 7ff90effd700 1 -- 172.21.15.178:0/1632363959 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff900005040 con 0x7ff9100f7b60 2024-02-11T17:03:08.962 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:08.959+0000 7ff90effd700 1 --2- 172.21.15.178:0/1632363959 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff9100f7b60 0x7ff9100f7430 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7ff900013510 tx=0x7ff90000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:03:08.962 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:08.959+0000 7ff8f77fe700 1 -- 172.21.15.178:0/1632363959 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff900009320 con 0x7ff9100f7b60 2024-02-11T17:03:08.962 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:08.959+0000 7ff915308700 1 -- 172.21.15.178:0/1632363959 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9100f5a50 con 0x7ff9100f7b60 2024-02-11T17:03:08.962 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:08.959+0000 7ff8f77fe700 1 -- 172.21.15.178:0/1632363959 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff900013f00 con 0x7ff9100f7b60 2024-02-11T17:03:08.962 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:08.959+0000 7ff8f77fe700 1 -- 172.21.15.178:0/1632363959 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff900014d90 con 0x7ff9100f7b60 2024-02-11T17:03:08.962 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:08.959+0000 7ff915308700 1 -- 172.21.15.178:0/1632363959 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9100f5ee0 con 0x7ff9100f7b60 2024-02-11T17:03:08.962 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:08.960+0000 7ff8f77fe700 1 -- 172.21.15.178:0/1632363959 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff900014420 con 0x7ff9100f7b60 2024-02-11T17:03:08.962 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:08.960+0000 7ff915308700 1 -- 172.21.15.178:0/1632363959 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff910059100 con 0x7ff9100f7b60 2024-02-11T17:03:08.962 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:08.960+0000 7ff8f77fe700 1 --2- 172.21.15.178:0/1632363959 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7ff8f802ffb0 0x7ff8f8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:03:08.962 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:08.961+0000 7ff8f77fe700 1 -- 172.21.15.178:0/1632363959 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff900040fd0 con 0x7ff9100f7b60 2024-02-11T17:03:08.962 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:08.961+0000 7ff90e7fc700 1 --2- 172.21.15.178:0/1632363959 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7ff8f802ffb0 0x7ff8f8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:03:08.962 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:08.961+0000 7ff90e7fc700 1 --2- 172.21.15.178:0/1632363959 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7ff8f802ffb0 0x7ff8f8032470 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7ff9040096a0 tx=0x7ff904006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:03:08.966 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:08.965+0000 7ff8f77fe700 1 -- 172.21.15.178:0/1632363959 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff90006d2b0 con 0x7ff9100f7b60 2024-02-11T17:03:09.170 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:09.170+0000 7ff915308700 1 -- 172.21.15.178:0/1632363959 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff9100fbf60 con 0x7ff9100f7b60 2024-02-11T17:03:09.171 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:09.170+0000 7ff8f77fe700 1 -- 172.21.15.178:0/1632363959 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff9000095f0 con 0x7ff9100f7b60 2024-02-11T17:03:09.171 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:03:09.171 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:03:09.173 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:09.172+0000 7ff915308700 1 -- 172.21.15.178:0/1632363959 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7ff8f802ffb0 msgr2=0x7ff8f8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:03:09.173 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:09.172+0000 7ff915308700 1 --2- 172.21.15.178:0/1632363959 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7ff8f802ffb0 0x7ff8f8032470 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7ff9040096a0 tx=0x7ff904006b40).stop 2024-02-11T17:03:09.173 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:09.172+0000 7ff915308700 1 -- 172.21.15.178:0/1632363959 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff9100f7b60 msgr2=0x7ff9100f7430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:03:09.173 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:09.172+0000 7ff915308700 1 --2- 172.21.15.178:0/1632363959 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff9100f7b60 0x7ff9100f7430 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7ff900013510 tx=0x7ff90000b040).stop 2024-02-11T17:03:09.173 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:09.172+0000 7ff915308700 1 -- 172.21.15.178:0/1632363959 shutdown_connections 2024-02-11T17:03:09.173 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:09.172+0000 7ff915308700 1 --2- 172.21.15.178:0/1632363959 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7ff8f802ffb0 0x7ff8f8032470 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:03:09.173 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:09.172+0000 7ff915308700 1 --2- 172.21.15.178:0/1632363959 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff9100f7b60 0x7ff9100f7430 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:03:09.173 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:09.172+0000 7ff915308700 1 -- 172.21.15.178:0/1632363959 >> 172.21.15.178:0/1632363959 conn(0x7ff9100f02e0 msgr2=0x7ff9100f0e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:03:09.174 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:09.172+0000 7ff915308700 1 -- 172.21.15.178:0/1632363959 shutdown_connections 2024-02-11T17:03:09.174 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:09.172+0000 7ff915308700 1 -- 172.21.15.178:0/1632363959 wait complete. 2024-02-11T17:03:09.174 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:03:09.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:03:09 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:03:09.170649+0000 mon.smithi037 (mon.0) 163 : audit [DBG] from='client.? 172.21.15.178:0/1632363959' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:03:10.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:03:10 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:03:09.156507+0000 mgr.smithi037.eofnaq (mgr.14162) 94 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:03:10.810 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:03:10.810 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:03:12.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:03:12 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:03:11.156871+0000 mgr.smithi037.eofnaq (mgr.14162) 95 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:03:12.764 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:12.763+0000 7fbd4df93700 1 -- 172.21.15.178:0/1941170596 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbd48060220 msgr2=0x7fbd480f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:03:12.765 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:12.763+0000 7fbd4df93700 1 --2- 172.21.15.178:0/1941170596 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbd48060220 0x7fbd480f9150 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fbd38009fb0 tx=0x7fbd38009b30).stop 2024-02-11T17:03:12.765 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:12.763+0000 7fbd4df93700 1 -- 172.21.15.178:0/1941170596 shutdown_connections 2024-02-11T17:03:12.765 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:12.763+0000 7fbd4df93700 1 --2- 172.21.15.178:0/1941170596 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbd48060220 0x7fbd480f9150 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:03:12.765 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:12.763+0000 7fbd4df93700 1 -- 172.21.15.178:0/1941170596 >> 172.21.15.178:0/1941170596 conn(0x7fbd480f0b70 msgr2=0x7fbd480f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:03:12.765 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:12.763+0000 7fbd4df93700 1 -- 172.21.15.178:0/1941170596 shutdown_connections 2024-02-11T17:03:12.765 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:12.763+0000 7fbd4df93700 1 -- 172.21.15.178:0/1941170596 wait complete. 2024-02-11T17:03:12.765 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:12.764+0000 7fbd4df93700 1 Processor -- start 2024-02-11T17:03:12.765 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:12.764+0000 7fbd4df93700 1 -- start start 2024-02-11T17:03:12.765 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:12.764+0000 7fbd4df93700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbd48060220 0x7fbd481066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:03:12.765 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:12.764+0000 7fbd4df93700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd48106bd0 con 0x7fbd48060220 2024-02-11T17:03:12.767 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:12.764+0000 7fbd477fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbd48060220 0x7fbd481066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:03:12.767 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:12.764+0000 7fbd477fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbd48060220 0x7fbd481066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:52362/0 (socket says 172.21.15.178:52362) 2024-02-11T17:03:12.767 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:12.764+0000 7fbd477fe700 1 -- 172.21.15.178:0/102285010 learned_addr learned my addr 172.21.15.178:0/102285010 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:03:12.767 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:12.765+0000 7fbd477fe700 1 -- 172.21.15.178:0/102285010 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd38005040 con 0x7fbd48060220 2024-02-11T17:03:12.767 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:12.765+0000 7fbd477fe700 1 --2- 172.21.15.178:0/102285010 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbd48060220 0x7fbd481066e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fbd38013510 tx=0x7fbd3800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:03:12.767 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:12.765+0000 7fbd3ffff700 1 -- 172.21.15.178:0/102285010 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd38009320 con 0x7fbd48060220 2024-02-11T17:03:12.767 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:12.765+0000 7fbd4df93700 1 -- 172.21.15.178:0/102285010 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd48106dd0 con 0x7fbd48060220 2024-02-11T17:03:12.768 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:12.765+0000 7fbd3ffff700 1 -- 172.21.15.178:0/102285010 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbd38013f00 con 0x7fbd48060220 2024-02-11T17:03:12.768 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:12.765+0000 7fbd3ffff700 1 -- 172.21.15.178:0/102285010 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd38014dc0 con 0x7fbd48060220 2024-02-11T17:03:12.768 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:12.765+0000 7fbd4df93700 1 -- 172.21.15.178:0/102285010 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd48107260 con 0x7fbd48060220 2024-02-11T17:03:12.768 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:12.766+0000 7fbd3ffff700 1 -- 172.21.15.178:0/102285010 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbd38014420 con 0x7fbd48060220 2024-02-11T17:03:12.768 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:12.766+0000 7fbd4df93700 1 -- 172.21.15.178:0/102285010 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd48045ad0 con 0x7fbd48060220 2024-02-11T17:03:12.768 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:12.766+0000 7fbd3ffff700 1 --2- 172.21.15.178:0/102285010 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fbd3402ff60 0x7fbd34032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:03:12.768 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:12.766+0000 7fbd3ffff700 1 -- 172.21.15.178:0/102285010 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbd380410e0 con 0x7fbd48060220 2024-02-11T17:03:12.768 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:12.766+0000 7fbd46ffd700 1 --2- 172.21.15.178:0/102285010 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fbd3402ff60 0x7fbd34032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:03:12.768 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:12.766+0000 7fbd46ffd700 1 --2- 172.21.15.178:0/102285010 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fbd3402ff60 0x7fbd34032420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fbd300096a0 tx=0x7fbd30006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:03:12.771 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:12.771+0000 7fbd3ffff700 1 -- 172.21.15.178:0/102285010 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbd380095f0 con 0x7fbd48060220 2024-02-11T17:03:12.976 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:12.975+0000 7fbd4df93700 1 -- 172.21.15.178:0/102285010 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbd481080b0 con 0x7fbd48060220 2024-02-11T17:03:12.976 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:12.976+0000 7fbd3ffff700 1 -- 172.21.15.178:0/102285010 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbd3801cb10 con 0x7fbd48060220 2024-02-11T17:03:12.976 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:03:12.976 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:03:12.978 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:12.977+0000 7fbd4df93700 1 -- 172.21.15.178:0/102285010 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fbd3402ff60 msgr2=0x7fbd34032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:03:12.978 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:12.977+0000 7fbd4df93700 1 --2- 172.21.15.178:0/102285010 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fbd3402ff60 0x7fbd34032420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7fbd300096a0 tx=0x7fbd30006b40).stop 2024-02-11T17:03:12.978 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:12.977+0000 7fbd4df93700 1 -- 172.21.15.178:0/102285010 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbd48060220 msgr2=0x7fbd481066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:03:12.978 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:12.977+0000 7fbd4df93700 1 --2- 172.21.15.178:0/102285010 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbd48060220 0x7fbd481066e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fbd38013510 tx=0x7fbd3800b040).stop 2024-02-11T17:03:12.978 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:12.977+0000 7fbd4df93700 1 -- 172.21.15.178:0/102285010 shutdown_connections 2024-02-11T17:03:12.978 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:12.977+0000 7fbd4df93700 1 --2- 172.21.15.178:0/102285010 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fbd3402ff60 0x7fbd34032420 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:03:12.978 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:12.977+0000 7fbd4df93700 1 --2- 172.21.15.178:0/102285010 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbd48060220 0x7fbd481066e0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:03:12.978 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:12.977+0000 7fbd4df93700 1 -- 172.21.15.178:0/102285010 >> 172.21.15.178:0/102285010 conn(0x7fbd480f0b70 msgr2=0x7fbd480f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:03:12.978 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:12.977+0000 7fbd4df93700 1 -- 172.21.15.178:0/102285010 shutdown_connections 2024-02-11T17:03:12.978 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:12.977+0000 7fbd4df93700 1 -- 172.21.15.178:0/102285010 wait complete. 2024-02-11T17:03:12.979 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:03:13.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:03:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:03:12.975898+0000 mon.smithi037 (mon.0) 164 : audit [DBG] from='client.? 172.21.15.178:0/102285010' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:03:14.637 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:03:14.638 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:03:14.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:03:14 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:03:13.157165+0000 mgr.smithi037.eofnaq (mgr.14162) 96 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:03:16.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:03:16 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:03:15.157497+0000 mgr.smithi037.eofnaq (mgr.14162) 97 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:03:16.480 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:16.476+0000 7f312bd89700 1 -- 172.21.15.178:0/1866704138 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31240f5410 msgr2=0x7f31240f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:03:16.480 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:16.476+0000 7f312bd89700 1 --2- 172.21.15.178:0/1866704138 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31240f5410 0x7f31240f5830 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f3114009fb0 tx=0x7f3114009b30).stop 2024-02-11T17:03:16.480 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:16.476+0000 7f312bd89700 1 -- 172.21.15.178:0/1866704138 shutdown_connections 2024-02-11T17:03:16.480 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:16.476+0000 7f312bd89700 1 --2- 172.21.15.178:0/1866704138 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31240f5410 0x7f31240f5830 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:03:16.480 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:16.476+0000 7f312bd89700 1 -- 172.21.15.178:0/1866704138 >> 172.21.15.178:0/1866704138 conn(0x7f31240f0b70 msgr2=0x7f31240f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:03:16.480 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:16.477+0000 7f312bd89700 1 -- 172.21.15.178:0/1866704138 shutdown_connections 2024-02-11T17:03:16.480 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:16.477+0000 7f312bd89700 1 -- 172.21.15.178:0/1866704138 wait complete. 2024-02-11T17:03:16.480 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:16.477+0000 7f312bd89700 1 Processor -- start 2024-02-11T17:03:16.480 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:16.477+0000 7f312bd89700 1 -- start start 2024-02-11T17:03:16.480 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:16.477+0000 7f312bd89700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31240f5410 0x7f3124106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:03:16.480 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:16.477+0000 7f312bd89700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3124106c10 con 0x7f31240f5410 2024-02-11T17:03:16.481 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:16.478+0000 7f3129b25700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31240f5410 0x7f3124106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:03:16.481 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:16.478+0000 7f3129b25700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31240f5410 0x7f3124106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:52382/0 (socket says 172.21.15.178:52382) 2024-02-11T17:03:16.481 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:16.478+0000 7f3129b25700 1 -- 172.21.15.178:0/2015958938 learned_addr learned my addr 172.21.15.178:0/2015958938 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:03:16.481 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:16.478+0000 7f3129b25700 1 -- 172.21.15.178:0/2015958938 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3114005040 con 0x7f31240f5410 2024-02-11T17:03:16.481 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:16.478+0000 7f3129b25700 1 --2- 172.21.15.178:0/2015958938 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31240f5410 0x7f3124106720 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f3114014000 tx=0x7f311400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:03:16.481 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:16.478+0000 7f311a7fc700 1 -- 172.21.15.178:0/2015958938 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f31140095f0 con 0x7f31240f5410 2024-02-11T17:03:16.481 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:16.478+0000 7f312bd89700 1 -- 172.21.15.178:0/2015958938 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3124106e10 con 0x7f31240f5410 2024-02-11T17:03:16.481 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:16.478+0000 7f311a7fc700 1 -- 172.21.15.178:0/2015958938 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3114007e70 con 0x7f31240f5410 2024-02-11T17:03:16.481 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:16.478+0000 7f311a7fc700 1 -- 172.21.15.178:0/2015958938 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3114014ea0 con 0x7f31240f5410 2024-02-11T17:03:16.481 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:16.478+0000 7f312bd89700 1 -- 172.21.15.178:0/2015958938 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31241072a0 con 0x7f31240f5410 2024-02-11T17:03:16.482 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:16.479+0000 7f311a7fc700 1 -- 172.21.15.178:0/2015958938 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3114025070 con 0x7f31240f5410 2024-02-11T17:03:16.482 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:16.479+0000 7f312bd89700 1 -- 172.21.15.178:0/2015958938 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3124045ad0 con 0x7f31240f5410 2024-02-11T17:03:16.482 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:16.479+0000 7f311a7fc700 1 --2- 172.21.15.178:0/2015958938 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f311002ffb0 0x7f3110032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:03:16.482 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:16.479+0000 7f311a7fc700 1 -- 172.21.15.178:0/2015958938 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f31140409a0 con 0x7f31240f5410 2024-02-11T17:03:16.482 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:16.479+0000 7f3129324700 1 --2- 172.21.15.178:0/2015958938 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f311002ffb0 0x7f3110032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:03:16.483 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:16.480+0000 7f3129324700 1 --2- 172.21.15.178:0/2015958938 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f311002ffb0 0x7f3110032470 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f31200096a0 tx=0x7f3120006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:03:16.485 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:16.484+0000 7f311a7fc700 1 -- 172.21.15.178:0/2015958938 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f311401d4b0 con 0x7f31240f5410 2024-02-11T17:03:16.691 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:16.689+0000 7f312bd89700 1 -- 172.21.15.178:0/2015958938 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3124108150 con 0x7f31240f5410 2024-02-11T17:03:16.692 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:16.690+0000 7f311a7fc700 1 -- 172.21.15.178:0/2015958938 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f311400f300 con 0x7f31240f5410 2024-02-11T17:03:16.692 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:03:16.692 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:03:16.694 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:16.692+0000 7f312bd89700 1 -- 172.21.15.178:0/2015958938 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f311002ffb0 msgr2=0x7f3110032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:03:16.694 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:16.692+0000 7f312bd89700 1 --2- 172.21.15.178:0/2015958938 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f311002ffb0 0x7f3110032470 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f31200096a0 tx=0x7f3120006b40).stop 2024-02-11T17:03:16.694 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:16.692+0000 7f312bd89700 1 -- 172.21.15.178:0/2015958938 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31240f5410 msgr2=0x7f3124106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:03:16.694 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:16.692+0000 7f312bd89700 1 --2- 172.21.15.178:0/2015958938 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31240f5410 0x7f3124106720 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f3114014000 tx=0x7f311400b040).stop 2024-02-11T17:03:16.694 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:16.692+0000 7f312bd89700 1 -- 172.21.15.178:0/2015958938 shutdown_connections 2024-02-11T17:03:16.694 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:16.692+0000 7f312bd89700 1 --2- 172.21.15.178:0/2015958938 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f311002ffb0 0x7f3110032470 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:03:16.694 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:16.692+0000 7f312bd89700 1 --2- 172.21.15.178:0/2015958938 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31240f5410 0x7f3124106720 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:03:16.695 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:16.692+0000 7f312bd89700 1 -- 172.21.15.178:0/2015958938 >> 172.21.15.178:0/2015958938 conn(0x7f31240f0b70 msgr2=0x7f312405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:03:16.695 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:16.692+0000 7f312bd89700 1 -- 172.21.15.178:0/2015958938 shutdown_connections 2024-02-11T17:03:16.695 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:16.692+0000 7f312bd89700 1 -- 172.21.15.178:0/2015958938 wait complete. 2024-02-11T17:03:16.695 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:03:17.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:03:17 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:03:16.691556+0000 mon.smithi037 (mon.0) 165 : audit [DBG] from='client.? 172.21.15.178:0/2015958938' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:03:18.293 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:03:18.293 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:03:18.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:03:18 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:03:17.157836+0000 mgr.smithi037.eofnaq (mgr.14162) 98 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:03:20.268 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:20.265+0000 7f668759e700 1 -- 172.21.15.178:0/390281017 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f66880f62a0 msgr2=0x7f66880f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:03:20.268 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:20.265+0000 7f668759e700 1 --2- 172.21.15.178:0/390281017 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f66880f62a0 0x7f66880f66c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f6678009fb0 tx=0x7f6678009b30).stop 2024-02-11T17:03:20.268 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:20.265+0000 7f668759e700 1 -- 172.21.15.178:0/390281017 shutdown_connections 2024-02-11T17:03:20.268 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:20.265+0000 7f668759e700 1 --2- 172.21.15.178:0/390281017 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f66880f62a0 0x7f66880f66c0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:03:20.268 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:20.265+0000 7f668759e700 1 -- 172.21.15.178:0/390281017 >> 172.21.15.178:0/390281017 conn(0x7f66880f1aa0 msgr2=0x7f66880f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:03:20.268 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:20.265+0000 7f668759e700 1 -- 172.21.15.178:0/390281017 shutdown_connections 2024-02-11T17:03:20.269 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:20.265+0000 7f668759e700 1 -- 172.21.15.178:0/390281017 wait complete. 2024-02-11T17:03:20.269 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:20.265+0000 7f668759e700 1 Processor -- start 2024-02-11T17:03:20.269 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:20.265+0000 7f668759e700 1 -- start start 2024-02-11T17:03:20.269 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:20.266+0000 7f668759e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f66880f62a0 0x7f66881064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:03:20.269 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:20.266+0000 7f668759e700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66881069a0 con 0x7f66880f62a0 2024-02-11T17:03:20.269 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:20.266+0000 7f668659c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f66880f62a0 0x7f66881064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:03:20.269 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:20.266+0000 7f668659c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f66880f62a0 0x7f66881064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:43076/0 (socket says 172.21.15.178:43076) 2024-02-11T17:03:20.269 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:20.266+0000 7f668659c700 1 -- 172.21.15.178:0/1758496264 learned_addr learned my addr 172.21.15.178:0/1758496264 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:03:20.269 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:20.266+0000 7f668659c700 1 -- 172.21.15.178:0/1758496264 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6678005040 con 0x7f66880f62a0 2024-02-11T17:03:20.269 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:20.266+0000 7f668659c700 1 --2- 172.21.15.178:0/1758496264 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f66880f62a0 0x7f66881064b0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f6678013510 tx=0x7f667800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:03:20.269 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:20.266+0000 7f6676ffd700 1 -- 172.21.15.178:0/1758496264 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6678009320 con 0x7f66880f62a0 2024-02-11T17:03:20.269 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:20.266+0000 7f668759e700 1 -- 172.21.15.178:0/1758496264 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6688106ba0 con 0x7f66880f62a0 2024-02-11T17:03:20.269 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:20.266+0000 7f6676ffd700 1 -- 172.21.15.178:0/1758496264 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6678013f00 con 0x7f66880f62a0 2024-02-11T17:03:20.269 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:20.266+0000 7f6676ffd700 1 -- 172.21.15.178:0/1758496264 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6678014dc0 con 0x7f66880f62a0 2024-02-11T17:03:20.270 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:20.266+0000 7f668759e700 1 -- 172.21.15.178:0/1758496264 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6688107030 con 0x7f66880f62a0 2024-02-11T17:03:20.270 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:20.267+0000 7f6676ffd700 1 -- 172.21.15.178:0/1758496264 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f66780095f0 con 0x7f66880f62a0 2024-02-11T17:03:20.270 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:20.267+0000 7f668759e700 1 -- 172.21.15.178:0/1758496264 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6688045ad0 con 0x7f66880f62a0 2024-02-11T17:03:20.271 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:20.268+0000 7f6676ffd700 1 --2- 172.21.15.178:0/1758496264 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f667002ff60 0x7f6670032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:03:20.271 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:20.268+0000 7f6676ffd700 1 -- 172.21.15.178:0/1758496264 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6678041000 con 0x7f66880f62a0 2024-02-11T17:03:20.271 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:20.268+0000 7f6685d9b700 1 --2- 172.21.15.178:0/1758496264 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f667002ff60 0x7f6670032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:03:20.271 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:20.268+0000 7f6685d9b700 1 --2- 172.21.15.178:0/1758496264 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f667002ff60 0x7f6670032420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f667c0096a0 tx=0x7f667c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:03:20.274 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:20.272+0000 7f6676ffd700 1 -- 172.21.15.178:0/1758496264 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f667801c790 con 0x7f66880f62a0 2024-02-11T17:03:20.478 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:20.476+0000 7f668759e700 1 -- 172.21.15.178:0/1758496264 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6688107ee0 con 0x7f66880f62a0 2024-02-11T17:03:20.478 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:20.477+0000 7f6676ffd700 1 -- 172.21.15.178:0/1758496264 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6678010230 con 0x7f66880f62a0 2024-02-11T17:03:20.479 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:03:20.479 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:03:20.480 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:20.478+0000 7f668759e700 1 -- 172.21.15.178:0/1758496264 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f667002ff60 msgr2=0x7f6670032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:03:20.480 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:20.478+0000 7f668759e700 1 --2- 172.21.15.178:0/1758496264 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f667002ff60 0x7f6670032420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f667c0096a0 tx=0x7f667c006b40).stop 2024-02-11T17:03:20.480 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:20.478+0000 7f668759e700 1 -- 172.21.15.178:0/1758496264 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f66880f62a0 msgr2=0x7f66881064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:03:20.481 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:20.478+0000 7f668759e700 1 --2- 172.21.15.178:0/1758496264 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f66880f62a0 0x7f66881064b0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f6678013510 tx=0x7f667800b040).stop 2024-02-11T17:03:20.481 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:20.479+0000 7f668759e700 1 -- 172.21.15.178:0/1758496264 shutdown_connections 2024-02-11T17:03:20.481 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:20.479+0000 7f668759e700 1 --2- 172.21.15.178:0/1758496264 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f667002ff60 0x7f6670032420 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:03:20.481 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:20.479+0000 7f668759e700 1 --2- 172.21.15.178:0/1758496264 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f66880f62a0 0x7f66881064b0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:03:20.481 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:20.479+0000 7f668759e700 1 -- 172.21.15.178:0/1758496264 >> 172.21.15.178:0/1758496264 conn(0x7f66880f1aa0 msgr2=0x7f66880f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:03:20.481 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:20.479+0000 7f668759e700 1 -- 172.21.15.178:0/1758496264 shutdown_connections 2024-02-11T17:03:20.481 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:20.479+0000 7f668759e700 1 -- 172.21.15.178:0/1758496264 wait complete. 2024-02-11T17:03:20.481 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:03:20.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:03:20 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:03:19.158135+0000 mgr.smithi037.eofnaq (mgr.14162) 99 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:03:21.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:03:21 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:03:20.478225+0000 mon.smithi037 (mon.0) 166 : audit [DBG] from='client.? 172.21.15.178:0/1758496264' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:03:22.176 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:03:22.176 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:03:22.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:03:22 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:03:21.158504+0000 mgr.smithi037.eofnaq (mgr.14162) 100 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:03:24.005 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:24.002+0000 7f8a344cf700 1 -- 172.21.15.178:0/109806198 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a2c0f5410 msgr2=0x7f8a2c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:03:24.006 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:24.002+0000 7f8a344cf700 1 --2- 172.21.15.178:0/109806198 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a2c0f5410 0x7f8a2c0f5830 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f8a1c009fb0 tx=0x7f8a1c009b30).stop 2024-02-11T17:03:24.006 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:24.002+0000 7f8a344cf700 1 -- 172.21.15.178:0/109806198 shutdown_connections 2024-02-11T17:03:24.006 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:24.002+0000 7f8a344cf700 1 --2- 172.21.15.178:0/109806198 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a2c0f5410 0x7f8a2c0f5830 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:03:24.006 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:24.002+0000 7f8a344cf700 1 -- 172.21.15.178:0/109806198 >> 172.21.15.178:0/109806198 conn(0x7f8a2c0f0b70 msgr2=0x7f8a2c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:03:24.006 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:24.002+0000 7f8a344cf700 1 -- 172.21.15.178:0/109806198 shutdown_connections 2024-02-11T17:03:24.006 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:24.002+0000 7f8a344cf700 1 -- 172.21.15.178:0/109806198 wait complete. 2024-02-11T17:03:24.006 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:24.003+0000 7f8a344cf700 1 Processor -- start 2024-02-11T17:03:24.006 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:24.003+0000 7f8a344cf700 1 -- start start 2024-02-11T17:03:24.006 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:24.003+0000 7f8a344cf700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a2c0f5410 0x7f8a2c109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:03:24.006 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:24.003+0000 7f8a344cf700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a2c060220 con 0x7f8a2c0f5410 2024-02-11T17:03:24.006 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:24.003+0000 7f8a3226b700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a2c0f5410 0x7f8a2c109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:03:24.006 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:24.003+0000 7f8a3226b700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a2c0f5410 0x7f8a2c109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:43100/0 (socket says 172.21.15.178:43100) 2024-02-11T17:03:24.007 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:24.003+0000 7f8a3226b700 1 -- 172.21.15.178:0/2378533508 learned_addr learned my addr 172.21.15.178:0/2378533508 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:03:24.007 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:24.004+0000 7f8a3226b700 1 -- 172.21.15.178:0/2378533508 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a1c005040 con 0x7f8a2c0f5410 2024-02-11T17:03:24.007 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:24.004+0000 7f8a3226b700 1 --2- 172.21.15.178:0/2378533508 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a2c0f5410 0x7f8a2c109450 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f8a1c014000 tx=0x7f8a1c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:03:24.007 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:24.004+0000 7f8a22ffd700 1 -- 172.21.15.178:0/2378533508 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a1c0095f0 con 0x7f8a2c0f5410 2024-02-11T17:03:24.007 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:24.004+0000 7f8a344cf700 1 -- 172.21.15.178:0/2378533508 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a2c060420 con 0x7f8a2c0f5410 2024-02-11T17:03:24.007 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:24.004+0000 7f8a22ffd700 1 -- 172.21.15.178:0/2378533508 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8a1c007e70 con 0x7f8a2c0f5410 2024-02-11T17:03:24.007 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:24.004+0000 7f8a22ffd700 1 -- 172.21.15.178:0/2378533508 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a1c014ea0 con 0x7f8a2c0f5410 2024-02-11T17:03:24.007 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:24.004+0000 7f8a344cf700 1 -- 172.21.15.178:0/2378533508 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a2c109b00 con 0x7f8a2c0f5410 2024-02-11T17:03:24.008 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:24.005+0000 7f8a22ffd700 1 -- 172.21.15.178:0/2378533508 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8a1c025070 con 0x7f8a2c0f5410 2024-02-11T17:03:24.008 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:24.005+0000 7f8a22ffd700 1 --2- 172.21.15.178:0/2378533508 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f8a1802ff60 0x7f8a18032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:03:24.008 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:24.005+0000 7f8a344cf700 1 -- 172.21.15.178:0/2378533508 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a2c059100 con 0x7f8a2c0f5410 2024-02-11T17:03:24.008 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:24.005+0000 7f8a22ffd700 1 -- 172.21.15.178:0/2378533508 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8a1c040970 con 0x7f8a2c0f5410 2024-02-11T17:03:24.008 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:24.005+0000 7f8a31a6a700 1 --2- 172.21.15.178:0/2378533508 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f8a1802ff60 0x7f8a18032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:03:24.008 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:24.006+0000 7f8a31a6a700 1 --2- 172.21.15.178:0/2378533508 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f8a1802ff60 0x7f8a18032420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f8a280096a0 tx=0x7f8a28006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:03:24.011 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:24.010+0000 7f8a22ffd700 1 -- 172.21.15.178:0/2378533508 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8a1c00f710 con 0x7f8a2c0f5410 2024-02-11T17:03:24.216 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:24.214+0000 7f8a344cf700 1 -- 172.21.15.178:0/2378533508 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8a2c045ad0 con 0x7f8a2c0f5410 2024-02-11T17:03:24.216 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:24.215+0000 7f8a22ffd700 1 -- 172.21.15.178:0/2378533508 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8a1c025300 con 0x7f8a2c0f5410 2024-02-11T17:03:24.216 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:03:24.216 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:03:24.218 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:24.216+0000 7f8a344cf700 1 -- 172.21.15.178:0/2378533508 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f8a1802ff60 msgr2=0x7f8a18032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:03:24.218 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:24.216+0000 7f8a344cf700 1 --2- 172.21.15.178:0/2378533508 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f8a1802ff60 0x7f8a18032420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f8a280096a0 tx=0x7f8a28006b40).stop 2024-02-11T17:03:24.218 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:24.216+0000 7f8a344cf700 1 -- 172.21.15.178:0/2378533508 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a2c0f5410 msgr2=0x7f8a2c109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:03:24.218 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:24.216+0000 7f8a344cf700 1 --2- 172.21.15.178:0/2378533508 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a2c0f5410 0x7f8a2c109450 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f8a1c014000 tx=0x7f8a1c00b040).stop 2024-02-11T17:03:24.218 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:24.216+0000 7f8a344cf700 1 -- 172.21.15.178:0/2378533508 shutdown_connections 2024-02-11T17:03:24.218 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:24.216+0000 7f8a344cf700 1 --2- 172.21.15.178:0/2378533508 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f8a1802ff60 0x7f8a18032420 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:03:24.218 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:24.216+0000 7f8a344cf700 1 --2- 172.21.15.178:0/2378533508 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a2c0f5410 0x7f8a2c109450 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:03:24.218 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:24.216+0000 7f8a344cf700 1 -- 172.21.15.178:0/2378533508 >> 172.21.15.178:0/2378533508 conn(0x7f8a2c0f0b70 msgr2=0x7f8a2c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:03:24.218 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:24.216+0000 7f8a344cf700 1 -- 172.21.15.178:0/2378533508 shutdown_connections 2024-02-11T17:03:24.219 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:24.216+0000 7f8a344cf700 1 -- 172.21.15.178:0/2378533508 wait complete. 2024-02-11T17:03:24.219 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:03:24.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:03:24 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:03:23.158804+0000 mgr.smithi037.eofnaq (mgr.14162) 101 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:03:24.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:03:24 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:03:24.216059+0000 mon.smithi037 (mon.0) 167 : audit [DBG] from='client.? 172.21.15.178:0/2378533508' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:03:25.878 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:03:25.879 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:03:26.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:03:26 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:03:25.159096+0000 mgr.smithi037.eofnaq (mgr.14162) 102 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:03:27.815 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:27.813+0000 7f9ec69c9700 1 -- 172.21.15.178:0/1775963684 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9ec00f51a0 msgr2=0x7f9ec00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:03:27.815 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:27.813+0000 7f9ec69c9700 1 --2- 172.21.15.178:0/1775963684 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9ec00f51a0 0x7f9ec00f55c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f9ebc009fb0 tx=0x7f9ebc009b30).stop 2024-02-11T17:03:27.815 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:27.813+0000 7f9ec69c9700 1 -- 172.21.15.178:0/1775963684 shutdown_connections 2024-02-11T17:03:27.815 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:27.813+0000 7f9ec69c9700 1 --2- 172.21.15.178:0/1775963684 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9ec00f51a0 0x7f9ec00f55c0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:03:27.816 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:27.813+0000 7f9ec69c9700 1 -- 172.21.15.178:0/1775963684 >> 172.21.15.178:0/1775963684 conn(0x7f9ec00f09a0 msgr2=0x7f9ec00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:03:27.816 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:27.813+0000 7f9ec69c9700 1 -- 172.21.15.178:0/1775963684 shutdown_connections 2024-02-11T17:03:27.816 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:27.813+0000 7f9ec69c9700 1 -- 172.21.15.178:0/1775963684 wait complete. 2024-02-11T17:03:27.816 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:27.813+0000 7f9ec69c9700 1 Processor -- start 2024-02-11T17:03:27.816 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:27.813+0000 7f9ec69c9700 1 -- start start 2024-02-11T17:03:27.816 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:27.813+0000 7f9ec69c9700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9ec00f51a0 0x7f9ec01064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:03:27.816 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:27.814+0000 7f9ec69c9700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec01069d0 con 0x7f9ec00f51a0 2024-02-11T17:03:27.816 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:27.814+0000 7f9ec59c7700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9ec00f51a0 0x7f9ec01064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:03:27.816 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:27.814+0000 7f9ec59c7700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9ec00f51a0 0x7f9ec01064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:43108/0 (socket says 172.21.15.178:43108) 2024-02-11T17:03:27.816 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:27.814+0000 7f9ec59c7700 1 -- 172.21.15.178:0/4165362394 learned_addr learned my addr 172.21.15.178:0/4165362394 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:03:27.818 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:27.814+0000 7f9ec59c7700 1 -- 172.21.15.178:0/4165362394 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ebc005040 con 0x7f9ec00f51a0 2024-02-11T17:03:27.818 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:27.814+0000 7f9ec59c7700 1 --2- 172.21.15.178:0/4165362394 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9ec00f51a0 0x7f9ec01064e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f9ebc013510 tx=0x7f9ebc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:03:27.818 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:27.814+0000 7f9eb67fc700 1 -- 172.21.15.178:0/4165362394 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ebc009320 con 0x7f9ec00f51a0 2024-02-11T17:03:27.818 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:27.814+0000 7f9eb67fc700 1 -- 172.21.15.178:0/4165362394 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9ebc013f00 con 0x7f9ec00f51a0 2024-02-11T17:03:27.818 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:27.814+0000 7f9ec69c9700 1 -- 172.21.15.178:0/4165362394 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ec0106bd0 con 0x7f9ec00f51a0 2024-02-11T17:03:27.818 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:27.814+0000 7f9eb67fc700 1 -- 172.21.15.178:0/4165362394 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ebc014dc0 con 0x7f9ec00f51a0 2024-02-11T17:03:27.818 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:27.814+0000 7f9ec69c9700 1 -- 172.21.15.178:0/4165362394 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ec0107060 con 0x7f9ec00f51a0 2024-02-11T17:03:27.818 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:27.815+0000 7f9eb67fc700 1 -- 172.21.15.178:0/4165362394 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9ebc0095f0 con 0x7f9ec00f51a0 2024-02-11T17:03:27.818 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:27.815+0000 7f9ec69c9700 1 -- 172.21.15.178:0/4165362394 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ec0045ad0 con 0x7f9ec00f51a0 2024-02-11T17:03:27.818 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:27.815+0000 7f9eb67fc700 1 --2- 172.21.15.178:0/4165362394 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f9eac02ff60 0x7f9eac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:03:27.818 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:27.815+0000 7f9eb67fc700 1 -- 172.21.15.178:0/4165362394 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9ebc0410c0 con 0x7f9ec00f51a0 2024-02-11T17:03:27.818 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:27.816+0000 7f9ec51c6700 1 --2- 172.21.15.178:0/4165362394 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f9eac02ff60 0x7f9eac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:03:27.818 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:27.816+0000 7f9ec51c6700 1 --2- 172.21.15.178:0/4165362394 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f9eac02ff60 0x7f9eac032420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f9eb00096a0 tx=0x7f9eb0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:03:27.822 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:27.820+0000 7f9eb67fc700 1 -- 172.21.15.178:0/4165362394 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9ebc014420 con 0x7f9ec00f51a0 2024-02-11T17:03:28.023 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:28.021+0000 7f9ec69c9700 1 -- 172.21.15.178:0/4165362394 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9ec0107f10 con 0x7f9ec00f51a0 2024-02-11T17:03:28.023 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:28.022+0000 7f9eb67fc700 1 -- 172.21.15.178:0/4165362394 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9ebc010230 con 0x7f9ec00f51a0 2024-02-11T17:03:28.024 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:03:28.024 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:03:28.025 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:28.023+0000 7f9ec69c9700 1 -- 172.21.15.178:0/4165362394 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f9eac02ff60 msgr2=0x7f9eac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:03:28.025 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:28.023+0000 7f9ec69c9700 1 --2- 172.21.15.178:0/4165362394 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f9eac02ff60 0x7f9eac032420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f9eb00096a0 tx=0x7f9eb0006b40).stop 2024-02-11T17:03:28.025 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:28.023+0000 7f9ec69c9700 1 -- 172.21.15.178:0/4165362394 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9ec00f51a0 msgr2=0x7f9ec01064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:03:28.025 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:28.023+0000 7f9ec69c9700 1 --2- 172.21.15.178:0/4165362394 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9ec00f51a0 0x7f9ec01064e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f9ebc013510 tx=0x7f9ebc00b040).stop 2024-02-11T17:03:28.025 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:28.023+0000 7f9ec69c9700 1 -- 172.21.15.178:0/4165362394 shutdown_connections 2024-02-11T17:03:28.026 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:28.023+0000 7f9ec69c9700 1 --2- 172.21.15.178:0/4165362394 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f9eac02ff60 0x7f9eac032420 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:03:28.026 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:28.023+0000 7f9ec69c9700 1 --2- 172.21.15.178:0/4165362394 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9ec00f51a0 0x7f9ec01064e0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:03:28.026 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:28.023+0000 7f9ec69c9700 1 -- 172.21.15.178:0/4165362394 >> 172.21.15.178:0/4165362394 conn(0x7f9ec00f09a0 msgr2=0x7f9ec005fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:03:28.026 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:28.023+0000 7f9ec69c9700 1 -- 172.21.15.178:0/4165362394 shutdown_connections 2024-02-11T17:03:28.026 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:28.023+0000 7f9ec69c9700 1 -- 172.21.15.178:0/4165362394 wait complete. 2024-02-11T17:03:28.026 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:03:28.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:03:28 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:03:27.159413+0000 mgr.smithi037.eofnaq (mgr.14162) 103 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:03:28.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:03:28 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:03:28.023103+0000 mon.smithi037 (mon.0) 168 : audit [DBG] from='client.? 172.21.15.178:0/4165362394' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:03:29.679 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:03:29.681 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:03:30.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:03:30 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:03:29.159786+0000 mgr.smithi037.eofnaq (mgr.14162) 104 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:03:31.567 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:31.565+0000 7f9a9759e700 1 -- 172.21.15.178:0/3562963338 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9a980f62a0 msgr2=0x7f9a980f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:03:31.567 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:31.565+0000 7f9a9759e700 1 --2- 172.21.15.178:0/3562963338 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9a980f62a0 0x7f9a980f66c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f9a8c009fb0 tx=0x7f9a8c009b30).stop 2024-02-11T17:03:31.567 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:31.565+0000 7f9a9759e700 1 -- 172.21.15.178:0/3562963338 shutdown_connections 2024-02-11T17:03:31.568 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:31.565+0000 7f9a9759e700 1 --2- 172.21.15.178:0/3562963338 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9a980f62a0 0x7f9a980f66c0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:03:31.568 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:31.565+0000 7f9a9759e700 1 -- 172.21.15.178:0/3562963338 >> 172.21.15.178:0/3562963338 conn(0x7f9a980f1aa0 msgr2=0x7f9a980f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:03:31.568 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:31.565+0000 7f9a9759e700 1 -- 172.21.15.178:0/3562963338 shutdown_connections 2024-02-11T17:03:31.568 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:31.565+0000 7f9a9759e700 1 -- 172.21.15.178:0/3562963338 wait complete. 2024-02-11T17:03:31.568 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:31.565+0000 7f9a9759e700 1 Processor -- start 2024-02-11T17:03:31.568 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:31.565+0000 7f9a9759e700 1 -- start start 2024-02-11T17:03:31.568 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:31.565+0000 7f9a9759e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9a980f62a0 0x7f9a9806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:03:31.568 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:31.565+0000 7f9a9759e700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a98069e20 con 0x7f9a980f62a0 2024-02-11T17:03:31.568 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:31.566+0000 7f9a9659c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9a980f62a0 0x7f9a9806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:03:31.568 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:31.566+0000 7f9a9659c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9a980f62a0 0x7f9a9806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:58848/0 (socket says 172.21.15.178:58848) 2024-02-11T17:03:31.568 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:31.566+0000 7f9a9659c700 1 -- 172.21.15.178:0/4056373104 learned_addr learned my addr 172.21.15.178:0/4056373104 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:03:31.570 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:31.566+0000 7f9a9659c700 1 -- 172.21.15.178:0/4056373104 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a8c005040 con 0x7f9a980f62a0 2024-02-11T17:03:31.570 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:31.566+0000 7f9a9659c700 1 --2- 172.21.15.178:0/4056373104 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9a980f62a0 0x7f9a9806b9a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f9a8c013510 tx=0x7f9a8c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:03:31.570 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:31.566+0000 7f9a86ffd700 1 -- 172.21.15.178:0/4056373104 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a8c009320 con 0x7f9a980f62a0 2024-02-11T17:03:31.570 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:31.566+0000 7f9a9759e700 1 -- 172.21.15.178:0/4056373104 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a98069fc0 con 0x7f9a980f62a0 2024-02-11T17:03:31.570 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:31.566+0000 7f9a86ffd700 1 -- 172.21.15.178:0/4056373104 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9a8c013f00 con 0x7f9a980f62a0 2024-02-11T17:03:31.570 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:31.566+0000 7f9a86ffd700 1 -- 172.21.15.178:0/4056373104 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a8c014dc0 con 0x7f9a980f62a0 2024-02-11T17:03:31.570 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:31.566+0000 7f9a9759e700 1 -- 172.21.15.178:0/4056373104 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a9806a450 con 0x7f9a980f62a0 2024-02-11T17:03:31.570 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:31.567+0000 7f9a86ffd700 1 -- 172.21.15.178:0/4056373104 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9a8c014420 con 0x7f9a980f62a0 2024-02-11T17:03:31.570 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:31.567+0000 7f9a9759e700 1 -- 172.21.15.178:0/4056373104 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a98059100 con 0x7f9a980f62a0 2024-02-11T17:03:31.570 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:31.567+0000 7f9a86ffd700 1 --2- 172.21.15.178:0/4056373104 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f9a8002ff60 0x7f9a80032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:03:31.571 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:31.567+0000 7f9a86ffd700 1 -- 172.21.15.178:0/4056373104 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9a8c041110 con 0x7f9a980f62a0 2024-02-11T17:03:31.571 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:31.568+0000 7f9a95d9b700 1 --2- 172.21.15.178:0/4056373104 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f9a8002ff60 0x7f9a80032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:03:31.571 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:31.568+0000 7f9a95d9b700 1 --2- 172.21.15.178:0/4056373104 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f9a8002ff60 0x7f9a80032420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f9a880096a0 tx=0x7f9a88006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:03:31.574 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:31.572+0000 7f9a86ffd700 1 -- 172.21.15.178:0/4056373104 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9a8c0095f0 con 0x7f9a980f62a0 2024-02-11T17:03:31.780 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:31.778+0000 7f9a9759e700 1 -- 172.21.15.178:0/4056373104 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9a980fbce0 con 0x7f9a980f62a0 2024-02-11T17:03:31.780 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:31.779+0000 7f9a86ffd700 1 -- 172.21.15.178:0/4056373104 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9a8c01cb10 con 0x7f9a980f62a0 2024-02-11T17:03:31.780 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:03:31.781 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:03:31.782 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:31.780+0000 7f9a9759e700 1 -- 172.21.15.178:0/4056373104 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f9a8002ff60 msgr2=0x7f9a80032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:03:31.782 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:31.780+0000 7f9a9759e700 1 --2- 172.21.15.178:0/4056373104 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f9a8002ff60 0x7f9a80032420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f9a880096a0 tx=0x7f9a88006b40).stop 2024-02-11T17:03:31.782 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:31.780+0000 7f9a9759e700 1 -- 172.21.15.178:0/4056373104 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9a980f62a0 msgr2=0x7f9a9806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:03:31.783 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:31.780+0000 7f9a9759e700 1 --2- 172.21.15.178:0/4056373104 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9a980f62a0 0x7f9a9806b9a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f9a8c013510 tx=0x7f9a8c00b040).stop 2024-02-11T17:03:31.783 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:31.780+0000 7f9a9759e700 1 -- 172.21.15.178:0/4056373104 shutdown_connections 2024-02-11T17:03:31.783 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:31.780+0000 7f9a9759e700 1 --2- 172.21.15.178:0/4056373104 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f9a8002ff60 0x7f9a80032420 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:03:31.783 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:31.780+0000 7f9a9759e700 1 --2- 172.21.15.178:0/4056373104 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9a980f62a0 0x7f9a9806b9a0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:03:31.783 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:31.780+0000 7f9a9759e700 1 -- 172.21.15.178:0/4056373104 >> 172.21.15.178:0/4056373104 conn(0x7f9a980f1aa0 msgr2=0x7f9a980f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:03:31.783 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:31.780+0000 7f9a9759e700 1 -- 172.21.15.178:0/4056373104 shutdown_connections 2024-02-11T17:03:31.783 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:31.780+0000 7f9a9759e700 1 -- 172.21.15.178:0/4056373104 wait complete. 2024-02-11T17:03:31.783 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:03:32.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:03:32 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:03:31.160097+0000 mgr.smithi037.eofnaq (mgr.14162) 105 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:03:32.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:03:32 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:03:31.780030+0000 mon.smithi037 (mon.0) 169 : audit [DBG] from='client.? 172.21.15.178:0/4056373104' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:03:33.482 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:03:33.482 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:03:34.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:03:34 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:03:33.160373+0000 mgr.smithi037.eofnaq (mgr.14162) 106 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:03:35.295 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:35.292+0000 7f0c2f1fa700 1 -- 172.21.15.178:0/2232941983 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0c280f62c0 msgr2=0x7f0c280f66e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:03:35.295 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:35.292+0000 7f0c2f1fa700 1 --2- 172.21.15.178:0/2232941983 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0c280f62c0 0x7f0c280f66e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f0c24009fb0 tx=0x7f0c24009b30).stop 2024-02-11T17:03:35.295 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:35.292+0000 7f0c2f1fa700 1 -- 172.21.15.178:0/2232941983 shutdown_connections 2024-02-11T17:03:35.295 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:35.292+0000 7f0c2f1fa700 1 --2- 172.21.15.178:0/2232941983 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0c280f62c0 0x7f0c280f66e0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:03:35.295 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:35.292+0000 7f0c2f1fa700 1 -- 172.21.15.178:0/2232941983 >> 172.21.15.178:0/2232941983 conn(0x7f0c280f1ac0 msgr2=0x7f0c280f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:03:35.296 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:35.292+0000 7f0c2f1fa700 1 -- 172.21.15.178:0/2232941983 shutdown_connections 2024-02-11T17:03:35.296 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:35.292+0000 7f0c2f1fa700 1 -- 172.21.15.178:0/2232941983 wait complete. 2024-02-11T17:03:35.296 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:35.292+0000 7f0c2f1fa700 1 Processor -- start 2024-02-11T17:03:35.296 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:35.292+0000 7f0c2f1fa700 1 -- start start 2024-02-11T17:03:35.296 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:35.293+0000 7f0c2f1fa700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0c280f62c0 0x7f0c28106520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:03:35.296 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:35.293+0000 7f0c2f1fa700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c28106a10 con 0x7f0c280f62c0 2024-02-11T17:03:35.296 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:35.293+0000 7f0c2e1f8700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0c280f62c0 0x7f0c28106520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:03:35.296 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:35.293+0000 7f0c2e1f8700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0c280f62c0 0x7f0c28106520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:58872/0 (socket says 172.21.15.178:58872) 2024-02-11T17:03:35.296 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:35.293+0000 7f0c2e1f8700 1 -- 172.21.15.178:0/4142618143 learned_addr learned my addr 172.21.15.178:0/4142618143 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:03:35.296 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:35.293+0000 7f0c2e1f8700 1 -- 172.21.15.178:0/4142618143 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c24005040 con 0x7f0c280f62c0 2024-02-11T17:03:35.296 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:35.293+0000 7f0c2e1f8700 1 --2- 172.21.15.178:0/4142618143 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0c280f62c0 0x7f0c28106520 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f0c24013510 tx=0x7f0c2400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:03:35.296 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:35.293+0000 7f0c1effd700 1 -- 172.21.15.178:0/4142618143 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c24009320 con 0x7f0c280f62c0 2024-02-11T17:03:35.296 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:35.294+0000 7f0c2f1fa700 1 -- 172.21.15.178:0/4142618143 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c28106c10 con 0x7f0c280f62c0 2024-02-11T17:03:35.296 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:35.294+0000 7f0c1effd700 1 -- 172.21.15.178:0/4142618143 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0c24013f00 con 0x7f0c280f62c0 2024-02-11T17:03:35.297 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:35.294+0000 7f0c1effd700 1 -- 172.21.15.178:0/4142618143 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c24014dc0 con 0x7f0c280f62c0 2024-02-11T17:03:35.297 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:35.294+0000 7f0c2f1fa700 1 -- 172.21.15.178:0/4142618143 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c281070a0 con 0x7f0c280f62c0 2024-02-11T17:03:35.298 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:35.294+0000 7f0c1effd700 1 -- 172.21.15.178:0/4142618143 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0c240095f0 con 0x7f0c280f62c0 2024-02-11T17:03:35.298 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:35.294+0000 7f0c2f1fa700 1 -- 172.21.15.178:0/4142618143 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c28045ad0 con 0x7f0c280f62c0 2024-02-11T17:03:35.298 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:35.295+0000 7f0c1effd700 1 --2- 172.21.15.178:0/4142618143 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f0c1402ff60 0x7f0c14032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:03:35.298 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:35.295+0000 7f0c1effd700 1 -- 172.21.15.178:0/4142618143 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0c24041100 con 0x7f0c280f62c0 2024-02-11T17:03:35.298 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:35.295+0000 7f0c2d9f7700 1 --2- 172.21.15.178:0/4142618143 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f0c1402ff60 0x7f0c14032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:03:35.298 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:35.295+0000 7f0c2d9f7700 1 --2- 172.21.15.178:0/4142618143 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f0c1402ff60 0x7f0c14032420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f0c180096a0 tx=0x7f0c18006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:03:35.301 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:35.299+0000 7f0c1effd700 1 -- 172.21.15.178:0/4142618143 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0c24014420 con 0x7f0c280f62c0 2024-02-11T17:03:35.506 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:35.505+0000 7f0c2f1fa700 1 -- 172.21.15.178:0/4142618143 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0c28107f50 con 0x7f0c280f62c0 2024-02-11T17:03:35.507 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:35.506+0000 7f0c1effd700 1 -- 172.21.15.178:0/4142618143 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0c24010230 con 0x7f0c280f62c0 2024-02-11T17:03:35.507 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:03:35.507 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:03:35.509 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:35.507+0000 7f0c2f1fa700 1 -- 172.21.15.178:0/4142618143 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f0c1402ff60 msgr2=0x7f0c14032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:03:35.509 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:35.507+0000 7f0c2f1fa700 1 --2- 172.21.15.178:0/4142618143 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f0c1402ff60 0x7f0c14032420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f0c180096a0 tx=0x7f0c18006b40).stop 2024-02-11T17:03:35.509 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:35.507+0000 7f0c2f1fa700 1 -- 172.21.15.178:0/4142618143 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0c280f62c0 msgr2=0x7f0c28106520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:03:35.510 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:35.507+0000 7f0c2f1fa700 1 --2- 172.21.15.178:0/4142618143 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0c280f62c0 0x7f0c28106520 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f0c24013510 tx=0x7f0c2400b040).stop 2024-02-11T17:03:35.510 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:35.507+0000 7f0c2f1fa700 1 -- 172.21.15.178:0/4142618143 shutdown_connections 2024-02-11T17:03:35.510 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:35.507+0000 7f0c2f1fa700 1 --2- 172.21.15.178:0/4142618143 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f0c1402ff60 0x7f0c14032420 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:03:35.510 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:35.507+0000 7f0c2f1fa700 1 --2- 172.21.15.178:0/4142618143 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0c280f62c0 0x7f0c28106520 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:03:35.510 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:35.507+0000 7f0c2f1fa700 1 -- 172.21.15.178:0/4142618143 >> 172.21.15.178:0/4142618143 conn(0x7f0c280f1ac0 msgr2=0x7f0c280f2530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:03:35.510 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:35.507+0000 7f0c2f1fa700 1 -- 172.21.15.178:0/4142618143 shutdown_connections 2024-02-11T17:03:35.510 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:35.507+0000 7f0c2f1fa700 1 -- 172.21.15.178:0/4142618143 wait complete. 2024-02-11T17:03:35.510 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:03:36.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:03:36 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:03:35.160679+0000 mgr.smithi037.eofnaq (mgr.14162) 107 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:03:36.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:03:36 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:03:35.506951+0000 mon.smithi037 (mon.0) 170 : audit [DBG] from='client.? 172.21.15.178:0/4142618143' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:03:37.135 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:03:37.135 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:03:38.684 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:03:38 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:03:37.161055+0000 mgr.smithi037.eofnaq (mgr.14162) 108 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:03:39.070 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:39.068+0000 7fce85e75700 1 -- 172.21.15.178:0/634863932 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce800f51a0 msgr2=0x7fce800f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:03:39.070 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:39.068+0000 7fce85e75700 1 --2- 172.21.15.178:0/634863932 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce800f51a0 0x7fce800f55c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fce74009fb0 tx=0x7fce74009b30).stop 2024-02-11T17:03:39.070 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:39.068+0000 7fce85e75700 1 -- 172.21.15.178:0/634863932 shutdown_connections 2024-02-11T17:03:39.070 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:39.068+0000 7fce85e75700 1 --2- 172.21.15.178:0/634863932 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce800f51a0 0x7fce800f55c0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:03:39.071 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:39.068+0000 7fce85e75700 1 -- 172.21.15.178:0/634863932 >> 172.21.15.178:0/634863932 conn(0x7fce800f09a0 msgr2=0x7fce800f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:03:39.071 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:39.068+0000 7fce85e75700 1 -- 172.21.15.178:0/634863932 shutdown_connections 2024-02-11T17:03:39.071 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:39.068+0000 7fce85e75700 1 -- 172.21.15.178:0/634863932 wait complete. 2024-02-11T17:03:39.071 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:39.068+0000 7fce85e75700 1 Processor -- start 2024-02-11T17:03:39.071 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:39.068+0000 7fce85e75700 1 -- start start 2024-02-11T17:03:39.071 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:39.069+0000 7fce85e75700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce800f51a0 0x7fce80109170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:03:39.071 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:39.069+0000 7fce85e75700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce80060280 con 0x7fce800f51a0 2024-02-11T17:03:39.073 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:39.069+0000 7fce84e73700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce800f51a0 0x7fce80109170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:03:39.073 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:39.069+0000 7fce84e73700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce800f51a0 0x7fce80109170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:58890/0 (socket says 172.21.15.178:58890) 2024-02-11T17:03:39.073 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:39.069+0000 7fce84e73700 1 -- 172.21.15.178:0/2910342292 learned_addr learned my addr 172.21.15.178:0/2910342292 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:03:39.073 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:39.069+0000 7fce84e73700 1 -- 172.21.15.178:0/2910342292 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce74005040 con 0x7fce800f51a0 2024-02-11T17:03:39.073 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:39.069+0000 7fce84e73700 1 --2- 172.21.15.178:0/2910342292 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce800f51a0 0x7fce80109170 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fce74013510 tx=0x7fce7400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:03:39.073 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:39.070+0000 7fce7d7fa700 1 -- 172.21.15.178:0/2910342292 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce74009320 con 0x7fce800f51a0 2024-02-11T17:03:39.073 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:39.070+0000 7fce85e75700 1 -- 172.21.15.178:0/2910342292 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce80060480 con 0x7fce800f51a0 2024-02-11T17:03:39.073 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:39.070+0000 7fce85e75700 1 -- 172.21.15.178:0/2910342292 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce801098a0 con 0x7fce800f51a0 2024-02-11T17:03:39.073 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:39.070+0000 7fce7d7fa700 1 -- 172.21.15.178:0/2910342292 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fce74013f00 con 0x7fce800f51a0 2024-02-11T17:03:39.073 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:39.070+0000 7fce7d7fa700 1 -- 172.21.15.178:0/2910342292 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce74014d90 con 0x7fce800f51a0 2024-02-11T17:03:39.074 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:39.071+0000 7fce7d7fa700 1 -- 172.21.15.178:0/2910342292 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fce740263a0 con 0x7fce800f51a0 2024-02-11T17:03:39.074 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:39.071+0000 7fce85e75700 1 -- 172.21.15.178:0/2910342292 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce60004fa0 con 0x7fce800f51a0 2024-02-11T17:03:39.074 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:39.071+0000 7fce7d7fa700 1 --2- 172.21.15.178:0/2910342292 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fce7002ff60 0x7fce70032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:03:39.074 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:39.071+0000 7fce7d7fa700 1 -- 172.21.15.178:0/2910342292 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fce74040f60 con 0x7fce800f51a0 2024-02-11T17:03:39.074 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:39.071+0000 7fce7ffff700 1 --2- 172.21.15.178:0/2910342292 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fce7002ff60 0x7fce70032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:03:39.074 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:39.071+0000 7fce7ffff700 1 --2- 172.21.15.178:0/2910342292 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fce7002ff60 0x7fce70032420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fce6c0096a0 tx=0x7fce6c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:03:39.077 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:39.075+0000 7fce7d7fa700 1 -- 172.21.15.178:0/2910342292 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fce7406d2b0 con 0x7fce800f51a0 2024-02-11T17:03:39.282 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:39.280+0000 7fce85e75700 1 -- 172.21.15.178:0/2910342292 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fce60005670 con 0x7fce800f51a0 2024-02-11T17:03:39.283 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:39.281+0000 7fce7d7fa700 1 -- 172.21.15.178:0/2910342292 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fce74012740 con 0x7fce800f51a0 2024-02-11T17:03:39.283 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:03:39.283 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:03:39.284 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:39.283+0000 7fce85e75700 1 -- 172.21.15.178:0/2910342292 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fce7002ff60 msgr2=0x7fce70032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:03:39.284 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:39.283+0000 7fce85e75700 1 --2- 172.21.15.178:0/2910342292 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fce7002ff60 0x7fce70032420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fce6c0096a0 tx=0x7fce6c006b40).stop 2024-02-11T17:03:39.285 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:39.283+0000 7fce85e75700 1 -- 172.21.15.178:0/2910342292 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce800f51a0 msgr2=0x7fce80109170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:03:39.285 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:39.283+0000 7fce85e75700 1 --2- 172.21.15.178:0/2910342292 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce800f51a0 0x7fce80109170 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fce74013510 tx=0x7fce7400b040).stop 2024-02-11T17:03:39.285 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:39.283+0000 7fce85e75700 1 -- 172.21.15.178:0/2910342292 shutdown_connections 2024-02-11T17:03:39.285 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:39.283+0000 7fce85e75700 1 --2- 172.21.15.178:0/2910342292 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fce7002ff60 0x7fce70032420 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:03:39.285 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:39.283+0000 7fce85e75700 1 --2- 172.21.15.178:0/2910342292 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce800f51a0 0x7fce80109170 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:03:39.285 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:39.283+0000 7fce85e75700 1 -- 172.21.15.178:0/2910342292 >> 172.21.15.178:0/2910342292 conn(0x7fce800f09a0 msgr2=0x7fce800f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:03:39.285 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:39.283+0000 7fce85e75700 1 -- 172.21.15.178:0/2910342292 shutdown_connections 2024-02-11T17:03:39.285 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:39.283+0000 7fce85e75700 1 -- 172.21.15.178:0/2910342292 wait complete. 2024-02-11T17:03:39.285 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:03:40.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:03:40 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:03:39.161400+0000 mgr.smithi037.eofnaq (mgr.14162) 109 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:03:40.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:03:40 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:03:39.282578+0000 mon.smithi037 (mon.0) 171 : audit [DBG] from='client.? 172.21.15.178:0/2910342292' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:03:40.985 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:03:40.985 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:03:42.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:03:42 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:03:41.161741+0000 mgr.smithi037.eofnaq (mgr.14162) 110 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:03:42.880 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:42.878+0000 7f1978ab6700 1 -- 172.21.15.178:0/642406805 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f19740f5410 msgr2=0x7f19740f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:03:42.880 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:42.878+0000 7f1978ab6700 1 --2- 172.21.15.178:0/642406805 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f19740f5410 0x7f19740f5830 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f1964009fb0 tx=0x7f1964009b30).stop 2024-02-11T17:03:42.880 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:42.878+0000 7f1978ab6700 1 -- 172.21.15.178:0/642406805 shutdown_connections 2024-02-11T17:03:42.880 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:42.878+0000 7f1978ab6700 1 --2- 172.21.15.178:0/642406805 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f19740f5410 0x7f19740f5830 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:03:42.880 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:42.878+0000 7f1978ab6700 1 -- 172.21.15.178:0/642406805 >> 172.21.15.178:0/642406805 conn(0x7f19740f0b70 msgr2=0x7f19740f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:03:42.880 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:42.878+0000 7f1978ab6700 1 -- 172.21.15.178:0/642406805 shutdown_connections 2024-02-11T17:03:42.881 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:42.878+0000 7f1978ab6700 1 -- 172.21.15.178:0/642406805 wait complete. 2024-02-11T17:03:42.882 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:42.879+0000 7f1978ab6700 1 Processor -- start 2024-02-11T17:03:42.882 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:42.879+0000 7f1978ab6700 1 -- start start 2024-02-11T17:03:42.882 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:42.879+0000 7f1978ab6700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f19740f5410 0x7f19741093d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:03:42.882 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:42.879+0000 7f1978ab6700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1974060240 con 0x7f19740f5410 2024-02-11T17:03:42.882 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:42.880+0000 7f197259c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f19740f5410 0x7f19741093d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:03:42.882 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:42.880+0000 7f197259c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f19740f5410 0x7f19741093d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:44526/0 (socket says 172.21.15.178:44526) 2024-02-11T17:03:42.882 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:42.880+0000 7f197259c700 1 -- 172.21.15.178:0/4294372371 learned_addr learned my addr 172.21.15.178:0/4294372371 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:03:42.882 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:42.880+0000 7f197259c700 1 -- 172.21.15.178:0/4294372371 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1964005040 con 0x7f19740f5410 2024-02-11T17:03:42.883 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:42.880+0000 7f197259c700 1 --2- 172.21.15.178:0/4294372371 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f19740f5410 0x7f19741093d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f1964013510 tx=0x7f196400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:03:42.883 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:42.880+0000 7f196affd700 1 -- 172.21.15.178:0/4294372371 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1964009320 con 0x7f19740f5410 2024-02-11T17:03:42.883 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:42.880+0000 7f1978ab6700 1 -- 172.21.15.178:0/4294372371 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1974060440 con 0x7f19740f5410 2024-02-11T17:03:42.883 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:42.880+0000 7f196affd700 1 -- 172.21.15.178:0/4294372371 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1964013f00 con 0x7f19740f5410 2024-02-11T17:03:42.883 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:42.880+0000 7f196affd700 1 -- 172.21.15.178:0/4294372371 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1964014d90 con 0x7f19740f5410 2024-02-11T17:03:42.883 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:42.880+0000 7f1978ab6700 1 -- 172.21.15.178:0/4294372371 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1974109a80 con 0x7f19740f5410 2024-02-11T17:03:42.884 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:42.881+0000 7f196affd700 1 -- 172.21.15.178:0/4294372371 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1964014420 con 0x7f19740f5410 2024-02-11T17:03:42.884 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:42.881+0000 7f1978ab6700 1 -- 172.21.15.178:0/4294372371 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1974045ad0 con 0x7f19740f5410 2024-02-11T17:03:42.884 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:42.881+0000 7f196affd700 1 --2- 172.21.15.178:0/4294372371 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f196002ffb0 0x7f1960032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:03:42.884 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:42.881+0000 7f196affd700 1 -- 172.21.15.178:0/4294372371 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1964040ff0 con 0x7f19740f5410 2024-02-11T17:03:42.884 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:42.882+0000 7f1971d9b700 1 --2- 172.21.15.178:0/4294372371 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f196002ffb0 0x7f1960032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:03:42.884 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:42.882+0000 7f1971d9b700 1 --2- 172.21.15.178:0/4294372371 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f196002ffb0 0x7f1960032470 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f195c0096a0 tx=0x7f195c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:03:42.888 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:42.886+0000 7f196affd700 1 -- 172.21.15.178:0/4294372371 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f196406d2b0 con 0x7f19740f5410 2024-02-11T17:03:43.091 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:43.090+0000 7f1978ab6700 1 -- 172.21.15.178:0/4294372371 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f197410a2a0 con 0x7f19740f5410 2024-02-11T17:03:43.092 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:43.091+0000 7f196affd700 1 -- 172.21.15.178:0/4294372371 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f19640095f0 con 0x7f19740f5410 2024-02-11T17:03:43.093 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:03:43.093 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:03:43.094 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:43.092+0000 7f1978ab6700 1 -- 172.21.15.178:0/4294372371 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f196002ffb0 msgr2=0x7f1960032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:03:43.094 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:43.092+0000 7f1978ab6700 1 --2- 172.21.15.178:0/4294372371 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f196002ffb0 0x7f1960032470 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f195c0096a0 tx=0x7f195c006b40).stop 2024-02-11T17:03:43.094 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:43.092+0000 7f1978ab6700 1 -- 172.21.15.178:0/4294372371 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f19740f5410 msgr2=0x7f19741093d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:03:43.094 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:43.092+0000 7f1978ab6700 1 --2- 172.21.15.178:0/4294372371 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f19740f5410 0x7f19741093d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f1964013510 tx=0x7f196400b040).stop 2024-02-11T17:03:43.094 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:43.092+0000 7f1978ab6700 1 -- 172.21.15.178:0/4294372371 shutdown_connections 2024-02-11T17:03:43.095 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:43.092+0000 7f1978ab6700 1 --2- 172.21.15.178:0/4294372371 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f196002ffb0 0x7f1960032470 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:03:43.095 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:43.092+0000 7f1978ab6700 1 --2- 172.21.15.178:0/4294372371 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f19740f5410 0x7f19741093d0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:03:43.095 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:43.092+0000 7f1978ab6700 1 -- 172.21.15.178:0/4294372371 >> 172.21.15.178:0/4294372371 conn(0x7f19740f0b70 msgr2=0x7f19740f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:03:43.095 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:43.092+0000 7f1978ab6700 1 -- 172.21.15.178:0/4294372371 shutdown_connections 2024-02-11T17:03:43.095 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:43.092+0000 7f1978ab6700 1 -- 172.21.15.178:0/4294372371 wait complete. 2024-02-11T17:03:43.095 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:03:43.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:03:43 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:03:43.091882+0000 mon.smithi037 (mon.0) 172 : audit [DBG] from='client.? 172.21.15.178:0/4294372371' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:03:44.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:03:44 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:03:43.162061+0000 mgr.smithi037.eofnaq (mgr.14162) 111 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:03:44.719 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:03:44.720 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:03:46.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:03:46 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:03:45.162380+0000 mgr.smithi037.eofnaq (mgr.14162) 112 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:03:46.515 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:46.491+0000 7fe62f05f700 1 -- 172.21.15.178:0/3984397405 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe6280601e0 msgr2=0x7fe6280f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:03:46.516 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:46.491+0000 7fe62f05f700 1 --2- 172.21.15.178:0/3984397405 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe6280601e0 0x7fe6280f8f20 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7fe624009fb0 tx=0x7fe624009b30).stop 2024-02-11T17:03:46.516 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:46.491+0000 7fe62f05f700 1 -- 172.21.15.178:0/3984397405 shutdown_connections 2024-02-11T17:03:46.516 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:46.491+0000 7fe62f05f700 1 --2- 172.21.15.178:0/3984397405 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe6280601e0 0x7fe6280f8f20 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:03:46.516 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:46.491+0000 7fe62f05f700 1 -- 172.21.15.178:0/3984397405 >> 172.21.15.178:0/3984397405 conn(0x7fe6280f09a0 msgr2=0x7fe6280f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:03:46.516 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:46.491+0000 7fe62f05f700 1 -- 172.21.15.178:0/3984397405 shutdown_connections 2024-02-11T17:03:46.516 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:46.491+0000 7fe62f05f700 1 -- 172.21.15.178:0/3984397405 wait complete. 2024-02-11T17:03:46.516 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:46.491+0000 7fe62f05f700 1 Processor -- start 2024-02-11T17:03:46.516 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:46.491+0000 7fe62f05f700 1 -- start start 2024-02-11T17:03:46.516 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:46.491+0000 7fe62f05f700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe6280601e0 0x7fe628106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:03:46.516 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:46.491+0000 7fe62f05f700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6281069f0 con 0x7fe6280601e0 2024-02-11T17:03:46.516 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:46.492+0000 7fe62e05d700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe6280601e0 0x7fe628106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:03:46.517 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:46.492+0000 7fe62e05d700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe6280601e0 0x7fe628106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:44548/0 (socket says 172.21.15.178:44548) 2024-02-11T17:03:46.517 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:46.492+0000 7fe62e05d700 1 -- 172.21.15.178:0/3188358877 learned_addr learned my addr 172.21.15.178:0/3188358877 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:03:46.517 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:46.492+0000 7fe62e05d700 1 -- 172.21.15.178:0/3188358877 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe624005040 con 0x7fe6280601e0 2024-02-11T17:03:46.517 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:46.492+0000 7fe62e05d700 1 --2- 172.21.15.178:0/3188358877 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe6280601e0 0x7fe628106500 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fe624013510 tx=0x7fe62400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:03:46.517 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:46.492+0000 7fe61effd700 1 -- 172.21.15.178:0/3188358877 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe624009320 con 0x7fe6280601e0 2024-02-11T17:03:46.517 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:46.492+0000 7fe62f05f700 1 -- 172.21.15.178:0/3188358877 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe628106bf0 con 0x7fe6280601e0 2024-02-11T17:03:46.517 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:46.492+0000 7fe61effd700 1 -- 172.21.15.178:0/3188358877 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe624013f00 con 0x7fe6280601e0 2024-02-11T17:03:46.517 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:46.492+0000 7fe61effd700 1 -- 172.21.15.178:0/3188358877 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe624014d90 con 0x7fe6280601e0 2024-02-11T17:03:46.517 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:46.492+0000 7fe62f05f700 1 -- 172.21.15.178:0/3188358877 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe628107080 con 0x7fe6280601e0 2024-02-11T17:03:46.517 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:46.493+0000 7fe61effd700 1 -- 172.21.15.178:0/3188358877 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe6240095f0 con 0x7fe6280601e0 2024-02-11T17:03:46.517 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:46.493+0000 7fe62f05f700 1 -- 172.21.15.178:0/3188358877 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe628045ad0 con 0x7fe6280601e0 2024-02-11T17:03:46.517 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:46.493+0000 7fe61effd700 1 --2- 172.21.15.178:0/3188358877 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fe61402ff60 0x7fe614032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:03:46.517 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:46.493+0000 7fe61effd700 1 -- 172.21.15.178:0/3188358877 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe624041000 con 0x7fe6280601e0 2024-02-11T17:03:46.517 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:46.494+0000 7fe62d85c700 1 --2- 172.21.15.178:0/3188358877 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fe61402ff60 0x7fe614032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:03:46.518 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:46.494+0000 7fe62d85c700 1 --2- 172.21.15.178:0/3188358877 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fe61402ff60 0x7fe614032420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7fe6180096a0 tx=0x7fe618006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:03:46.518 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:46.498+0000 7fe61effd700 1 -- 172.21.15.178:0/3188358877 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe624014420 con 0x7fe6280601e0 2024-02-11T17:03:46.710 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:46.708+0000 7fe62f05f700 1 -- 172.21.15.178:0/3188358877 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe628107f30 con 0x7fe6280601e0 2024-02-11T17:03:46.711 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:46.709+0000 7fe61effd700 1 -- 172.21.15.178:0/3188358877 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe624010230 con 0x7fe6280601e0 2024-02-11T17:03:46.711 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:03:46.711 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:03:46.712 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:46.710+0000 7fe62f05f700 1 -- 172.21.15.178:0/3188358877 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fe61402ff60 msgr2=0x7fe614032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:03:46.712 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:46.710+0000 7fe62f05f700 1 --2- 172.21.15.178:0/3188358877 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fe61402ff60 0x7fe614032420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7fe6180096a0 tx=0x7fe618006b40).stop 2024-02-11T17:03:46.712 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:46.710+0000 7fe62f05f700 1 -- 172.21.15.178:0/3188358877 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe6280601e0 msgr2=0x7fe628106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:03:46.712 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:46.710+0000 7fe62f05f700 1 --2- 172.21.15.178:0/3188358877 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe6280601e0 0x7fe628106500 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fe624013510 tx=0x7fe62400b040).stop 2024-02-11T17:03:46.713 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:46.710+0000 7fe62f05f700 1 -- 172.21.15.178:0/3188358877 shutdown_connections 2024-02-11T17:03:46.713 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:46.710+0000 7fe62f05f700 1 --2- 172.21.15.178:0/3188358877 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fe61402ff60 0x7fe614032420 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:03:46.713 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:46.710+0000 7fe62f05f700 1 --2- 172.21.15.178:0/3188358877 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe6280601e0 0x7fe628106500 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:03:46.713 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:46.710+0000 7fe62f05f700 1 -- 172.21.15.178:0/3188358877 >> 172.21.15.178:0/3188358877 conn(0x7fe6280f09a0 msgr2=0x7fe6280f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:03:46.713 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:46.711+0000 7fe62f05f700 1 -- 172.21.15.178:0/3188358877 shutdown_connections 2024-02-11T17:03:46.713 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:46.711+0000 7fe62f05f700 1 -- 172.21.15.178:0/3188358877 wait complete. 2024-02-11T17:03:46.713 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:03:47.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:03:47 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:03:46.710103+0000 mon.smithi037 (mon.0) 173 : audit [DBG] from='client.? 172.21.15.178:0/3188358877' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:03:48.379 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:03:48.379 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:03:48.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:03:48 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:03:47.162676+0000 mgr.smithi037.eofnaq (mgr.14162) 113 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:03:50.250 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:50.247+0000 7f9548535700 1 -- 172.21.15.178:0/926189336 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95400f5410 msgr2=0x7f95400f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:03:50.250 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:50.247+0000 7f9548535700 1 --2- 172.21.15.178:0/926189336 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95400f5410 0x7f95400f5830 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f9530009fb0 tx=0x7f9530009b30).stop 2024-02-11T17:03:50.250 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:50.247+0000 7f9548535700 1 -- 172.21.15.178:0/926189336 shutdown_connections 2024-02-11T17:03:50.250 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:50.247+0000 7f9548535700 1 --2- 172.21.15.178:0/926189336 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95400f5410 0x7f95400f5830 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:03:50.250 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:50.247+0000 7f9548535700 1 -- 172.21.15.178:0/926189336 >> 172.21.15.178:0/926189336 conn(0x7f95400f0b70 msgr2=0x7f95400f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:03:50.251 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:50.248+0000 7f9548535700 1 -- 172.21.15.178:0/926189336 shutdown_connections 2024-02-11T17:03:50.251 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:50.248+0000 7f9548535700 1 -- 172.21.15.178:0/926189336 wait complete. 2024-02-11T17:03:50.251 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:50.248+0000 7f9548535700 1 Processor -- start 2024-02-11T17:03:50.251 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:50.248+0000 7f9548535700 1 -- start start 2024-02-11T17:03:50.251 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:50.248+0000 7f9548535700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95400f5410 0x7f9540108910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:03:50.251 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:50.248+0000 7f9548535700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9540108e00 con 0x7f95400f5410 2024-02-11T17:03:50.251 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:50.249+0000 7f95462d1700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95400f5410 0x7f9540108910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:03:50.251 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:50.249+0000 7f95462d1700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95400f5410 0x7f9540108910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:44006/0 (socket says 172.21.15.178:44006) 2024-02-11T17:03:50.251 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:50.249+0000 7f95462d1700 1 -- 172.21.15.178:0/280618657 learned_addr learned my addr 172.21.15.178:0/280618657 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:03:50.252 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:50.249+0000 7f95462d1700 1 -- 172.21.15.178:0/280618657 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9530005040 con 0x7f95400f5410 2024-02-11T17:03:50.253 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:50.249+0000 7f95462d1700 1 --2- 172.21.15.178:0/280618657 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95400f5410 0x7f9540108910 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f9530013510 tx=0x7f953000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:03:50.253 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:50.249+0000 7f9536ffd700 1 -- 172.21.15.178:0/280618657 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9530009320 con 0x7f95400f5410 2024-02-11T17:03:50.253 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:50.249+0000 7f9548535700 1 -- 172.21.15.178:0/280618657 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9540109000 con 0x7f95400f5410 2024-02-11T17:03:50.253 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:50.249+0000 7f9536ffd700 1 -- 172.21.15.178:0/280618657 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9530013f00 con 0x7f95400f5410 2024-02-11T17:03:50.253 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:50.249+0000 7f9536ffd700 1 -- 172.21.15.178:0/280618657 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9530014d90 con 0x7f95400f5410 2024-02-11T17:03:50.253 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:50.249+0000 7f9548535700 1 -- 172.21.15.178:0/280618657 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9540109490 con 0x7f95400f5410 2024-02-11T17:03:50.253 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:50.250+0000 7f9536ffd700 1 -- 172.21.15.178:0/280618657 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f95300095f0 con 0x7f95400f5410 2024-02-11T17:03:50.253 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:50.250+0000 7f9548535700 1 -- 172.21.15.178:0/280618657 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9540045ad0 con 0x7f95400f5410 2024-02-11T17:03:50.253 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:50.250+0000 7f9536ffd700 1 --2- 172.21.15.178:0/280618657 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f952c02ff60 0x7f952c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:03:50.253 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:50.250+0000 7f9536ffd700 1 -- 172.21.15.178:0/280618657 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f95300410a0 con 0x7f95400f5410 2024-02-11T17:03:50.253 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:50.251+0000 7f9545ad0700 1 --2- 172.21.15.178:0/280618657 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f952c02ff60 0x7f952c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:03:50.253 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:50.251+0000 7f9545ad0700 1 --2- 172.21.15.178:0/280618657 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f952c02ff60 0x7f952c032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f953c0096a0 tx=0x7f953c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:03:50.256 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:50.255+0000 7f9536ffd700 1 -- 172.21.15.178:0/280618657 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f953001d7f0 con 0x7f95400f5410 2024-02-11T17:03:50.463 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:50.461+0000 7f9548535700 1 -- 172.21.15.178:0/280618657 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f95400fae60 con 0x7f95400f5410 2024-02-11T17:03:50.464 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:50.462+0000 7f9536ffd700 1 -- 172.21.15.178:0/280618657 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9530010230 con 0x7f95400f5410 2024-02-11T17:03:50.464 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:03:50.464 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:03:50.465 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:50.464+0000 7f9548535700 1 -- 172.21.15.178:0/280618657 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f952c02ff60 msgr2=0x7f952c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:03:50.465 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:50.464+0000 7f9548535700 1 --2- 172.21.15.178:0/280618657 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f952c02ff60 0x7f952c032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f953c0096a0 tx=0x7f953c006b40).stop 2024-02-11T17:03:50.465 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:50.464+0000 7f9548535700 1 -- 172.21.15.178:0/280618657 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95400f5410 msgr2=0x7f9540108910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:03:50.465 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:50.464+0000 7f9548535700 1 --2- 172.21.15.178:0/280618657 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95400f5410 0x7f9540108910 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f9530013510 tx=0x7f953000b040).stop 2024-02-11T17:03:50.466 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:50.464+0000 7f9548535700 1 -- 172.21.15.178:0/280618657 shutdown_connections 2024-02-11T17:03:50.466 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:50.464+0000 7f9548535700 1 --2- 172.21.15.178:0/280618657 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f952c02ff60 0x7f952c032420 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:03:50.466 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:50.464+0000 7f9548535700 1 --2- 172.21.15.178:0/280618657 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95400f5410 0x7f9540108910 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:03:50.466 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:50.464+0000 7f9548535700 1 -- 172.21.15.178:0/280618657 >> 172.21.15.178:0/280618657 conn(0x7f95400f0b70 msgr2=0x7f954005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:03:50.466 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:50.464+0000 7f9548535700 1 -- 172.21.15.178:0/280618657 shutdown_connections 2024-02-11T17:03:50.466 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:50.464+0000 7f9548535700 1 -- 172.21.15.178:0/280618657 wait complete. 2024-02-11T17:03:50.466 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:03:50.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:03:50 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:03:49.162979+0000 mgr.smithi037.eofnaq (mgr.14162) 114 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:03:51.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:03:51 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:03:50.463308+0000 mon.smithi037 (mon.0) 174 : audit [DBG] from='client.? 172.21.15.178:0/280618657' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:03:52.178 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:03:52.179 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:03:52.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:03:52 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:03:51.163323+0000 mgr.smithi037.eofnaq (mgr.14162) 115 : cluster [DBG] pgmap v108: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:03:53.952 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:53.949+0000 7ff4ac899700 1 -- 172.21.15.178:0/2116564813 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff4a80f51a0 msgr2=0x7ff4a80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:03:53.953 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:53.949+0000 7ff4ac899700 1 --2- 172.21.15.178:0/2116564813 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff4a80f51a0 0x7ff4a80f55c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7ff490009fb0 tx=0x7ff490009b30).stop 2024-02-11T17:03:53.953 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:53.949+0000 7ff4ac899700 1 -- 172.21.15.178:0/2116564813 shutdown_connections 2024-02-11T17:03:53.953 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:53.949+0000 7ff4ac899700 1 --2- 172.21.15.178:0/2116564813 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff4a80f51a0 0x7ff4a80f55c0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:03:53.953 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:53.949+0000 7ff4ac899700 1 -- 172.21.15.178:0/2116564813 >> 172.21.15.178:0/2116564813 conn(0x7ff4a80f09a0 msgr2=0x7ff4a80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:03:53.953 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:53.949+0000 7ff4ac899700 1 -- 172.21.15.178:0/2116564813 shutdown_connections 2024-02-11T17:03:53.953 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:53.949+0000 7ff4ac899700 1 -- 172.21.15.178:0/2116564813 wait complete. 2024-02-11T17:03:53.953 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:53.950+0000 7ff4ac899700 1 Processor -- start 2024-02-11T17:03:53.953 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:53.950+0000 7ff4ac899700 1 -- start start 2024-02-11T17:03:53.953 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:53.950+0000 7ff4ac899700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff4a80f51a0 0x7ff4a81064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:03:53.953 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:53.950+0000 7ff4ac899700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a81069d0 con 0x7ff4a80f51a0 2024-02-11T17:03:53.953 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:53.950+0000 7ff4a77fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff4a80f51a0 0x7ff4a81064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:03:53.953 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:53.950+0000 7ff4a77fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff4a80f51a0 0x7ff4a81064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:44026/0 (socket says 172.21.15.178:44026) 2024-02-11T17:03:53.953 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:53.950+0000 7ff4a77fe700 1 -- 172.21.15.178:0/2591409938 learned_addr learned my addr 172.21.15.178:0/2591409938 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:03:53.954 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:53.951+0000 7ff4a77fe700 1 -- 172.21.15.178:0/2591409938 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff490005040 con 0x7ff4a80f51a0 2024-02-11T17:03:53.954 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:53.951+0000 7ff4a77fe700 1 --2- 172.21.15.178:0/2591409938 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff4a80f51a0 0x7ff4a81064e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7ff490009830 tx=0x7ff49000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:03:53.954 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:53.951+0000 7ff49ffff700 1 -- 172.21.15.178:0/2591409938 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff490009320 con 0x7ff4a80f51a0 2024-02-11T17:03:53.954 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:53.951+0000 7ff4ac899700 1 -- 172.21.15.178:0/2591409938 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4a8106bd0 con 0x7ff4a80f51a0 2024-02-11T17:03:53.954 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:53.951+0000 7ff49ffff700 1 -- 172.21.15.178:0/2591409938 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff4900136c0 con 0x7ff4a80f51a0 2024-02-11T17:03:53.954 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:53.951+0000 7ff49ffff700 1 -- 172.21.15.178:0/2591409938 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff49001ee90 con 0x7ff4a80f51a0 2024-02-11T17:03:53.954 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:53.951+0000 7ff4ac899700 1 -- 172.21.15.178:0/2591409938 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4a8107060 con 0x7ff4a80f51a0 2024-02-11T17:03:53.955 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:53.952+0000 7ff49ffff700 1 -- 172.21.15.178:0/2591409938 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff490014480 con 0x7ff4a80f51a0 2024-02-11T17:03:53.955 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:53.952+0000 7ff4ac899700 1 -- 172.21.15.178:0/2591409938 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4a8045ad0 con 0x7ff4a80f51a0 2024-02-11T17:03:53.955 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:53.952+0000 7ff49ffff700 1 --2- 172.21.15.178:0/2591409938 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7ff49402ff10 0x7ff4940323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:03:53.955 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:53.952+0000 7ff49ffff700 1 -- 172.21.15.178:0/2591409938 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff490040ef0 con 0x7ff4a80f51a0 2024-02-11T17:03:53.955 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:53.952+0000 7ff4a6ffd700 1 --2- 172.21.15.178:0/2591409938 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7ff49402ff10 0x7ff4940323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:03:53.955 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:53.952+0000 7ff4a6ffd700 1 --2- 172.21.15.178:0/2591409938 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7ff49402ff10 0x7ff4940323d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7ff4980096a0 tx=0x7ff498006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:03:53.958 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:53.957+0000 7ff49ffff700 1 -- 172.21.15.178:0/2591409938 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff49001c910 con 0x7ff4a80f51a0 2024-02-11T17:03:54.161 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:54.159+0000 7ff4ac899700 1 -- 172.21.15.178:0/2591409938 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff4a8107f10 con 0x7ff4a80f51a0 2024-02-11T17:03:54.162 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:54.161+0000 7ff49ffff700 1 -- 172.21.15.178:0/2591409938 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff49001c320 con 0x7ff4a80f51a0 2024-02-11T17:03:54.162 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:03:54.162 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:03:54.163 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:54.161+0000 7ff4ac899700 1 -- 172.21.15.178:0/2591409938 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7ff49402ff10 msgr2=0x7ff4940323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:03:54.163 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:54.161+0000 7ff4ac899700 1 --2- 172.21.15.178:0/2591409938 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7ff49402ff10 0x7ff4940323d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7ff4980096a0 tx=0x7ff498006b40).stop 2024-02-11T17:03:54.163 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:54.161+0000 7ff4ac899700 1 -- 172.21.15.178:0/2591409938 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff4a80f51a0 msgr2=0x7ff4a81064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:03:54.164 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:54.161+0000 7ff4ac899700 1 --2- 172.21.15.178:0/2591409938 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff4a80f51a0 0x7ff4a81064e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7ff490009830 tx=0x7ff49000b040).stop 2024-02-11T17:03:54.164 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:54.162+0000 7ff4ac899700 1 -- 172.21.15.178:0/2591409938 shutdown_connections 2024-02-11T17:03:54.164 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:54.162+0000 7ff4ac899700 1 --2- 172.21.15.178:0/2591409938 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7ff49402ff10 0x7ff4940323d0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:03:54.164 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:54.162+0000 7ff4ac899700 1 --2- 172.21.15.178:0/2591409938 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff4a80f51a0 0x7ff4a81064e0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:03:54.164 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:54.162+0000 7ff4ac899700 1 -- 172.21.15.178:0/2591409938 >> 172.21.15.178:0/2591409938 conn(0x7ff4a80f09a0 msgr2=0x7ff4a805fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:03:54.164 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:54.162+0000 7ff4ac899700 1 -- 172.21.15.178:0/2591409938 shutdown_connections 2024-02-11T17:03:54.164 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:54.162+0000 7ff4ac899700 1 -- 172.21.15.178:0/2591409938 wait complete. 2024-02-11T17:03:54.164 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:03:54.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:03:54 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:03:53.163583+0000 mgr.smithi037.eofnaq (mgr.14162) 116 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:03:54.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:03:54 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:03:54.161149+0000 mon.smithi037 (mon.0) 175 : audit [DBG] from='client.? 172.21.15.178:0/2591409938' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:03:55.788 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:03:55.788 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:03:56.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:03:56 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:03:55.163958+0000 mgr.smithi037.eofnaq (mgr.14162) 117 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:03:57.551 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:57.548+0000 7f8f8a8d7700 1 -- 172.21.15.178:0/1348239780 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8f84060280 msgr2=0x7f8f840f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:03:57.551 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:57.548+0000 7f8f8a8d7700 1 --2- 172.21.15.178:0/1348239780 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8f84060280 0x7f8f840f90f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f8f6c009fb0 tx=0x7f8f6c009b30).stop 2024-02-11T17:03:57.551 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:57.548+0000 7f8f8a8d7700 1 -- 172.21.15.178:0/1348239780 shutdown_connections 2024-02-11T17:03:57.551 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:57.548+0000 7f8f8a8d7700 1 --2- 172.21.15.178:0/1348239780 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8f84060280 0x7f8f840f90f0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:03:57.551 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:57.548+0000 7f8f8a8d7700 1 -- 172.21.15.178:0/1348239780 >> 172.21.15.178:0/1348239780 conn(0x7f8f840f0b70 msgr2=0x7f8f840f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:03:57.551 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:57.548+0000 7f8f8a8d7700 1 -- 172.21.15.178:0/1348239780 shutdown_connections 2024-02-11T17:03:57.551 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:57.548+0000 7f8f8a8d7700 1 -- 172.21.15.178:0/1348239780 wait complete. 2024-02-11T17:03:57.552 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:57.548+0000 7f8f8a8d7700 1 Processor -- start 2024-02-11T17:03:57.552 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:57.549+0000 7f8f8a8d7700 1 -- start start 2024-02-11T17:03:57.552 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:57.549+0000 7f8f8a8d7700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8f84060280 0x7f8f841066c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:03:57.552 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:57.549+0000 7f8f8a8d7700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f84106bb0 con 0x7f8f84060280 2024-02-11T17:03:57.552 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:57.549+0000 7f8f83fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8f84060280 0x7f8f841066c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:03:57.552 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:57.549+0000 7f8f83fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8f84060280 0x7f8f841066c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:44044/0 (socket says 172.21.15.178:44044) 2024-02-11T17:03:57.552 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:57.549+0000 7f8f83fff700 1 -- 172.21.15.178:0/3832369695 learned_addr learned my addr 172.21.15.178:0/3832369695 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:03:57.552 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:57.549+0000 7f8f83fff700 1 -- 172.21.15.178:0/3832369695 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f6c005040 con 0x7f8f84060280 2024-02-11T17:03:57.552 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:57.549+0000 7f8f83fff700 1 --2- 172.21.15.178:0/3832369695 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8f84060280 0x7f8f841066c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f8f6c013510 tx=0x7f8f6c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:03:57.554 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:57.550+0000 7f8f80ff9700 1 -- 172.21.15.178:0/3832369695 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f6c009320 con 0x7f8f84060280 2024-02-11T17:03:57.554 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:57.550+0000 7f8f8a8d7700 1 -- 172.21.15.178:0/3832369695 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f84106db0 con 0x7f8f84060280 2024-02-11T17:03:57.554 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:57.550+0000 7f8f80ff9700 1 -- 172.21.15.178:0/3832369695 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8f6c013f00 con 0x7f8f84060280 2024-02-11T17:03:57.554 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:57.550+0000 7f8f80ff9700 1 -- 172.21.15.178:0/3832369695 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f6c014dc0 con 0x7f8f84060280 2024-02-11T17:03:57.554 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:57.550+0000 7f8f8a8d7700 1 -- 172.21.15.178:0/3832369695 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f84107240 con 0x7f8f84060280 2024-02-11T17:03:57.554 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:57.551+0000 7f8f80ff9700 1 -- 172.21.15.178:0/3832369695 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8f6c0095f0 con 0x7f8f84060280 2024-02-11T17:03:57.554 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:57.551+0000 7f8f8a8d7700 1 -- 172.21.15.178:0/3832369695 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f84045ad0 con 0x7f8f84060280 2024-02-11T17:03:57.555 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:57.551+0000 7f8f80ff9700 1 --2- 172.21.15.178:0/3832369695 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f8f7002ff60 0x7f8f70032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:03:57.555 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:57.551+0000 7f8f80ff9700 1 -- 172.21.15.178:0/3832369695 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8f6c041000 con 0x7f8f84060280 2024-02-11T17:03:57.555 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:57.551+0000 7f8f837fe700 1 --2- 172.21.15.178:0/3832369695 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f8f7002ff60 0x7f8f70032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:03:57.555 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:57.551+0000 7f8f837fe700 1 --2- 172.21.15.178:0/3832369695 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f8f7002ff60 0x7f8f70032420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f8f740096a0 tx=0x7f8f74006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:03:57.557 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:57.556+0000 7f8f80ff9700 1 -- 172.21.15.178:0/3832369695 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8f6c01c790 con 0x7f8f84060280 2024-02-11T17:03:57.760 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:57.758+0000 7f8f8a8d7700 1 -- 172.21.15.178:0/3832369695 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8f841080f0 con 0x7f8f84060280 2024-02-11T17:03:57.761 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:57.759+0000 7f8f80ff9700 1 -- 172.21.15.178:0/3832369695 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8f6c010230 con 0x7f8f84060280 2024-02-11T17:03:57.761 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:03:57.761 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:03:57.763 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:57.761+0000 7f8f8a8d7700 1 -- 172.21.15.178:0/3832369695 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f8f7002ff60 msgr2=0x7f8f70032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:03:57.763 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:57.761+0000 7f8f8a8d7700 1 --2- 172.21.15.178:0/3832369695 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f8f7002ff60 0x7f8f70032420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f8f740096a0 tx=0x7f8f74006b40).stop 2024-02-11T17:03:57.763 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:57.761+0000 7f8f8a8d7700 1 -- 172.21.15.178:0/3832369695 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8f84060280 msgr2=0x7f8f841066c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:03:57.763 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:57.761+0000 7f8f8a8d7700 1 --2- 172.21.15.178:0/3832369695 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8f84060280 0x7f8f841066c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f8f6c013510 tx=0x7f8f6c00b040).stop 2024-02-11T17:03:57.763 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:57.761+0000 7f8f8a8d7700 1 -- 172.21.15.178:0/3832369695 shutdown_connections 2024-02-11T17:03:57.763 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:57.761+0000 7f8f8a8d7700 1 --2- 172.21.15.178:0/3832369695 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f8f7002ff60 0x7f8f70032420 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:03:57.763 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:57.761+0000 7f8f8a8d7700 1 --2- 172.21.15.178:0/3832369695 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8f84060280 0x7f8f841066c0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:03:57.764 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:57.761+0000 7f8f8a8d7700 1 -- 172.21.15.178:0/3832369695 >> 172.21.15.178:0/3832369695 conn(0x7f8f840f0b70 msgr2=0x7f8f840f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:03:57.764 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:57.761+0000 7f8f8a8d7700 1 -- 172.21.15.178:0/3832369695 shutdown_connections 2024-02-11T17:03:57.764 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:03:57.761+0000 7f8f8a8d7700 1 -- 172.21.15.178:0/3832369695 wait complete. 2024-02-11T17:03:57.764 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:03:58.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:03:58 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:03:57.164341+0000 mgr.smithi037.eofnaq (mgr.14162) 118 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:03:58.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:03:58 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:03:57.760549+0000 mon.smithi037 (mon.0) 176 : audit [DBG] from='client.? 172.21.15.178:0/3832369695' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:03:59.465 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:03:59.466 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:04:00.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:04:00 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:03:59.164756+0000 mgr.smithi037.eofnaq (mgr.14162) 119 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:04:01.350 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:01.347+0000 7f1fdf614700 1 -- 172.21.15.178:0/3640188596 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1fd80f6510 msgr2=0x7f1fd80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:04:01.351 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:01.347+0000 7f1fdf614700 1 --2- 172.21.15.178:0/3640188596 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1fd80f6510 0x7f1fd80f6930 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f1fc8009fb0 tx=0x7f1fc8009b30).stop 2024-02-11T17:04:01.351 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:01.347+0000 7f1fdf614700 1 -- 172.21.15.178:0/3640188596 shutdown_connections 2024-02-11T17:04:01.351 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:01.347+0000 7f1fdf614700 1 --2- 172.21.15.178:0/3640188596 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1fd80f6510 0x7f1fd80f6930 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:04:01.351 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:01.347+0000 7f1fdf614700 1 -- 172.21.15.178:0/3640188596 >> 172.21.15.178:0/3640188596 conn(0x7f1fd80f1c70 msgr2=0x7f1fd80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:04:01.351 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:01.348+0000 7f1fdf614700 1 -- 172.21.15.178:0/3640188596 shutdown_connections 2024-02-11T17:04:01.351 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:01.348+0000 7f1fdf614700 1 -- 172.21.15.178:0/3640188596 wait complete. 2024-02-11T17:04:01.351 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:01.348+0000 7f1fdf614700 1 Processor -- start 2024-02-11T17:04:01.351 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:01.348+0000 7f1fdf614700 1 -- start start 2024-02-11T17:04:01.351 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:01.348+0000 7f1fdf614700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1fd80f6510 0x7f1fd8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:04:01.351 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:01.348+0000 7f1fdf614700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd8106c20 con 0x7f1fd80f6510 2024-02-11T17:04:01.351 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:01.349+0000 7f1fdd3b0700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1fd80f6510 0x7f1fd8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:04:01.351 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:01.349+0000 7f1fdd3b0700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1fd80f6510 0x7f1fd8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:53960/0 (socket says 172.21.15.178:53960) 2024-02-11T17:04:01.351 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:01.349+0000 7f1fdd3b0700 1 -- 172.21.15.178:0/1378745462 learned_addr learned my addr 172.21.15.178:0/1378745462 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:04:01.351 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:01.349+0000 7f1fdd3b0700 1 -- 172.21.15.178:0/1378745462 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fc8005040 con 0x7f1fd80f6510 2024-02-11T17:04:01.352 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:01.349+0000 7f1fdd3b0700 1 --2- 172.21.15.178:0/1378745462 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1fd80f6510 0x7f1fd8106730 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f1fc8013510 tx=0x7f1fc800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:04:01.353 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:01.349+0000 7f1fcdffb700 1 -- 172.21.15.178:0/1378745462 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1fc8009320 con 0x7f1fd80f6510 2024-02-11T17:04:01.353 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:01.349+0000 7f1fdf614700 1 -- 172.21.15.178:0/1378745462 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fd8106e20 con 0x7f1fd80f6510 2024-02-11T17:04:01.353 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:01.349+0000 7f1fcdffb700 1 -- 172.21.15.178:0/1378745462 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1fc8013f00 con 0x7f1fd80f6510 2024-02-11T17:04:01.353 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:01.349+0000 7f1fcdffb700 1 -- 172.21.15.178:0/1378745462 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1fc8014d90 con 0x7f1fd80f6510 2024-02-11T17:04:01.353 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:01.349+0000 7f1fdf614700 1 -- 172.21.15.178:0/1378745462 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fd81072b0 con 0x7f1fd80f6510 2024-02-11T17:04:01.353 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:01.350+0000 7f1fcdffb700 1 -- 172.21.15.178:0/1378745462 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1fc80263a0 con 0x7f1fd80f6510 2024-02-11T17:04:01.353 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:01.350+0000 7f1fcdffb700 1 --2- 172.21.15.178:0/1378745462 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f1fc402ffb0 0x7f1fc4032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:04:01.354 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:01.350+0000 7f1fdf614700 1 -- 172.21.15.178:0/1378745462 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fd8045ad0 con 0x7f1fd80f6510 2024-02-11T17:04:01.354 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:01.350+0000 7f1fcdffb700 1 -- 172.21.15.178:0/1378745462 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1fc80410a0 con 0x7f1fd80f6510 2024-02-11T17:04:01.354 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:01.351+0000 7f1fdcbaf700 1 --2- 172.21.15.178:0/1378745462 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f1fc402ffb0 0x7f1fc4032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:04:01.354 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:01.351+0000 7f1fdcbaf700 1 --2- 172.21.15.178:0/1378745462 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f1fc402ffb0 0x7f1fc4032470 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f1fd40096a0 tx=0x7f1fd4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:04:01.357 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:01.356+0000 7f1fcdffb700 1 -- 172.21.15.178:0/1378745462 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1fc8012920 con 0x7f1fd80f6510 2024-02-11T17:04:01.565 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:01.563+0000 7f1fdf614700 1 -- 172.21.15.178:0/1378745462 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1fd80fc050 con 0x7f1fd80f6510 2024-02-11T17:04:01.566 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:01.564+0000 7f1fcdffb700 1 -- 172.21.15.178:0/1378745462 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1fc800f340 con 0x7f1fd80f6510 2024-02-11T17:04:01.566 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:04:01.566 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:04:01.568 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:01.566+0000 7f1fdf614700 1 -- 172.21.15.178:0/1378745462 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f1fc402ffb0 msgr2=0x7f1fc4032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:04:01.568 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:01.566+0000 7f1fdf614700 1 --2- 172.21.15.178:0/1378745462 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f1fc402ffb0 0x7f1fc4032470 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f1fd40096a0 tx=0x7f1fd4006b40).stop 2024-02-11T17:04:01.568 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:01.566+0000 7f1fdf614700 1 -- 172.21.15.178:0/1378745462 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1fd80f6510 msgr2=0x7f1fd8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:04:01.568 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:01.566+0000 7f1fdf614700 1 --2- 172.21.15.178:0/1378745462 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1fd80f6510 0x7f1fd8106730 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f1fc8013510 tx=0x7f1fc800b040).stop 2024-02-11T17:04:01.568 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:01.566+0000 7f1fdf614700 1 -- 172.21.15.178:0/1378745462 shutdown_connections 2024-02-11T17:04:01.568 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:01.566+0000 7f1fdf614700 1 --2- 172.21.15.178:0/1378745462 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f1fc402ffb0 0x7f1fc4032470 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:04:01.568 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:01.566+0000 7f1fdf614700 1 --2- 172.21.15.178:0/1378745462 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1fd80f6510 0x7f1fd8106730 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:04:01.568 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:01.566+0000 7f1fdf614700 1 -- 172.21.15.178:0/1378745462 >> 172.21.15.178:0/1378745462 conn(0x7f1fd80f1c70 msgr2=0x7f1fd80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:04:01.568 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:01.566+0000 7f1fdf614700 1 -- 172.21.15.178:0/1378745462 shutdown_connections 2024-02-11T17:04:01.569 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:01.566+0000 7f1fdf614700 1 -- 172.21.15.178:0/1378745462 wait complete. 2024-02-11T17:04:01.569 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:04:02.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:04:02 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:04:01.164998+0000 mgr.smithi037.eofnaq (mgr.14162) 120 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:04:02.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:04:02 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:04:01.565336+0000 mon.smithi037 (mon.0) 177 : audit [DBG] from='client.? 172.21.15.178:0/1378745462' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:04:03.221 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:04:03.221 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:04:04.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:04:04 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:04:03.165239+0000 mgr.smithi037.eofnaq (mgr.14162) 121 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:04:04.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:04:04 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:04:03.273808+0000 mon.smithi037 (mon.0) 178 : audit [INF] from='mgr.14162 172.21.15.37:0/2745465012' entity='mgr.smithi037.eofnaq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi037.eofnaq/mirror_snapshot_schedule"}]: dispatch 2024-02-11T17:04:04.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:04:04 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:04:03.277509+0000 mon.smithi037 (mon.0) 179 : audit [INF] from='mgr.14162 172.21.15.37:0/2745465012' entity='mgr.smithi037.eofnaq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi037.eofnaq/trash_purge_schedule"}]: dispatch 2024-02-11T17:04:05.129 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:05.127+0000 7fd825dc9700 1 -- 172.21.15.178:0/802057439 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd8200f5410 msgr2=0x7fd8200f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:04:05.129 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:05.127+0000 7fd825dc9700 1 --2- 172.21.15.178:0/802057439 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd8200f5410 0x7fd8200f5830 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7fd808009fb0 tx=0x7fd808009b30).stop 2024-02-11T17:04:05.129 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:05.127+0000 7fd825dc9700 1 -- 172.21.15.178:0/802057439 shutdown_connections 2024-02-11T17:04:05.129 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:05.127+0000 7fd825dc9700 1 --2- 172.21.15.178:0/802057439 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd8200f5410 0x7fd8200f5830 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:04:05.129 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:05.127+0000 7fd825dc9700 1 -- 172.21.15.178:0/802057439 >> 172.21.15.178:0/802057439 conn(0x7fd8200f0b70 msgr2=0x7fd8200f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:04:05.129 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:05.127+0000 7fd825dc9700 1 -- 172.21.15.178:0/802057439 shutdown_connections 2024-02-11T17:04:05.129 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:05.127+0000 7fd825dc9700 1 -- 172.21.15.178:0/802057439 wait complete. 2024-02-11T17:04:05.129 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:05.128+0000 7fd825dc9700 1 Processor -- start 2024-02-11T17:04:05.129 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:05.128+0000 7fd825dc9700 1 -- start start 2024-02-11T17:04:05.131 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:05.129+0000 7fd825dc9700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd8200f5410 0x7fd820109390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:04:05.132 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:05.129+0000 7fd825dc9700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd820060280 con 0x7fd8200f5410 2024-02-11T17:04:05.132 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:05.129+0000 7fd81f7fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd8200f5410 0x7fd820109390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:04:05.132 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:05.129+0000 7fd81f7fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd8200f5410 0x7fd820109390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:53974/0 (socket says 172.21.15.178:53974) 2024-02-11T17:04:05.132 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:05.129+0000 7fd81f7fe700 1 -- 172.21.15.178:0/2224886910 learned_addr learned my addr 172.21.15.178:0/2224886910 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:04:05.132 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:05.129+0000 7fd81f7fe700 1 -- 172.21.15.178:0/2224886910 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd808005040 con 0x7fd8200f5410 2024-02-11T17:04:05.132 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:05.129+0000 7fd81f7fe700 1 --2- 172.21.15.178:0/2224886910 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd8200f5410 0x7fd820109390 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7fd808013510 tx=0x7fd80800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:04:05.132 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:05.130+0000 7fd817fff700 1 -- 172.21.15.178:0/2224886910 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd808009320 con 0x7fd8200f5410 2024-02-11T17:04:05.132 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:05.130+0000 7fd825dc9700 1 -- 172.21.15.178:0/2224886910 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd820060480 con 0x7fd8200f5410 2024-02-11T17:04:05.132 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:05.130+0000 7fd817fff700 1 -- 172.21.15.178:0/2224886910 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd808013f00 con 0x7fd8200f5410 2024-02-11T17:04:05.132 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:05.130+0000 7fd817fff700 1 -- 172.21.15.178:0/2224886910 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd808014d90 con 0x7fd8200f5410 2024-02-11T17:04:05.132 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:05.130+0000 7fd825dc9700 1 -- 172.21.15.178:0/2224886910 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd820109ac0 con 0x7fd8200f5410 2024-02-11T17:04:05.132 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:05.130+0000 7fd817fff700 1 -- 172.21.15.178:0/2224886910 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd808014420 con 0x7fd8200f5410 2024-02-11T17:04:05.133 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:05.130+0000 7fd825dc9700 1 -- 172.21.15.178:0/2224886910 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd820045ad0 con 0x7fd8200f5410 2024-02-11T17:04:05.133 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:05.131+0000 7fd817fff700 1 --2- 172.21.15.178:0/2224886910 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fd80c02ff60 0x7fd80c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:04:05.133 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:05.131+0000 7fd817fff700 1 -- 172.21.15.178:0/2224886910 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd808041070 con 0x7fd8200f5410 2024-02-11T17:04:05.133 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:05.131+0000 7fd81effd700 1 --2- 172.21.15.178:0/2224886910 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fd80c02ff60 0x7fd80c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:04:05.133 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:05.131+0000 7fd81effd700 1 --2- 172.21.15.178:0/2224886910 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fd80c02ff60 0x7fd80c032420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fd8100096a0 tx=0x7fd810006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:04:05.136 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:05.135+0000 7fd817fff700 1 -- 172.21.15.178:0/2224886910 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd8080095f0 con 0x7fd8200f5410 2024-02-11T17:04:05.340 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:05.339+0000 7fd825dc9700 1 -- 172.21.15.178:0/2224886910 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd8200fae60 con 0x7fd8200f5410 2024-02-11T17:04:05.340 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:05.340+0000 7fd817fff700 1 -- 172.21.15.178:0/2224886910 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd80801db70 con 0x7fd8200f5410 2024-02-11T17:04:05.341 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:04:05.341 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:04:05.342 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:05.341+0000 7fd825dc9700 1 -- 172.21.15.178:0/2224886910 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fd80c02ff60 msgr2=0x7fd80c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:04:05.342 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:05.341+0000 7fd825dc9700 1 --2- 172.21.15.178:0/2224886910 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fd80c02ff60 0x7fd80c032420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fd8100096a0 tx=0x7fd810006b40).stop 2024-02-11T17:04:05.342 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:05.341+0000 7fd825dc9700 1 -- 172.21.15.178:0/2224886910 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd8200f5410 msgr2=0x7fd820109390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:04:05.343 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:05.341+0000 7fd825dc9700 1 --2- 172.21.15.178:0/2224886910 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd8200f5410 0x7fd820109390 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7fd808013510 tx=0x7fd80800b040).stop 2024-02-11T17:04:05.343 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:05.341+0000 7fd825dc9700 1 -- 172.21.15.178:0/2224886910 shutdown_connections 2024-02-11T17:04:05.343 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:05.341+0000 7fd825dc9700 1 --2- 172.21.15.178:0/2224886910 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fd80c02ff60 0x7fd80c032420 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:04:05.343 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:05.341+0000 7fd825dc9700 1 --2- 172.21.15.178:0/2224886910 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd8200f5410 0x7fd820109390 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:04:05.343 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:05.341+0000 7fd825dc9700 1 -- 172.21.15.178:0/2224886910 >> 172.21.15.178:0/2224886910 conn(0x7fd8200f0b70 msgr2=0x7fd8200f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:04:05.343 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:05.342+0000 7fd825dc9700 1 -- 172.21.15.178:0/2224886910 shutdown_connections 2024-02-11T17:04:05.343 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:05.342+0000 7fd825dc9700 1 -- 172.21.15.178:0/2224886910 wait complete. 2024-02-11T17:04:05.343 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:04:06.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:04:06 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:04:05.165623+0000 mgr.smithi037.eofnaq (mgr.14162) 122 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:04:06.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:04:06 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:04:05.340296+0000 mon.smithi037 (mon.0) 180 : audit [DBG] from='client.? 172.21.15.178:0/2224886910' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:04:07.022 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:04:07.023 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:04:08.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:04:08 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:04:07.166005+0000 mgr.smithi037.eofnaq (mgr.14162) 123 : cluster [DBG] pgmap v116: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:04:08.969 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:08.967+0000 7fe2b889a700 1 -- 172.21.15.178:0/1641307998 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe2b40f5410 msgr2=0x7fe2b40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:04:08.969 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:08.967+0000 7fe2b889a700 1 --2- 172.21.15.178:0/1641307998 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe2b40f5410 0x7fe2b40f5830 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7fe2a8009fb0 tx=0x7fe2a8009b30).stop 2024-02-11T17:04:08.969 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:08.967+0000 7fe2b889a700 1 -- 172.21.15.178:0/1641307998 shutdown_connections 2024-02-11T17:04:08.970 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:08.967+0000 7fe2b889a700 1 --2- 172.21.15.178:0/1641307998 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe2b40f5410 0x7fe2b40f5830 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:04:08.970 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:08.967+0000 7fe2b889a700 1 -- 172.21.15.178:0/1641307998 >> 172.21.15.178:0/1641307998 conn(0x7fe2b40f0b70 msgr2=0x7fe2b40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:04:08.970 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:08.967+0000 7fe2b889a700 1 -- 172.21.15.178:0/1641307998 shutdown_connections 2024-02-11T17:04:08.970 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:08.967+0000 7fe2b889a700 1 -- 172.21.15.178:0/1641307998 wait complete. 2024-02-11T17:04:08.970 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:08.967+0000 7fe2b889a700 1 Processor -- start 2024-02-11T17:04:08.970 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:08.968+0000 7fe2b889a700 1 -- start start 2024-02-11T17:04:08.970 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:08.968+0000 7fe2b889a700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe2b40f5410 0x7fe2b41055d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:04:08.970 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:08.968+0000 7fe2b889a700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b4105ac0 con 0x7fe2b40f5410 2024-02-11T17:04:08.970 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:08.968+0000 7fe2b259c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe2b40f5410 0x7fe2b41055d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:04:08.970 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:08.968+0000 7fe2b259c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe2b40f5410 0x7fe2b41055d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:53994/0 (socket says 172.21.15.178:53994) 2024-02-11T17:04:08.970 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:08.968+0000 7fe2b259c700 1 -- 172.21.15.178:0/1093682916 learned_addr learned my addr 172.21.15.178:0/1093682916 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:04:08.971 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:08.968+0000 7fe2b259c700 1 -- 172.21.15.178:0/1093682916 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2a8005040 con 0x7fe2b40f5410 2024-02-11T17:04:08.971 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:08.968+0000 7fe2b259c700 1 --2- 172.21.15.178:0/1093682916 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe2b40f5410 0x7fe2b41055d0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7fe2a8013510 tx=0x7fe2a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:04:08.971 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:08.968+0000 7fe2a2ffd700 1 -- 172.21.15.178:0/1093682916 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe2a8009320 con 0x7fe2b40f5410 2024-02-11T17:04:08.971 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:08.969+0000 7fe2b889a700 1 -- 172.21.15.178:0/1093682916 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2b4105cc0 con 0x7fe2b40f5410 2024-02-11T17:04:08.971 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:08.969+0000 7fe2a2ffd700 1 -- 172.21.15.178:0/1093682916 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe2a8013f00 con 0x7fe2b40f5410 2024-02-11T17:04:08.971 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:08.969+0000 7fe2a2ffd700 1 -- 172.21.15.178:0/1093682916 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe2a8014d90 con 0x7fe2b40f5410 2024-02-11T17:04:08.971 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:08.969+0000 7fe2b889a700 1 -- 172.21.15.178:0/1093682916 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2b4106150 con 0x7fe2b40f5410 2024-02-11T17:04:08.972 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:08.969+0000 7fe2a2ffd700 1 -- 172.21.15.178:0/1093682916 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe2a80095f0 con 0x7fe2b40f5410 2024-02-11T17:04:08.972 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:08.970+0000 7fe2b889a700 1 -- 172.21.15.178:0/1093682916 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2b4045ad0 con 0x7fe2b40f5410 2024-02-11T17:04:08.972 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:08.970+0000 7fe2a2ffd700 1 --2- 172.21.15.178:0/1093682916 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fe29c02ff60 0x7fe29c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:04:08.972 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:08.970+0000 7fe2a2ffd700 1 -- 172.21.15.178:0/1093682916 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe2a8040fe0 con 0x7fe2b40f5410 2024-02-11T17:04:08.972 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:08.970+0000 7fe2b1d9b700 1 --2- 172.21.15.178:0/1093682916 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fe29c02ff60 0x7fe29c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:04:08.973 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:08.970+0000 7fe2b1d9b700 1 --2- 172.21.15.178:0/1093682916 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fe29c02ff60 0x7fe29c032420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fe2a40096a0 tx=0x7fe2a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:04:08.975 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:08.974+0000 7fe2a2ffd700 1 -- 172.21.15.178:0/1093682916 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe2a801d7f0 con 0x7fe2b40f5410 2024-02-11T17:04:09.181 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:09.180+0000 7fe2b889a700 1 -- 172.21.15.178:0/1093682916 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe2b4107000 con 0x7fe2b40f5410 2024-02-11T17:04:09.181 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:09.181+0000 7fe2a2ffd700 1 -- 172.21.15.178:0/1093682916 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe2a8010230 con 0x7fe2b40f5410 2024-02-11T17:04:09.182 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:04:09.182 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:04:09.183 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:09.182+0000 7fe2b889a700 1 -- 172.21.15.178:0/1093682916 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fe29c02ff60 msgr2=0x7fe29c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:04:09.183 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:09.182+0000 7fe2b889a700 1 --2- 172.21.15.178:0/1093682916 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fe29c02ff60 0x7fe29c032420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fe2a40096a0 tx=0x7fe2a4006b40).stop 2024-02-11T17:04:09.183 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:09.182+0000 7fe2b889a700 1 -- 172.21.15.178:0/1093682916 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe2b40f5410 msgr2=0x7fe2b41055d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:04:09.183 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:09.182+0000 7fe2b889a700 1 --2- 172.21.15.178:0/1093682916 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe2b40f5410 0x7fe2b41055d0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7fe2a8013510 tx=0x7fe2a800b040).stop 2024-02-11T17:04:09.183 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:09.182+0000 7fe2b889a700 1 -- 172.21.15.178:0/1093682916 shutdown_connections 2024-02-11T17:04:09.183 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:09.182+0000 7fe2b889a700 1 --2- 172.21.15.178:0/1093682916 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fe29c02ff60 0x7fe29c032420 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:04:09.183 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:09.182+0000 7fe2b889a700 1 --2- 172.21.15.178:0/1093682916 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe2b40f5410 0x7fe2b41055d0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:04:09.184 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:09.182+0000 7fe2b889a700 1 -- 172.21.15.178:0/1093682916 >> 172.21.15.178:0/1093682916 conn(0x7fe2b40f0b70 msgr2=0x7fe2b40f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:04:09.184 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:09.182+0000 7fe2b889a700 1 -- 172.21.15.178:0/1093682916 shutdown_connections 2024-02-11T17:04:09.184 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:09.182+0000 7fe2b889a700 1 -- 172.21.15.178:0/1093682916 wait complete. 2024-02-11T17:04:09.184 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:04:09.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:04:09 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:04:09.180911+0000 mon.smithi037 (mon.0) 181 : audit [DBG] from='client.? 172.21.15.178:0/1093682916' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:04:10.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:04:10 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:04:09.166391+0000 mgr.smithi037.eofnaq (mgr.14162) 124 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:04:10.861 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:04:10.861 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:04:12.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:04:12 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:04:11.166760+0000 mgr.smithi037.eofnaq (mgr.14162) 125 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:04:12.791 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:12.789+0000 7fdb7dab1700 1 -- 172.21.15.178:0/1616037172 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdb780f5410 msgr2=0x7fdb780f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:04:12.792 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:12.789+0000 7fdb7dab1700 1 --2- 172.21.15.178:0/1616037172 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdb780f5410 0x7fdb780f5830 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7fdb6c009fb0 tx=0x7fdb6c009b30).stop 2024-02-11T17:04:12.792 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:12.789+0000 7fdb7dab1700 1 -- 172.21.15.178:0/1616037172 shutdown_connections 2024-02-11T17:04:12.792 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:12.789+0000 7fdb7dab1700 1 --2- 172.21.15.178:0/1616037172 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdb780f5410 0x7fdb780f5830 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:04:12.792 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:12.789+0000 7fdb7dab1700 1 -- 172.21.15.178:0/1616037172 >> 172.21.15.178:0/1616037172 conn(0x7fdb780f0b70 msgr2=0x7fdb780f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:04:12.792 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:12.789+0000 7fdb7dab1700 1 -- 172.21.15.178:0/1616037172 shutdown_connections 2024-02-11T17:04:12.792 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:12.789+0000 7fdb7dab1700 1 -- 172.21.15.178:0/1616037172 wait complete. 2024-02-11T17:04:12.792 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:12.789+0000 7fdb7dab1700 1 Processor -- start 2024-02-11T17:04:12.792 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:12.790+0000 7fdb7dab1700 1 -- start start 2024-02-11T17:04:12.792 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:12.790+0000 7fdb7dab1700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdb780f5410 0x7fdb78109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:04:12.792 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:12.790+0000 7fdb7dab1700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb78060240 con 0x7fdb780f5410 2024-02-11T17:04:12.792 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:12.790+0000 7fdb777fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdb780f5410 0x7fdb78109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:04:12.793 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:12.790+0000 7fdb777fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdb780f5410 0x7fdb78109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:42680/0 (socket says 172.21.15.178:42680) 2024-02-11T17:04:12.793 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:12.790+0000 7fdb777fe700 1 -- 172.21.15.178:0/965800329 learned_addr learned my addr 172.21.15.178:0/965800329 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:04:12.794 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:12.790+0000 7fdb777fe700 1 -- 172.21.15.178:0/965800329 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb6c005040 con 0x7fdb780f5410 2024-02-11T17:04:12.794 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:12.790+0000 7fdb777fe700 1 --2- 172.21.15.178:0/965800329 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdb780f5410 0x7fdb78109430 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7fdb6c013510 tx=0x7fdb6c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:04:12.794 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:12.791+0000 7fdb5ffff700 1 -- 172.21.15.178:0/965800329 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdb6c009320 con 0x7fdb780f5410 2024-02-11T17:04:12.794 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:12.791+0000 7fdb7dab1700 1 -- 172.21.15.178:0/965800329 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb78060440 con 0x7fdb780f5410 2024-02-11T17:04:12.795 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:12.791+0000 7fdb5ffff700 1 -- 172.21.15.178:0/965800329 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdb6c013f00 con 0x7fdb780f5410 2024-02-11T17:04:12.795 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:12.791+0000 7fdb5ffff700 1 -- 172.21.15.178:0/965800329 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdb6c014d90 con 0x7fdb780f5410 2024-02-11T17:04:12.795 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:12.791+0000 7fdb7dab1700 1 -- 172.21.15.178:0/965800329 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb78109ae0 con 0x7fdb780f5410 2024-02-11T17:04:12.795 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:12.792+0000 7fdb5ffff700 1 -- 172.21.15.178:0/965800329 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdb6c0095f0 con 0x7fdb780f5410 2024-02-11T17:04:12.795 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:12.792+0000 7fdb7dab1700 1 -- 172.21.15.178:0/965800329 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb78045ad0 con 0x7fdb780f5410 2024-02-11T17:04:12.795 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:12.792+0000 7fdb5ffff700 1 --2- 172.21.15.178:0/965800329 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fdb6002ff60 0x7fdb60032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:04:12.795 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:12.792+0000 7fdb5ffff700 1 -- 172.21.15.178:0/965800329 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdb6c041000 con 0x7fdb780f5410 2024-02-11T17:04:12.795 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:12.792+0000 7fdb76ffd700 1 --2- 172.21.15.178:0/965800329 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fdb6002ff60 0x7fdb60032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:04:12.795 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:12.792+0000 7fdb76ffd700 1 --2- 172.21.15.178:0/965800329 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fdb6002ff60 0x7fdb60032420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fdb680096a0 tx=0x7fdb68006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:04:12.798 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:12.796+0000 7fdb5ffff700 1 -- 172.21.15.178:0/965800329 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdb6c014420 con 0x7fdb780f5410 2024-02-11T17:04:13.002 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:13.001+0000 7fdb7dab1700 1 -- 172.21.15.178:0/965800329 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdb780fae60 con 0x7fdb780f5410 2024-02-11T17:04:13.003 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:13.002+0000 7fdb5ffff700 1 -- 172.21.15.178:0/965800329 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdb6c01d7f0 con 0x7fdb780f5410 2024-02-11T17:04:13.003 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:04:13.004 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:04:13.005 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:13.003+0000 7fdb7dab1700 1 -- 172.21.15.178:0/965800329 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fdb6002ff60 msgr2=0x7fdb60032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:04:13.005 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:13.003+0000 7fdb7dab1700 1 --2- 172.21.15.178:0/965800329 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fdb6002ff60 0x7fdb60032420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fdb680096a0 tx=0x7fdb68006b40).stop 2024-02-11T17:04:13.005 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:13.003+0000 7fdb7dab1700 1 -- 172.21.15.178:0/965800329 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdb780f5410 msgr2=0x7fdb78109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:04:13.005 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:13.003+0000 7fdb7dab1700 1 --2- 172.21.15.178:0/965800329 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdb780f5410 0x7fdb78109430 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7fdb6c013510 tx=0x7fdb6c00b040).stop 2024-02-11T17:04:13.005 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:13.003+0000 7fdb7dab1700 1 -- 172.21.15.178:0/965800329 shutdown_connections 2024-02-11T17:04:13.005 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:13.003+0000 7fdb7dab1700 1 --2- 172.21.15.178:0/965800329 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fdb6002ff60 0x7fdb60032420 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:04:13.005 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:13.003+0000 7fdb7dab1700 1 --2- 172.21.15.178:0/965800329 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdb780f5410 0x7fdb78109430 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:04:13.005 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:13.003+0000 7fdb7dab1700 1 -- 172.21.15.178:0/965800329 >> 172.21.15.178:0/965800329 conn(0x7fdb780f0b70 msgr2=0x7fdb780f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:04:13.006 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:13.003+0000 7fdb7dab1700 1 -- 172.21.15.178:0/965800329 shutdown_connections 2024-02-11T17:04:13.006 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:13.003+0000 7fdb7dab1700 1 -- 172.21.15.178:0/965800329 wait complete. 2024-02-11T17:04:13.006 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:04:13.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:04:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:04:13.002894+0000 mon.smithi037 (mon.0) 182 : audit [DBG] from='client.? 172.21.15.178:0/965800329' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:04:14.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:04:14 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:04:13.167104+0000 mgr.smithi037.eofnaq (mgr.14162) 126 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:04:14.689 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:04:14.690 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:04:16.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:04:16 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:04:15.167361+0000 mgr.smithi037.eofnaq (mgr.14162) 127 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:04:16.610 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:16.608+0000 7f155be7e700 1 -- 172.21.15.178:0/2239447944 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f15540f5410 msgr2=0x7f15540f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:04:16.610 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:16.608+0000 7f155be7e700 1 --2- 172.21.15.178:0/2239447944 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f15540f5410 0x7f15540f5830 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f1544009fb0 tx=0x7f1544009b30).stop 2024-02-11T17:04:16.610 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:16.608+0000 7f155be7e700 1 -- 172.21.15.178:0/2239447944 shutdown_connections 2024-02-11T17:04:16.610 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:16.608+0000 7f155be7e700 1 --2- 172.21.15.178:0/2239447944 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f15540f5410 0x7f15540f5830 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:04:16.611 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:16.608+0000 7f155be7e700 1 -- 172.21.15.178:0/2239447944 >> 172.21.15.178:0/2239447944 conn(0x7f15540f0b70 msgr2=0x7f15540f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:04:16.611 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:16.608+0000 7f155be7e700 1 -- 172.21.15.178:0/2239447944 shutdown_connections 2024-02-11T17:04:16.611 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:16.608+0000 7f155be7e700 1 -- 172.21.15.178:0/2239447944 wait complete. 2024-02-11T17:04:16.612 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:16.609+0000 7f155be7e700 1 Processor -- start 2024-02-11T17:04:16.612 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:16.609+0000 7f155be7e700 1 -- start start 2024-02-11T17:04:16.612 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:16.609+0000 7f155be7e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f15540f5410 0x7f15541055d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:04:16.612 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:16.609+0000 7f155be7e700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1554105ac0 con 0x7f15540f5410 2024-02-11T17:04:16.612 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:16.610+0000 7f1559c1a700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f15540f5410 0x7f15541055d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:04:16.612 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:16.610+0000 7f1559c1a700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f15540f5410 0x7f15541055d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:42686/0 (socket says 172.21.15.178:42686) 2024-02-11T17:04:16.613 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:16.610+0000 7f1559c1a700 1 -- 172.21.15.178:0/3718352472 learned_addr learned my addr 172.21.15.178:0/3718352472 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:04:16.613 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:16.610+0000 7f1559c1a700 1 -- 172.21.15.178:0/3718352472 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1544005040 con 0x7f15540f5410 2024-02-11T17:04:16.613 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:16.610+0000 7f1559c1a700 1 --2- 172.21.15.178:0/3718352472 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f15540f5410 0x7f15541055d0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f1544013510 tx=0x7f154400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:04:16.613 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:16.611+0000 7f154a7fc700 1 -- 172.21.15.178:0/3718352472 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1544009320 con 0x7f15540f5410 2024-02-11T17:04:16.613 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:16.611+0000 7f154a7fc700 1 -- 172.21.15.178:0/3718352472 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1544013f00 con 0x7f15540f5410 2024-02-11T17:04:16.613 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:16.611+0000 7f155be7e700 1 -- 172.21.15.178:0/3718352472 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1554105cc0 con 0x7f15540f5410 2024-02-11T17:04:16.613 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:16.611+0000 7f154a7fc700 1 -- 172.21.15.178:0/3718352472 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1544014d90 con 0x7f15540f5410 2024-02-11T17:04:16.613 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:16.611+0000 7f155be7e700 1 -- 172.21.15.178:0/3718352472 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1554106150 con 0x7f15540f5410 2024-02-11T17:04:16.614 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:16.611+0000 7f154a7fc700 1 -- 172.21.15.178:0/3718352472 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f15440095f0 con 0x7f15540f5410 2024-02-11T17:04:16.614 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:16.611+0000 7f155be7e700 1 -- 172.21.15.178:0/3718352472 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1554045ad0 con 0x7f15540f5410 2024-02-11T17:04:16.614 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:16.612+0000 7f154a7fc700 1 --2- 172.21.15.178:0/3718352472 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f154002ffb0 0x7f1540032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:04:16.614 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:16.612+0000 7f154a7fc700 1 -- 172.21.15.178:0/3718352472 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1544041040 con 0x7f15540f5410 2024-02-11T17:04:16.614 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:16.612+0000 7f1559419700 1 --2- 172.21.15.178:0/3718352472 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f154002ffb0 0x7f1540032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:04:16.614 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:16.612+0000 7f1559419700 1 --2- 172.21.15.178:0/3718352472 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f154002ffb0 0x7f1540032470 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f15500096a0 tx=0x7f1550006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:04:16.618 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:16.617+0000 7f154a7fc700 1 -- 172.21.15.178:0/3718352472 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1544014420 con 0x7f15540f5410 2024-02-11T17:04:16.821 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:16.819+0000 7f155be7e700 1 -- 172.21.15.178:0/3718352472 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1554107000 con 0x7f15540f5410 2024-02-11T17:04:16.821 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:16.820+0000 7f154a7fc700 1 -- 172.21.15.178:0/3718352472 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f154401d7f0 con 0x7f15540f5410 2024-02-11T17:04:16.822 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:04:16.822 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:04:16.823 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:16.821+0000 7f155be7e700 1 -- 172.21.15.178:0/3718352472 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f154002ffb0 msgr2=0x7f1540032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:04:16.823 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:16.821+0000 7f155be7e700 1 --2- 172.21.15.178:0/3718352472 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f154002ffb0 0x7f1540032470 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f15500096a0 tx=0x7f1550006b40).stop 2024-02-11T17:04:16.823 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:16.821+0000 7f155be7e700 1 -- 172.21.15.178:0/3718352472 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f15540f5410 msgr2=0x7f15541055d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:04:16.823 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:16.821+0000 7f155be7e700 1 --2- 172.21.15.178:0/3718352472 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f15540f5410 0x7f15541055d0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f1544013510 tx=0x7f154400b040).stop 2024-02-11T17:04:16.823 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:16.821+0000 7f155be7e700 1 -- 172.21.15.178:0/3718352472 shutdown_connections 2024-02-11T17:04:16.823 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:16.821+0000 7f155be7e700 1 --2- 172.21.15.178:0/3718352472 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f154002ffb0 0x7f1540032470 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:04:16.824 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:16.821+0000 7f155be7e700 1 --2- 172.21.15.178:0/3718352472 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f15540f5410 0x7f15541055d0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:04:16.824 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:16.821+0000 7f155be7e700 1 -- 172.21.15.178:0/3718352472 >> 172.21.15.178:0/3718352472 conn(0x7f15540f0b70 msgr2=0x7f15540f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:04:16.824 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:16.822+0000 7f155be7e700 1 -- 172.21.15.178:0/3718352472 shutdown_connections 2024-02-11T17:04:16.824 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:16.822+0000 7f155be7e700 1 -- 172.21.15.178:0/3718352472 wait complete. 2024-02-11T17:04:16.824 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:04:17.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:04:17 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:04:16.821192+0000 mon.smithi037 (mon.0) 183 : audit [DBG] from='client.? 172.21.15.178:0/3718352472' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:04:18.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:04:18 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:04:17.167702+0000 mgr.smithi037.eofnaq (mgr.14162) 128 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:04:18.489 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:04:18.490 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:04:20.271 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:20.268+0000 7fb0bbdab700 1 -- 172.21.15.178:0/671281054 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb0b40f5410 msgr2=0x7fb0b40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:04:20.272 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:20.268+0000 7fb0bbdab700 1 --2- 172.21.15.178:0/671281054 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb0b40f5410 0x7fb0b40f5830 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7fb0a4009fb0 tx=0x7fb0a4009b30).stop 2024-02-11T17:04:20.272 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:20.268+0000 7fb0bbdab700 1 -- 172.21.15.178:0/671281054 shutdown_connections 2024-02-11T17:04:20.272 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:20.268+0000 7fb0bbdab700 1 --2- 172.21.15.178:0/671281054 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb0b40f5410 0x7fb0b40f5830 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:04:20.272 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:20.268+0000 7fb0bbdab700 1 -- 172.21.15.178:0/671281054 >> 172.21.15.178:0/671281054 conn(0x7fb0b40f0b70 msgr2=0x7fb0b40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:04:20.272 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:20.268+0000 7fb0bbdab700 1 -- 172.21.15.178:0/671281054 shutdown_connections 2024-02-11T17:04:20.272 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:20.268+0000 7fb0bbdab700 1 -- 172.21.15.178:0/671281054 wait complete. 2024-02-11T17:04:20.272 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:20.269+0000 7fb0bbdab700 1 Processor -- start 2024-02-11T17:04:20.272 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:20.269+0000 7fb0bbdab700 1 -- start start 2024-02-11T17:04:20.272 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:20.269+0000 7fb0bbdab700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb0b40f5410 0x7fb0b4109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:04:20.272 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:20.269+0000 7fb0bbdab700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b4060220 con 0x7fb0b40f5410 2024-02-11T17:04:20.272 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:20.269+0000 7fb0b9b47700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb0b40f5410 0x7fb0b4109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:04:20.272 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:20.269+0000 7fb0b9b47700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb0b40f5410 0x7fb0b4109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:54202/0 (socket says 172.21.15.178:54202) 2024-02-11T17:04:20.273 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:20.269+0000 7fb0b9b47700 1 -- 172.21.15.178:0/1625670174 learned_addr learned my addr 172.21.15.178:0/1625670174 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:04:20.273 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:20.269+0000 7fb0b9b47700 1 -- 172.21.15.178:0/1625670174 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0a4005040 con 0x7fb0b40f5410 2024-02-11T17:04:20.273 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:20.269+0000 7fb0b9b47700 1 --2- 172.21.15.178:0/1625670174 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb0b40f5410 0x7fb0b4109450 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7fb0a4013510 tx=0x7fb0a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:04:20.273 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:20.270+0000 7fb0aa7fc700 1 -- 172.21.15.178:0/1625670174 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb0a4009320 con 0x7fb0b40f5410 2024-02-11T17:04:20.273 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:20.270+0000 7fb0bbdab700 1 -- 172.21.15.178:0/1625670174 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0b4060420 con 0x7fb0b40f5410 2024-02-11T17:04:20.273 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:20.270+0000 7fb0aa7fc700 1 -- 172.21.15.178:0/1625670174 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb0a4013f00 con 0x7fb0b40f5410 2024-02-11T17:04:20.273 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:20.270+0000 7fb0aa7fc700 1 -- 172.21.15.178:0/1625670174 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb0a4014d90 con 0x7fb0b40f5410 2024-02-11T17:04:20.273 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:20.270+0000 7fb0bbdab700 1 -- 172.21.15.178:0/1625670174 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0b4109b00 con 0x7fb0b40f5410 2024-02-11T17:04:20.274 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:20.270+0000 7fb0aa7fc700 1 -- 172.21.15.178:0/1625670174 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb0a40095f0 con 0x7fb0b40f5410 2024-02-11T17:04:20.274 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:20.270+0000 7fb0bbdab700 1 -- 172.21.15.178:0/1625670174 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0b4059100 con 0x7fb0b40f5410 2024-02-11T17:04:20.274 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:20.271+0000 7fb0aa7fc700 1 --2- 172.21.15.178:0/1625670174 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb0a002ff60 0x7fb0a0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:04:20.274 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:20.271+0000 7fb0aa7fc700 1 -- 172.21.15.178:0/1625670174 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb0a4041000 con 0x7fb0b40f5410 2024-02-11T17:04:20.274 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:20.271+0000 7fb0b9346700 1 --2- 172.21.15.178:0/1625670174 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb0a002ff60 0x7fb0a0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:04:20.274 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:20.271+0000 7fb0b9346700 1 --2- 172.21.15.178:0/1625670174 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb0a002ff60 0x7fb0a0032420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7fb0b00096a0 tx=0x7fb0b0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:04:20.277 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:20.275+0000 7fb0aa7fc700 1 -- 172.21.15.178:0/1625670174 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb0a4014420 con 0x7fb0b40f5410 2024-02-11T17:04:20.482 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:20.480+0000 7fb0bbdab700 1 -- 172.21.15.178:0/1625670174 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb0b4045ad0 con 0x7fb0b40f5410 2024-02-11T17:04:20.482 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:20.481+0000 7fb0aa7fc700 1 -- 172.21.15.178:0/1625670174 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb0a4010230 con 0x7fb0b40f5410 2024-02-11T17:04:20.483 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:04:20.483 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:04:20.485 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:20.483+0000 7fb0bbdab700 1 -- 172.21.15.178:0/1625670174 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb0a002ff60 msgr2=0x7fb0a0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:04:20.485 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:20.483+0000 7fb0bbdab700 1 --2- 172.21.15.178:0/1625670174 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb0a002ff60 0x7fb0a0032420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7fb0b00096a0 tx=0x7fb0b0006b40).stop 2024-02-11T17:04:20.485 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:20.483+0000 7fb0bbdab700 1 -- 172.21.15.178:0/1625670174 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb0b40f5410 msgr2=0x7fb0b4109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:04:20.485 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:20.483+0000 7fb0bbdab700 1 --2- 172.21.15.178:0/1625670174 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb0b40f5410 0x7fb0b4109450 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7fb0a4013510 tx=0x7fb0a400b040).stop 2024-02-11T17:04:20.485 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:20.483+0000 7fb0bbdab700 1 -- 172.21.15.178:0/1625670174 shutdown_connections 2024-02-11T17:04:20.486 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:20.483+0000 7fb0bbdab700 1 --2- 172.21.15.178:0/1625670174 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb0a002ff60 0x7fb0a0032420 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:04:20.486 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:20.483+0000 7fb0bbdab700 1 --2- 172.21.15.178:0/1625670174 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb0b40f5410 0x7fb0b4109450 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:04:20.486 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:20.483+0000 7fb0bbdab700 1 -- 172.21.15.178:0/1625670174 >> 172.21.15.178:0/1625670174 conn(0x7fb0b40f0b70 msgr2=0x7fb0b40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:04:20.486 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:20.483+0000 7fb0bbdab700 1 -- 172.21.15.178:0/1625670174 shutdown_connections 2024-02-11T17:04:20.486 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:20.483+0000 7fb0bbdab700 1 -- 172.21.15.178:0/1625670174 wait complete. 2024-02-11T17:04:20.486 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:04:20.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:04:20 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:04:19.168003+0000 mgr.smithi037.eofnaq (mgr.14162) 129 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:04:21.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:04:21 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:04:20.482470+0000 mon.smithi037 (mon.0) 184 : audit [DBG] from='client.? 172.21.15.178:0/1625670174' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:04:22.144 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:04:22.144 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:04:22.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:04:22 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:04:21.168272+0000 mgr.smithi037.eofnaq (mgr.14162) 130 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:04:24.010 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:24.008+0000 7f74312a9700 1 -- 172.21.15.178:0/2924816168 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f742c0f6510 msgr2=0x7f742c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:04:24.010 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:24.008+0000 7f74312a9700 1 --2- 172.21.15.178:0/2924816168 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f742c0f6510 0x7f742c0f6930 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f7420009fb0 tx=0x7f7420009b30).stop 2024-02-11T17:04:24.011 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:24.009+0000 7f74312a9700 1 -- 172.21.15.178:0/2924816168 shutdown_connections 2024-02-11T17:04:24.011 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:24.009+0000 7f74312a9700 1 --2- 172.21.15.178:0/2924816168 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f742c0f6510 0x7f742c0f6930 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:04:24.011 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:24.009+0000 7f74312a9700 1 -- 172.21.15.178:0/2924816168 >> 172.21.15.178:0/2924816168 conn(0x7f742c0f1c70 msgr2=0x7f742c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:04:24.011 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:24.009+0000 7f74312a9700 1 -- 172.21.15.178:0/2924816168 shutdown_connections 2024-02-11T17:04:24.011 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:24.009+0000 7f74312a9700 1 -- 172.21.15.178:0/2924816168 wait complete. 2024-02-11T17:04:24.013 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:24.009+0000 7f74312a9700 1 Processor -- start 2024-02-11T17:04:24.013 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:24.009+0000 7f74312a9700 1 -- start start 2024-02-11T17:04:24.013 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:24.009+0000 7f74312a9700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f742c0f6510 0x7f742c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:04:24.013 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:24.009+0000 7f74312a9700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f742c106c20 con 0x7f742c0f6510 2024-02-11T17:04:24.013 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:24.010+0000 7f742affd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f742c0f6510 0x7f742c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:04:24.013 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:24.010+0000 7f742affd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f742c0f6510 0x7f742c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:54226/0 (socket says 172.21.15.178:54226) 2024-02-11T17:04:24.014 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:24.010+0000 7f742affd700 1 -- 172.21.15.178:0/2014425250 learned_addr learned my addr 172.21.15.178:0/2014425250 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:04:24.014 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:24.010+0000 7f742affd700 1 -- 172.21.15.178:0/2014425250 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7420005040 con 0x7f742c0f6510 2024-02-11T17:04:24.014 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:24.010+0000 7f742affd700 1 --2- 172.21.15.178:0/2014425250 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f742c0f6510 0x7f742c106730 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f7420013510 tx=0x7f742000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:04:24.014 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:24.010+0000 7f74137fe700 1 -- 172.21.15.178:0/2014425250 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7420009320 con 0x7f742c0f6510 2024-02-11T17:04:24.014 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:24.010+0000 7f74312a9700 1 -- 172.21.15.178:0/2014425250 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f742c106e20 con 0x7f742c0f6510 2024-02-11T17:04:24.014 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:24.010+0000 7f74137fe700 1 -- 172.21.15.178:0/2014425250 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7420013f00 con 0x7f742c0f6510 2024-02-11T17:04:24.015 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:24.010+0000 7f74137fe700 1 -- 172.21.15.178:0/2014425250 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7420014d90 con 0x7f742c0f6510 2024-02-11T17:04:24.015 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:24.010+0000 7f74312a9700 1 -- 172.21.15.178:0/2014425250 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f742c1072b0 con 0x7f742c0f6510 2024-02-11T17:04:24.015 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:24.011+0000 7f74137fe700 1 -- 172.21.15.178:0/2014425250 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7420014420 con 0x7f742c0f6510 2024-02-11T17:04:24.015 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:24.011+0000 7f74312a9700 1 -- 172.21.15.178:0/2014425250 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f742c045ad0 con 0x7f742c0f6510 2024-02-11T17:04:24.015 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:24.011+0000 7f74137fe700 1 --2- 172.21.15.178:0/2014425250 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f741402ff60 0x7f7414032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:04:24.015 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:24.011+0000 7f74137fe700 1 -- 172.21.15.178:0/2014425250 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f74200410c0 con 0x7f742c0f6510 2024-02-11T17:04:24.015 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:24.012+0000 7f742a7fc700 1 --2- 172.21.15.178:0/2014425250 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f741402ff60 0x7f7414032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:04:24.016 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:24.012+0000 7f742a7fc700 1 --2- 172.21.15.178:0/2014425250 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f741402ff60 0x7f7414032420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f741c0096a0 tx=0x7f741c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:04:24.018 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:24.016+0000 7f74137fe700 1 -- 172.21.15.178:0/2014425250 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f74200095f0 con 0x7f742c0f6510 2024-02-11T17:04:24.221 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:24.219+0000 7f74312a9700 1 -- 172.21.15.178:0/2014425250 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f742c0fbf60 con 0x7f742c0f6510 2024-02-11T17:04:24.222 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:24.220+0000 7f74137fe700 1 -- 172.21.15.178:0/2014425250 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f742001db70 con 0x7f742c0f6510 2024-02-11T17:04:24.222 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:04:24.222 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:04:24.223 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:24.222+0000 7f74312a9700 1 -- 172.21.15.178:0/2014425250 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f741402ff60 msgr2=0x7f7414032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:04:24.223 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:24.222+0000 7f74312a9700 1 --2- 172.21.15.178:0/2014425250 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f741402ff60 0x7f7414032420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f741c0096a0 tx=0x7f741c006b40).stop 2024-02-11T17:04:24.223 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:24.222+0000 7f74312a9700 1 -- 172.21.15.178:0/2014425250 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f742c0f6510 msgr2=0x7f742c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:04:24.224 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:24.222+0000 7f74312a9700 1 --2- 172.21.15.178:0/2014425250 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f742c0f6510 0x7f742c106730 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f7420013510 tx=0x7f742000b040).stop 2024-02-11T17:04:24.224 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:24.222+0000 7f74312a9700 1 -- 172.21.15.178:0/2014425250 shutdown_connections 2024-02-11T17:04:24.224 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:24.222+0000 7f74312a9700 1 --2- 172.21.15.178:0/2014425250 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f741402ff60 0x7f7414032420 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:04:24.224 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:24.222+0000 7f74312a9700 1 --2- 172.21.15.178:0/2014425250 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f742c0f6510 0x7f742c106730 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:04:24.224 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:24.222+0000 7f74312a9700 1 -- 172.21.15.178:0/2014425250 >> 172.21.15.178:0/2014425250 conn(0x7f742c0f1c70 msgr2=0x7f742c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:04:24.224 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:24.222+0000 7f74312a9700 1 -- 172.21.15.178:0/2014425250 shutdown_connections 2024-02-11T17:04:24.224 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:24.222+0000 7f74312a9700 1 -- 172.21.15.178:0/2014425250 wait complete. 2024-02-11T17:04:24.224 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:04:24.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:04:24 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:04:23.168592+0000 mgr.smithi037.eofnaq (mgr.14162) 131 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:04:24.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:04:24 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:04:24.221469+0000 mon.smithi037 (mon.0) 185 : audit [DBG] from='client.? 172.21.15.178:0/2014425250' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:04:25.818 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:04:25.819 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:04:26.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:04:26 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:04:25.168895+0000 mgr.smithi037.eofnaq (mgr.14162) 132 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:04:27.646 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:27.643+0000 7f35ef258700 1 -- 172.21.15.178:0/1661577866 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f35e80f62a0 msgr2=0x7f35e80f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:04:27.646 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:27.643+0000 7f35ef258700 1 --2- 172.21.15.178:0/1661577866 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f35e80f62a0 0x7f35e80f66c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f35d8009fb0 tx=0x7f35d8009b30).stop 2024-02-11T17:04:27.647 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:27.643+0000 7f35ef258700 1 -- 172.21.15.178:0/1661577866 shutdown_connections 2024-02-11T17:04:27.647 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:27.643+0000 7f35ef258700 1 --2- 172.21.15.178:0/1661577866 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f35e80f62a0 0x7f35e80f66c0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:04:27.647 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:27.643+0000 7f35ef258700 1 -- 172.21.15.178:0/1661577866 >> 172.21.15.178:0/1661577866 conn(0x7f35e80f1aa0 msgr2=0x7f35e80f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:04:27.647 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:27.643+0000 7f35ef258700 1 -- 172.21.15.178:0/1661577866 shutdown_connections 2024-02-11T17:04:27.647 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:27.643+0000 7f35ef258700 1 -- 172.21.15.178:0/1661577866 wait complete. 2024-02-11T17:04:27.647 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:27.644+0000 7f35ef258700 1 Processor -- start 2024-02-11T17:04:27.647 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:27.644+0000 7f35ef258700 1 -- start start 2024-02-11T17:04:27.647 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:27.644+0000 7f35ef258700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f35e80f62a0 0x7f35e81064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:04:27.647 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:27.644+0000 7f35ef258700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35e81069a0 con 0x7f35e80f62a0 2024-02-11T17:04:27.647 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:27.644+0000 7f35ee256700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f35e80f62a0 0x7f35e81064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:04:27.647 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:27.644+0000 7f35ee256700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f35e80f62a0 0x7f35e81064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:54240/0 (socket says 172.21.15.178:54240) 2024-02-11T17:04:27.647 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:27.644+0000 7f35ee256700 1 -- 172.21.15.178:0/1029519126 learned_addr learned my addr 172.21.15.178:0/1029519126 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:04:27.647 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:27.644+0000 7f35ee256700 1 -- 172.21.15.178:0/1029519126 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35d8005040 con 0x7f35e80f62a0 2024-02-11T17:04:27.647 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:27.645+0000 7f35ee256700 1 --2- 172.21.15.178:0/1029519126 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f35e80f62a0 0x7f35e81064b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f35d8013510 tx=0x7f35d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:04:27.648 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:27.645+0000 7f35deffd700 1 -- 172.21.15.178:0/1029519126 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35d8009320 con 0x7f35e80f62a0 2024-02-11T17:04:27.648 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:27.645+0000 7f35deffd700 1 -- 172.21.15.178:0/1029519126 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f35d8013f00 con 0x7f35e80f62a0 2024-02-11T17:04:27.648 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:27.645+0000 7f35ef258700 1 -- 172.21.15.178:0/1029519126 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35e8106ba0 con 0x7f35e80f62a0 2024-02-11T17:04:27.648 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:27.645+0000 7f35deffd700 1 -- 172.21.15.178:0/1029519126 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35d8014dc0 con 0x7f35e80f62a0 2024-02-11T17:04:27.648 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:27.645+0000 7f35ef258700 1 -- 172.21.15.178:0/1029519126 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35e8107030 con 0x7f35e80f62a0 2024-02-11T17:04:27.649 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:27.646+0000 7f35deffd700 1 -- 172.21.15.178:0/1029519126 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f35d80095f0 con 0x7f35e80f62a0 2024-02-11T17:04:27.649 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:27.646+0000 7f35deffd700 1 --2- 172.21.15.178:0/1029519126 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f35d402ff60 0x7f35d4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:04:27.649 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:27.646+0000 7f35ef258700 1 -- 172.21.15.178:0/1029519126 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35e8045ad0 con 0x7f35e80f62a0 2024-02-11T17:04:27.649 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:27.646+0000 7f35deffd700 1 -- 172.21.15.178:0/1029519126 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f35d8041000 con 0x7f35e80f62a0 2024-02-11T17:04:27.649 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:27.646+0000 7f35eda55700 1 --2- 172.21.15.178:0/1029519126 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f35d402ff60 0x7f35d4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:04:27.649 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:27.646+0000 7f35eda55700 1 --2- 172.21.15.178:0/1029519126 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f35d402ff60 0x7f35d4032420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f35e40096a0 tx=0x7f35e4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:04:27.652 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:27.650+0000 7f35deffd700 1 -- 172.21.15.178:0/1029519126 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f35d8014420 con 0x7f35e80f62a0 2024-02-11T17:04:27.856 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:27.854+0000 7f35ef258700 1 -- 172.21.15.178:0/1029519126 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f35e8107ee0 con 0x7f35e80f62a0 2024-02-11T17:04:27.856 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:27.855+0000 7f35deffd700 1 -- 172.21.15.178:0/1029519126 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f35d802b020 con 0x7f35e80f62a0 2024-02-11T17:04:27.857 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:04:27.857 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:04:27.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:27.856+0000 7f35ef258700 1 -- 172.21.15.178:0/1029519126 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f35d402ff60 msgr2=0x7f35d4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:04:27.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:27.856+0000 7f35ef258700 1 --2- 172.21.15.178:0/1029519126 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f35d402ff60 0x7f35d4032420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f35e40096a0 tx=0x7f35e4006b40).stop 2024-02-11T17:04:27.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:27.856+0000 7f35ef258700 1 -- 172.21.15.178:0/1029519126 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f35e80f62a0 msgr2=0x7f35e81064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:04:27.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:27.856+0000 7f35ef258700 1 --2- 172.21.15.178:0/1029519126 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f35e80f62a0 0x7f35e81064b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f35d8013510 tx=0x7f35d800b040).stop 2024-02-11T17:04:27.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:27.856+0000 7f35ef258700 1 -- 172.21.15.178:0/1029519126 shutdown_connections 2024-02-11T17:04:27.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:27.856+0000 7f35ef258700 1 --2- 172.21.15.178:0/1029519126 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f35d402ff60 0x7f35d4032420 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:04:27.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:27.856+0000 7f35ef258700 1 --2- 172.21.15.178:0/1029519126 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f35e80f62a0 0x7f35e81064b0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:04:27.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:27.856+0000 7f35ef258700 1 -- 172.21.15.178:0/1029519126 >> 172.21.15.178:0/1029519126 conn(0x7f35e80f1aa0 msgr2=0x7f35e80f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:04:27.859 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:27.856+0000 7f35ef258700 1 -- 172.21.15.178:0/1029519126 shutdown_connections 2024-02-11T17:04:27.859 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:27.856+0000 7f35ef258700 1 -- 172.21.15.178:0/1029519126 wait complete. 2024-02-11T17:04:27.859 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:04:28.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:04:28 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:04:27.169189+0000 mgr.smithi037.eofnaq (mgr.14162) 133 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:04:28.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:04:28 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:04:27.856212+0000 mon.smithi037 (mon.0) 186 : audit [DBG] from='client.? 172.21.15.178:0/1029519126' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:04:29.538 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:04:29.539 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:04:30.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:04:30 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:04:29.169511+0000 mgr.smithi037.eofnaq (mgr.14162) 134 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:04:31.424 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:31.421+0000 7ff1ff806700 1 -- 172.21.15.178:0/1994080065 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff1f8060280 msgr2=0x7ff1f80f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:04:31.425 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:31.421+0000 7ff1ff806700 1 --2- 172.21.15.178:0/1994080065 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff1f8060280 0x7ff1f80f90f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7ff1f4009fb0 tx=0x7ff1f4009b30).stop 2024-02-11T17:04:31.425 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:31.421+0000 7ff1ff806700 1 -- 172.21.15.178:0/1994080065 shutdown_connections 2024-02-11T17:04:31.425 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:31.421+0000 7ff1ff806700 1 --2- 172.21.15.178:0/1994080065 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff1f8060280 0x7ff1f80f90f0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:04:31.425 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:31.421+0000 7ff1ff806700 1 -- 172.21.15.178:0/1994080065 >> 172.21.15.178:0/1994080065 conn(0x7ff1f80f0b70 msgr2=0x7ff1f80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:04:31.425 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:31.421+0000 7ff1ff806700 1 -- 172.21.15.178:0/1994080065 shutdown_connections 2024-02-11T17:04:31.425 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:31.421+0000 7ff1ff806700 1 -- 172.21.15.178:0/1994080065 wait complete. 2024-02-11T17:04:31.425 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:31.422+0000 7ff1ff806700 1 Processor -- start 2024-02-11T17:04:31.425 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:31.422+0000 7ff1ff806700 1 -- start start 2024-02-11T17:04:31.425 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:31.422+0000 7ff1ff806700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff1f8060280 0x7ff1f80f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:04:31.426 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:31.422+0000 7ff1ff806700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1f80f4ff0 con 0x7ff1f8060280 2024-02-11T17:04:31.426 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:31.422+0000 7ff1fd5a2700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff1f8060280 0x7ff1f80f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:04:31.426 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:31.422+0000 7ff1fd5a2700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff1f8060280 0x7ff1f80f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:47712/0 (socket says 172.21.15.178:47712) 2024-02-11T17:04:31.426 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:31.422+0000 7ff1fd5a2700 1 -- 172.21.15.178:0/1043262936 learned_addr learned my addr 172.21.15.178:0/1043262936 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:04:31.426 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:31.422+0000 7ff1fd5a2700 1 -- 172.21.15.178:0/1043262936 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1f4005040 con 0x7ff1f8060280 2024-02-11T17:04:31.426 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:31.422+0000 7ff1fd5a2700 1 --2- 172.21.15.178:0/1043262936 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff1f8060280 0x7ff1f80f6b70 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7ff1f4013510 tx=0x7ff1f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:04:31.426 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:31.423+0000 7ff1edffb700 1 -- 172.21.15.178:0/1043262936 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff1f4009320 con 0x7ff1f8060280 2024-02-11T17:04:31.426 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:31.423+0000 7ff1ff806700 1 -- 172.21.15.178:0/1043262936 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1f80f5190 con 0x7ff1f8060280 2024-02-11T17:04:31.426 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:31.423+0000 7ff1edffb700 1 -- 172.21.15.178:0/1043262936 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff1f4013f00 con 0x7ff1f8060280 2024-02-11T17:04:31.426 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:31.423+0000 7ff1edffb700 1 -- 172.21.15.178:0/1043262936 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff1f4014dc0 con 0x7ff1f8060280 2024-02-11T17:04:31.426 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:31.423+0000 7ff1ff806700 1 -- 172.21.15.178:0/1043262936 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1f80f5620 con 0x7ff1f8060280 2024-02-11T17:04:31.427 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:31.424+0000 7ff1edffb700 1 -- 172.21.15.178:0/1043262936 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff1f40095f0 con 0x7ff1f8060280 2024-02-11T17:04:31.427 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:31.424+0000 7ff1ff806700 1 -- 172.21.15.178:0/1043262936 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1f8059100 con 0x7ff1f8060280 2024-02-11T17:04:31.428 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:31.424+0000 7ff1edffb700 1 --2- 172.21.15.178:0/1043262936 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7ff1e402ff60 0x7ff1e4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:04:31.428 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:31.424+0000 7ff1edffb700 1 -- 172.21.15.178:0/1043262936 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff1f4041040 con 0x7ff1f8060280 2024-02-11T17:04:31.428 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:31.424+0000 7ff1fcda1700 1 --2- 172.21.15.178:0/1043262936 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7ff1e402ff60 0x7ff1e4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:04:31.428 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:31.425+0000 7ff1fcda1700 1 --2- 172.21.15.178:0/1043262936 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7ff1e402ff60 0x7ff1e4032420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7ff1e80096a0 tx=0x7ff1e8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:04:31.430 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:31.429+0000 7ff1edffb700 1 -- 172.21.15.178:0/1043262936 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff1f4014420 con 0x7ff1f8060280 2024-02-11T17:04:31.636 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:31.634+0000 7ff1ff806700 1 -- 172.21.15.178:0/1043262936 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff1f80fbf50 con 0x7ff1f8060280 2024-02-11T17:04:31.637 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:31.635+0000 7ff1edffb700 1 -- 172.21.15.178:0/1043262936 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff1f4010230 con 0x7ff1f8060280 2024-02-11T17:04:31.637 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:04:31.637 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:04:31.639 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:31.637+0000 7ff1ff806700 1 -- 172.21.15.178:0/1043262936 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7ff1e402ff60 msgr2=0x7ff1e4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:04:31.639 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:31.637+0000 7ff1ff806700 1 --2- 172.21.15.178:0/1043262936 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7ff1e402ff60 0x7ff1e4032420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7ff1e80096a0 tx=0x7ff1e8006b40).stop 2024-02-11T17:04:31.639 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:31.637+0000 7ff1ff806700 1 -- 172.21.15.178:0/1043262936 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff1f8060280 msgr2=0x7ff1f80f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:04:31.639 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:31.637+0000 7ff1ff806700 1 --2- 172.21.15.178:0/1043262936 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff1f8060280 0x7ff1f80f6b70 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7ff1f4013510 tx=0x7ff1f400b040).stop 2024-02-11T17:04:31.639 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:31.637+0000 7ff1ff806700 1 -- 172.21.15.178:0/1043262936 shutdown_connections 2024-02-11T17:04:31.639 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:31.637+0000 7ff1ff806700 1 --2- 172.21.15.178:0/1043262936 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7ff1e402ff60 0x7ff1e4032420 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:04:31.639 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:31.637+0000 7ff1ff806700 1 --2- 172.21.15.178:0/1043262936 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff1f8060280 0x7ff1f80f6b70 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:04:31.639 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:31.637+0000 7ff1ff806700 1 -- 172.21.15.178:0/1043262936 >> 172.21.15.178:0/1043262936 conn(0x7ff1f80f0b70 msgr2=0x7ff1f80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:04:31.639 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:31.637+0000 7ff1ff806700 1 -- 172.21.15.178:0/1043262936 shutdown_connections 2024-02-11T17:04:31.640 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:31.637+0000 7ff1ff806700 1 -- 172.21.15.178:0/1043262936 wait complete. 2024-02-11T17:04:31.640 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:04:32.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:04:32 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:04:31.169910+0000 mgr.smithi037.eofnaq (mgr.14162) 135 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:04:32.684 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:04:32 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:04:31.636420+0000 mon.smithi037 (mon.0) 187 : audit [DBG] from='client.? 172.21.15.178:0/1043262936' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:04:33.385 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:04:33.385 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:04:34.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:04:34 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:04:33.170219+0000 mgr.smithi037.eofnaq (mgr.14162) 136 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:04:35.228 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:35.224+0000 7fbfb28cf700 1 -- 172.21.15.178:0/216542306 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbfac0f5410 msgr2=0x7fbfac0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:04:35.228 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:35.224+0000 7fbfb28cf700 1 --2- 172.21.15.178:0/216542306 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbfac0f5410 0x7fbfac0f5830 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7fbf9c009fb0 tx=0x7fbf9c009b30).stop 2024-02-11T17:04:35.228 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:35.224+0000 7fbfb28cf700 1 -- 172.21.15.178:0/216542306 shutdown_connections 2024-02-11T17:04:35.228 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:35.224+0000 7fbfb28cf700 1 --2- 172.21.15.178:0/216542306 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbfac0f5410 0x7fbfac0f5830 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:04:35.228 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:35.224+0000 7fbfb28cf700 1 -- 172.21.15.178:0/216542306 >> 172.21.15.178:0/216542306 conn(0x7fbfac0f0b70 msgr2=0x7fbfac0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:04:35.228 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:35.224+0000 7fbfb28cf700 1 -- 172.21.15.178:0/216542306 shutdown_connections 2024-02-11T17:04:35.228 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:35.224+0000 7fbfb28cf700 1 -- 172.21.15.178:0/216542306 wait complete. 2024-02-11T17:04:35.228 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:35.225+0000 7fbfb28cf700 1 Processor -- start 2024-02-11T17:04:35.229 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:35.225+0000 7fbfb28cf700 1 -- start start 2024-02-11T17:04:35.229 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:35.225+0000 7fbfb28cf700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbfac0f5410 0x7fbfac0fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:04:35.229 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:35.225+0000 7fbfb28cf700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfac0fb820 con 0x7fbfac0f5410 2024-02-11T17:04:35.229 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:35.226+0000 7fbfabfff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbfac0f5410 0x7fbfac0fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:04:35.229 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:35.226+0000 7fbfabfff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbfac0f5410 0x7fbfac0fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:47732/0 (socket says 172.21.15.178:47732) 2024-02-11T17:04:35.229 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:35.226+0000 7fbfabfff700 1 -- 172.21.15.178:0/773853343 learned_addr learned my addr 172.21.15.178:0/773853343 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:04:35.229 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:35.226+0000 7fbfabfff700 1 -- 172.21.15.178:0/773853343 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf9c005040 con 0x7fbfac0f5410 2024-02-11T17:04:35.229 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:35.226+0000 7fbfabfff700 1 --2- 172.21.15.178:0/773853343 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbfac0f5410 0x7fbfac0fd3a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7fbf9c0089f0 tx=0x7fbf9c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:04:35.229 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:35.226+0000 7fbfa8ff9700 1 -- 172.21.15.178:0/773853343 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf9c009320 con 0x7fbfac0f5410 2024-02-11T17:04:35.229 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:35.226+0000 7fbfa8ff9700 1 -- 172.21.15.178:0/773853343 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbf9c0136c0 con 0x7fbfac0f5410 2024-02-11T17:04:35.229 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:35.226+0000 7fbfb28cf700 1 -- 172.21.15.178:0/773853343 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfac0fb9c0 con 0x7fbfac0f5410 2024-02-11T17:04:35.229 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:35.226+0000 7fbfa8ff9700 1 -- 172.21.15.178:0/773853343 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf9c01ecb0 con 0x7fbfac0f5410 2024-02-11T17:04:35.229 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:35.226+0000 7fbfb28cf700 1 -- 172.21.15.178:0/773853343 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfac0fbe50 con 0x7fbfac0f5410 2024-02-11T17:04:35.230 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:35.227+0000 7fbfa8ff9700 1 -- 172.21.15.178:0/773853343 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbf9c014420 con 0x7fbfac0f5410 2024-02-11T17:04:35.230 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:35.227+0000 7fbfb28cf700 1 -- 172.21.15.178:0/773853343 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfac059100 con 0x7fbfac0f5410 2024-02-11T17:04:35.231 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:35.227+0000 7fbfa8ff9700 1 --2- 172.21.15.178:0/773853343 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fbf9802ffb0 0x7fbf98032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:04:35.231 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:35.227+0000 7fbfa8ff9700 1 -- 172.21.15.178:0/773853343 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbf9c041190 con 0x7fbfac0f5410 2024-02-11T17:04:35.231 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:35.227+0000 7fbfab7fe700 1 --2- 172.21.15.178:0/773853343 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fbf9802ffb0 0x7fbf98032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:04:35.231 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:35.228+0000 7fbfab7fe700 1 --2- 172.21.15.178:0/773853343 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fbf9802ffb0 0x7fbf98032470 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fbf940096a0 tx=0x7fbf94006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:04:35.233 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:35.232+0000 7fbfa8ff9700 1 -- 172.21.15.178:0/773853343 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbf9c00fd10 con 0x7fbfac0f5410 2024-02-11T17:04:35.439 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:35.437+0000 7fbfb28cf700 1 -- 172.21.15.178:0/773853343 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbfac0fae60 con 0x7fbfac0f5410 2024-02-11T17:04:35.439 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:35.438+0000 7fbfa8ff9700 1 -- 172.21.15.178:0/773853343 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbf9c00fb30 con 0x7fbfac0f5410 2024-02-11T17:04:35.440 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:04:35.440 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:04:35.441 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:35.439+0000 7fbfb28cf700 1 -- 172.21.15.178:0/773853343 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fbf9802ffb0 msgr2=0x7fbf98032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:04:35.441 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:35.439+0000 7fbfb28cf700 1 --2- 172.21.15.178:0/773853343 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fbf9802ffb0 0x7fbf98032470 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fbf940096a0 tx=0x7fbf94006b40).stop 2024-02-11T17:04:35.441 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:35.439+0000 7fbfb28cf700 1 -- 172.21.15.178:0/773853343 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbfac0f5410 msgr2=0x7fbfac0fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:04:35.441 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:35.439+0000 7fbfb28cf700 1 --2- 172.21.15.178:0/773853343 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbfac0f5410 0x7fbfac0fd3a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7fbf9c0089f0 tx=0x7fbf9c00b040).stop 2024-02-11T17:04:35.441 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:35.439+0000 7fbfb28cf700 1 -- 172.21.15.178:0/773853343 shutdown_connections 2024-02-11T17:04:35.442 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:35.439+0000 7fbfb28cf700 1 --2- 172.21.15.178:0/773853343 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fbf9802ffb0 0x7fbf98032470 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:04:35.442 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:35.439+0000 7fbfb28cf700 1 --2- 172.21.15.178:0/773853343 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbfac0f5410 0x7fbfac0fd3a0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:04:35.442 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:35.439+0000 7fbfb28cf700 1 -- 172.21.15.178:0/773853343 >> 172.21.15.178:0/773853343 conn(0x7fbfac0f0b70 msgr2=0x7fbfac05fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:04:35.442 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:35.439+0000 7fbfb28cf700 1 -- 172.21.15.178:0/773853343 shutdown_connections 2024-02-11T17:04:35.442 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:35.439+0000 7fbfb28cf700 1 -- 172.21.15.178:0/773853343 wait complete. 2024-02-11T17:04:35.442 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:04:36.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:04:36 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:04:35.170524+0000 mgr.smithi037.eofnaq (mgr.14162) 137 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:04:36.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:04:36 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:04:35.438972+0000 mon.smithi037 (mon.0) 188 : audit [DBG] from='client.? 172.21.15.178:0/773853343' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:04:37.101 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:04:37.101 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:04:38.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:04:38 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:04:37.170759+0000 mgr.smithi037.eofnaq (mgr.14162) 138 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:04:38.934 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:38.888+0000 7f0d74146700 1 -- 172.21.15.178:0/2859799482 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d6c0f5410 msgr2=0x7f0d6c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:04:38.934 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:38.888+0000 7f0d74146700 1 --2- 172.21.15.178:0/2859799482 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d6c0f5410 0x7f0d6c0f5830 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f0d5c009fb0 tx=0x7f0d5c009b30).stop 2024-02-11T17:04:38.934 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:38.888+0000 7f0d74146700 1 -- 172.21.15.178:0/2859799482 shutdown_connections 2024-02-11T17:04:38.934 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:38.888+0000 7f0d74146700 1 --2- 172.21.15.178:0/2859799482 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d6c0f5410 0x7f0d6c0f5830 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:04:38.934 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:38.888+0000 7f0d74146700 1 -- 172.21.15.178:0/2859799482 >> 172.21.15.178:0/2859799482 conn(0x7f0d6c0f0b70 msgr2=0x7f0d6c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:04:38.934 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:38.888+0000 7f0d74146700 1 -- 172.21.15.178:0/2859799482 shutdown_connections 2024-02-11T17:04:38.934 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:38.888+0000 7f0d74146700 1 -- 172.21.15.178:0/2859799482 wait complete. 2024-02-11T17:04:38.934 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:38.889+0000 7f0d74146700 1 Processor -- start 2024-02-11T17:04:38.934 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:38.889+0000 7f0d74146700 1 -- start start 2024-02-11T17:04:38.934 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:38.889+0000 7f0d74146700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d6c0f5410 0x7f0d6c105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:04:38.934 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:38.889+0000 7f0d74146700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6c105b20 con 0x7f0d6c0f5410 2024-02-11T17:04:38.934 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:38.889+0000 7f0d71ee2700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d6c0f5410 0x7f0d6c105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:04:38.935 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:38.889+0000 7f0d71ee2700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d6c0f5410 0x7f0d6c105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:47746/0 (socket says 172.21.15.178:47746) 2024-02-11T17:04:38.935 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:38.889+0000 7f0d71ee2700 1 -- 172.21.15.178:0/3247764258 learned_addr learned my addr 172.21.15.178:0/3247764258 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:04:38.935 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:38.889+0000 7f0d71ee2700 1 -- 172.21.15.178:0/3247764258 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d5c005040 con 0x7f0d6c0f5410 2024-02-11T17:04:38.935 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:38.890+0000 7f0d71ee2700 1 --2- 172.21.15.178:0/3247764258 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d6c0f5410 0x7f0d6c105630 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f0d5c013510 tx=0x7f0d5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:04:38.935 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:38.890+0000 7f0d627fc700 1 -- 172.21.15.178:0/3247764258 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d5c009320 con 0x7f0d6c0f5410 2024-02-11T17:04:38.935 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:38.890+0000 7f0d74146700 1 -- 172.21.15.178:0/3247764258 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d6c105d20 con 0x7f0d6c0f5410 2024-02-11T17:04:38.935 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:38.890+0000 7f0d627fc700 1 -- 172.21.15.178:0/3247764258 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0d5c013f00 con 0x7f0d6c0f5410 2024-02-11T17:04:38.935 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:38.890+0000 7f0d627fc700 1 -- 172.21.15.178:0/3247764258 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d5c014d90 con 0x7f0d6c0f5410 2024-02-11T17:04:38.935 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:38.890+0000 7f0d74146700 1 -- 172.21.15.178:0/3247764258 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d6c1061b0 con 0x7f0d6c0f5410 2024-02-11T17:04:38.935 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:38.891+0000 7f0d627fc700 1 -- 172.21.15.178:0/3247764258 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0d5c0095f0 con 0x7f0d6c0f5410 2024-02-11T17:04:38.935 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:38.891+0000 7f0d74146700 1 -- 172.21.15.178:0/3247764258 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d6c045ad0 con 0x7f0d6c0f5410 2024-02-11T17:04:38.935 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:38.891+0000 7f0d627fc700 1 --2- 172.21.15.178:0/3247764258 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f0d5802ff60 0x7f0d58032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:04:38.935 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:38.891+0000 7f0d627fc700 1 -- 172.21.15.178:0/3247764258 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0d5c041080 con 0x7f0d6c0f5410 2024-02-11T17:04:38.935 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:38.891+0000 7f0d716e1700 1 --2- 172.21.15.178:0/3247764258 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f0d5802ff60 0x7f0d58032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:04:38.936 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:38.891+0000 7f0d716e1700 1 --2- 172.21.15.178:0/3247764258 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f0d5802ff60 0x7f0d58032420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f0d680096a0 tx=0x7f0d68006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:04:38.936 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:38.896+0000 7f0d627fc700 1 -- 172.21.15.178:0/3247764258 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0d5c014420 con 0x7f0d6c0f5410 2024-02-11T17:04:39.101 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:39.099+0000 7f0d74146700 1 -- 172.21.15.178:0/3247764258 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0d6c0fae60 con 0x7f0d6c0f5410 2024-02-11T17:04:39.101 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:39.100+0000 7f0d627fc700 1 -- 172.21.15.178:0/3247764258 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0d5c010230 con 0x7f0d6c0f5410 2024-02-11T17:04:39.102 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:04:39.102 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:04:39.103 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:39.102+0000 7f0d74146700 1 -- 172.21.15.178:0/3247764258 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f0d5802ff60 msgr2=0x7f0d58032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:04:39.104 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:39.102+0000 7f0d74146700 1 --2- 172.21.15.178:0/3247764258 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f0d5802ff60 0x7f0d58032420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f0d680096a0 tx=0x7f0d68006b40).stop 2024-02-11T17:04:39.104 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:39.102+0000 7f0d74146700 1 -- 172.21.15.178:0/3247764258 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d6c0f5410 msgr2=0x7f0d6c105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:04:39.104 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:39.102+0000 7f0d74146700 1 --2- 172.21.15.178:0/3247764258 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d6c0f5410 0x7f0d6c105630 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f0d5c013510 tx=0x7f0d5c00b040).stop 2024-02-11T17:04:39.104 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:39.102+0000 7f0d74146700 1 -- 172.21.15.178:0/3247764258 shutdown_connections 2024-02-11T17:04:39.104 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:39.102+0000 7f0d74146700 1 --2- 172.21.15.178:0/3247764258 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f0d5802ff60 0x7f0d58032420 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:04:39.104 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:39.102+0000 7f0d74146700 1 --2- 172.21.15.178:0/3247764258 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d6c0f5410 0x7f0d6c105630 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:04:39.104 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:39.102+0000 7f0d74146700 1 -- 172.21.15.178:0/3247764258 >> 172.21.15.178:0/3247764258 conn(0x7f0d6c0f0b70 msgr2=0x7f0d6c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:04:39.104 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:39.102+0000 7f0d74146700 1 -- 172.21.15.178:0/3247764258 shutdown_connections 2024-02-11T17:04:39.104 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:39.102+0000 7f0d74146700 1 -- 172.21.15.178:0/3247764258 wait complete. 2024-02-11T17:04:39.104 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:04:39.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:04:39 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:04:39.101506+0000 mon.smithi037 (mon.0) 189 : audit [DBG] from='client.? 172.21.15.178:0/3247764258' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:04:40.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:04:40 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:04:39.171036+0000 mgr.smithi037.eofnaq (mgr.14162) 139 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:04:40.741 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:04:40.742 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:04:42.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:04:42 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:04:41.171347+0000 mgr.smithi037.eofnaq (mgr.14162) 140 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:04:42.704 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:42.701+0000 7fd3234a9700 1 -- 172.21.15.178:0/2643128737 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd31c0f5410 msgr2=0x7fd31c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:04:42.704 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:42.701+0000 7fd3234a9700 1 --2- 172.21.15.178:0/2643128737 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd31c0f5410 0x7fd31c0f5830 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7fd30c009fb0 tx=0x7fd30c009b30).stop 2024-02-11T17:04:42.704 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:42.701+0000 7fd3234a9700 1 -- 172.21.15.178:0/2643128737 shutdown_connections 2024-02-11T17:04:42.704 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:42.701+0000 7fd3234a9700 1 --2- 172.21.15.178:0/2643128737 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd31c0f5410 0x7fd31c0f5830 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:04:42.704 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:42.701+0000 7fd3234a9700 1 -- 172.21.15.178:0/2643128737 >> 172.21.15.178:0/2643128737 conn(0x7fd31c0f0b70 msgr2=0x7fd31c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:04:42.705 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:42.702+0000 7fd3234a9700 1 -- 172.21.15.178:0/2643128737 shutdown_connections 2024-02-11T17:04:42.705 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:42.702+0000 7fd3234a9700 1 -- 172.21.15.178:0/2643128737 wait complete. 2024-02-11T17:04:42.705 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:42.702+0000 7fd3234a9700 1 Processor -- start 2024-02-11T17:04:42.705 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:42.702+0000 7fd3234a9700 1 -- start start 2024-02-11T17:04:42.705 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:42.702+0000 7fd3234a9700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd31c0f5410 0x7fd31c108920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:04:42.705 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:42.702+0000 7fd3234a9700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd31c108e10 con 0x7fd31c0f5410 2024-02-11T17:04:42.705 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:42.702+0000 7fd321245700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd31c0f5410 0x7fd31c108920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:04:42.705 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:42.702+0000 7fd321245700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd31c0f5410 0x7fd31c108920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:44364/0 (socket says 172.21.15.178:44364) 2024-02-11T17:04:42.705 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:42.702+0000 7fd321245700 1 -- 172.21.15.178:0/18325794 learned_addr learned my addr 172.21.15.178:0/18325794 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:04:42.705 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:42.703+0000 7fd321245700 1 -- 172.21.15.178:0/18325794 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd30c005040 con 0x7fd31c0f5410 2024-02-11T17:04:42.705 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:42.703+0000 7fd321245700 1 --2- 172.21.15.178:0/18325794 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd31c0f5410 0x7fd31c108920 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7fd30c013510 tx=0x7fd30c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:04:42.707 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:42.703+0000 7fd311ffb700 1 -- 172.21.15.178:0/18325794 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd30c009320 con 0x7fd31c0f5410 2024-02-11T17:04:42.707 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:42.703+0000 7fd3234a9700 1 -- 172.21.15.178:0/18325794 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd31c109010 con 0x7fd31c0f5410 2024-02-11T17:04:42.707 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:42.703+0000 7fd311ffb700 1 -- 172.21.15.178:0/18325794 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd30c013f00 con 0x7fd31c0f5410 2024-02-11T17:04:42.707 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:42.703+0000 7fd311ffb700 1 -- 172.21.15.178:0/18325794 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd30c014d90 con 0x7fd31c0f5410 2024-02-11T17:04:42.707 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:42.703+0000 7fd3234a9700 1 -- 172.21.15.178:0/18325794 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd31c1094a0 con 0x7fd31c0f5410 2024-02-11T17:04:42.707 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:42.704+0000 7fd311ffb700 1 -- 172.21.15.178:0/18325794 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd30c0095f0 con 0x7fd31c0f5410 2024-02-11T17:04:42.707 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:42.704+0000 7fd3234a9700 1 -- 172.21.15.178:0/18325794 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd31c045ad0 con 0x7fd31c0f5410 2024-02-11T17:04:42.707 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:42.704+0000 7fd311ffb700 1 --2- 172.21.15.178:0/18325794 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fd30802ff60 0x7fd308032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:04:42.707 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:42.704+0000 7fd311ffb700 1 -- 172.21.15.178:0/18325794 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd30c040fc0 con 0x7fd31c0f5410 2024-02-11T17:04:42.707 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:42.705+0000 7fd320a44700 1 --2- 172.21.15.178:0/18325794 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fd30802ff60 0x7fd308032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:04:42.707 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:42.705+0000 7fd320a44700 1 --2- 172.21.15.178:0/18325794 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fd30802ff60 0x7fd308032420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fd3180096a0 tx=0x7fd318006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:04:42.710 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:42.709+0000 7fd311ffb700 1 -- 172.21.15.178:0/18325794 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd30c014420 con 0x7fd31c0f5410 2024-02-11T17:04:42.915 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:42.913+0000 7fd3234a9700 1 -- 172.21.15.178:0/18325794 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd31c10a2f0 con 0x7fd31c0f5410 2024-02-11T17:04:42.915 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:42.914+0000 7fd311ffb700 1 -- 172.21.15.178:0/18325794 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd30c010230 con 0x7fd31c0f5410 2024-02-11T17:04:42.916 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:04:42.916 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:04:42.917 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:42.915+0000 7fd3234a9700 1 -- 172.21.15.178:0/18325794 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fd30802ff60 msgr2=0x7fd308032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:04:42.917 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:42.915+0000 7fd3234a9700 1 --2- 172.21.15.178:0/18325794 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fd30802ff60 0x7fd308032420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fd3180096a0 tx=0x7fd318006b40).stop 2024-02-11T17:04:42.917 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:42.915+0000 7fd3234a9700 1 -- 172.21.15.178:0/18325794 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd31c0f5410 msgr2=0x7fd31c108920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:04:42.917 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:42.915+0000 7fd3234a9700 1 --2- 172.21.15.178:0/18325794 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd31c0f5410 0x7fd31c108920 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7fd30c013510 tx=0x7fd30c00b040).stop 2024-02-11T17:04:42.917 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:42.915+0000 7fd3234a9700 1 -- 172.21.15.178:0/18325794 shutdown_connections 2024-02-11T17:04:42.917 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:42.915+0000 7fd3234a9700 1 --2- 172.21.15.178:0/18325794 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fd30802ff60 0x7fd308032420 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:04:42.918 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:42.915+0000 7fd3234a9700 1 --2- 172.21.15.178:0/18325794 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd31c0f5410 0x7fd31c108920 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:04:42.918 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:42.915+0000 7fd3234a9700 1 -- 172.21.15.178:0/18325794 >> 172.21.15.178:0/18325794 conn(0x7fd31c0f0b70 msgr2=0x7fd31c05fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:04:42.918 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:42.915+0000 7fd3234a9700 1 -- 172.21.15.178:0/18325794 shutdown_connections 2024-02-11T17:04:42.918 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:42.915+0000 7fd3234a9700 1 -- 172.21.15.178:0/18325794 wait complete. 2024-02-11T17:04:42.918 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:04:43.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:04:43 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:04:42.915134+0000 mon.smithi037 (mon.0) 190 : audit [DBG] from='client.? 172.21.15.178:0/18325794' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:04:44.621 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:04:44.621 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:04:44.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:04:44 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:04:43.171665+0000 mgr.smithi037.eofnaq (mgr.14162) 141 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:04:46.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:04:46 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:04:45.172006+0000 mgr.smithi037.eofnaq (mgr.14162) 142 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:04:46.540 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:46.538+0000 7ff20139d700 1 -- 172.21.15.178:0/75650926 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff1fc0f4b80 msgr2=0x7ff1fc0f4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:04:46.540 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:46.538+0000 7ff20139d700 1 --2- 172.21.15.178:0/75650926 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff1fc0f4b80 0x7ff1fc0f4fa0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7ff1f0009fb0 tx=0x7ff1f0009b30).stop 2024-02-11T17:04:46.540 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:46.538+0000 7ff20139d700 1 -- 172.21.15.178:0/75650926 shutdown_connections 2024-02-11T17:04:46.540 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:46.538+0000 7ff20139d700 1 --2- 172.21.15.178:0/75650926 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff1fc0f4b80 0x7ff1fc0f4fa0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:04:46.541 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:46.538+0000 7ff20139d700 1 -- 172.21.15.178:0/75650926 >> 172.21.15.178:0/75650926 conn(0x7ff1fc0f02e0 msgr2=0x7ff1fc0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:04:46.541 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:46.538+0000 7ff20139d700 1 -- 172.21.15.178:0/75650926 shutdown_connections 2024-02-11T17:04:46.541 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:46.538+0000 7ff20139d700 1 -- 172.21.15.178:0/75650926 wait complete. 2024-02-11T17:04:46.541 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:46.538+0000 7ff20139d700 1 Processor -- start 2024-02-11T17:04:46.541 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:46.538+0000 7ff20139d700 1 -- start start 2024-02-11T17:04:46.541 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:46.538+0000 7ff20139d700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff1fc0f4b80 0x7ff1fc106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:04:46.541 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:46.539+0000 7ff20139d700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1fc106bf0 con 0x7ff1fc0f4b80 2024-02-11T17:04:46.541 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:46.539+0000 7ff1faffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff1fc0f4b80 0x7ff1fc106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:04:46.541 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:46.539+0000 7ff1faffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff1fc0f4b80 0x7ff1fc106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:44394/0 (socket says 172.21.15.178:44394) 2024-02-11T17:04:46.541 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:46.539+0000 7ff1faffd700 1 -- 172.21.15.178:0/3299728909 learned_addr learned my addr 172.21.15.178:0/3299728909 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:04:46.542 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:46.539+0000 7ff1faffd700 1 -- 172.21.15.178:0/3299728909 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1f0005040 con 0x7ff1fc0f4b80 2024-02-11T17:04:46.542 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:46.539+0000 7ff1faffd700 1 --2- 172.21.15.178:0/3299728909 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff1fc0f4b80 0x7ff1fc106700 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7ff1f0013510 tx=0x7ff1f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:04:46.543 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:46.539+0000 7ff1e37fe700 1 -- 172.21.15.178:0/3299728909 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff1f0009320 con 0x7ff1fc0f4b80 2024-02-11T17:04:46.543 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:46.539+0000 7ff20139d700 1 -- 172.21.15.178:0/3299728909 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1fc106df0 con 0x7ff1fc0f4b80 2024-02-11T17:04:46.543 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:46.539+0000 7ff1e37fe700 1 -- 172.21.15.178:0/3299728909 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff1f0013f00 con 0x7ff1fc0f4b80 2024-02-11T17:04:46.543 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:46.540+0000 7ff1e37fe700 1 -- 172.21.15.178:0/3299728909 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff1f0014d90 con 0x7ff1fc0f4b80 2024-02-11T17:04:46.543 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:46.540+0000 7ff20139d700 1 -- 172.21.15.178:0/3299728909 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1fc107280 con 0x7ff1fc0f4b80 2024-02-11T17:04:46.543 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:46.540+0000 7ff1e37fe700 1 -- 172.21.15.178:0/3299728909 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff1f0014420 con 0x7ff1fc0f4b80 2024-02-11T17:04:46.543 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:46.541+0000 7ff20139d700 1 -- 172.21.15.178:0/3299728909 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1fc045ad0 con 0x7ff1fc0f4b80 2024-02-11T17:04:46.543 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:46.541+0000 7ff1e37fe700 1 --2- 172.21.15.178:0/3299728909 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7ff1e402ff60 0x7ff1e4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:04:46.543 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:46.541+0000 7ff1e37fe700 1 -- 172.21.15.178:0/3299728909 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff1f0041000 con 0x7ff1fc0f4b80 2024-02-11T17:04:46.543 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:46.541+0000 7ff1fa7fc700 1 --2- 172.21.15.178:0/3299728909 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7ff1e402ff60 0x7ff1e4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:04:46.543 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:46.541+0000 7ff1fa7fc700 1 --2- 172.21.15.178:0/3299728909 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7ff1e402ff60 0x7ff1e4032420 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7ff1ec0096a0 tx=0x7ff1ec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:04:46.547 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:46.545+0000 7ff1e37fe700 1 -- 172.21.15.178:0/3299728909 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff1f00095f0 con 0x7ff1fc0f4b80 2024-02-11T17:04:46.751 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:46.749+0000 7ff20139d700 1 -- 172.21.15.178:0/3299728909 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff1fc0fa5d0 con 0x7ff1fc0f4b80 2024-02-11T17:04:46.751 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:46.750+0000 7ff1e37fe700 1 -- 172.21.15.178:0/3299728909 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff1f001db70 con 0x7ff1fc0f4b80 2024-02-11T17:04:46.752 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:04:46.752 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:04:46.753 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:46.751+0000 7ff20139d700 1 -- 172.21.15.178:0/3299728909 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7ff1e402ff60 msgr2=0x7ff1e4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:04:46.754 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:46.751+0000 7ff20139d700 1 --2- 172.21.15.178:0/3299728909 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7ff1e402ff60 0x7ff1e4032420 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7ff1ec0096a0 tx=0x7ff1ec006b40).stop 2024-02-11T17:04:46.754 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:46.751+0000 7ff20139d700 1 -- 172.21.15.178:0/3299728909 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff1fc0f4b80 msgr2=0x7ff1fc106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:04:46.754 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:46.751+0000 7ff20139d700 1 --2- 172.21.15.178:0/3299728909 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff1fc0f4b80 0x7ff1fc106700 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7ff1f0013510 tx=0x7ff1f000b040).stop 2024-02-11T17:04:46.754 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:46.752+0000 7ff20139d700 1 -- 172.21.15.178:0/3299728909 shutdown_connections 2024-02-11T17:04:46.754 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:46.752+0000 7ff20139d700 1 --2- 172.21.15.178:0/3299728909 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7ff1e402ff60 0x7ff1e4032420 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:04:46.754 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:46.752+0000 7ff20139d700 1 --2- 172.21.15.178:0/3299728909 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff1fc0f4b80 0x7ff1fc106700 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:04:46.754 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:46.752+0000 7ff20139d700 1 -- 172.21.15.178:0/3299728909 >> 172.21.15.178:0/3299728909 conn(0x7ff1fc0f02e0 msgr2=0x7ff1fc0f14e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:04:46.754 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:46.752+0000 7ff20139d700 1 -- 172.21.15.178:0/3299728909 shutdown_connections 2024-02-11T17:04:46.754 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:46.752+0000 7ff20139d700 1 -- 172.21.15.178:0/3299728909 wait complete. 2024-02-11T17:04:46.754 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:04:47.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:04:47 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:04:46.751049+0000 mon.smithi037 (mon.0) 191 : audit [DBG] from='client.? 172.21.15.178:0/3299728909' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:04:48.410 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:04:48.410 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:04:48.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:04:48 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:04:47.172380+0000 mgr.smithi037.eofnaq (mgr.14162) 143 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:04:50.278 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:50.275+0000 7fb06367d700 1 -- 172.21.15.178:0/3127478275 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb05c0f5410 msgr2=0x7fb05c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:04:50.278 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:50.275+0000 7fb06367d700 1 --2- 172.21.15.178:0/3127478275 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb05c0f5410 0x7fb05c0f5830 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7fb04c009fb0 tx=0x7fb04c009b30).stop 2024-02-11T17:04:50.279 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:50.275+0000 7fb06367d700 1 -- 172.21.15.178:0/3127478275 shutdown_connections 2024-02-11T17:04:50.279 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:50.275+0000 7fb06367d700 1 --2- 172.21.15.178:0/3127478275 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb05c0f5410 0x7fb05c0f5830 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:04:50.279 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:50.275+0000 7fb06367d700 1 -- 172.21.15.178:0/3127478275 >> 172.21.15.178:0/3127478275 conn(0x7fb05c0f0b70 msgr2=0x7fb05c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:04:50.279 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:50.275+0000 7fb06367d700 1 -- 172.21.15.178:0/3127478275 shutdown_connections 2024-02-11T17:04:50.279 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:50.275+0000 7fb06367d700 1 -- 172.21.15.178:0/3127478275 wait complete. 2024-02-11T17:04:50.279 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:50.276+0000 7fb06367d700 1 Processor -- start 2024-02-11T17:04:50.279 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:50.276+0000 7fb06367d700 1 -- start start 2024-02-11T17:04:50.279 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:50.276+0000 7fb06367d700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb05c0f5410 0x7fb05c105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:04:50.279 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:50.276+0000 7fb06367d700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb05c105b20 con 0x7fb05c0f5410 2024-02-11T17:04:50.279 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:50.276+0000 7fb061419700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb05c0f5410 0x7fb05c105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:04:50.279 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:50.276+0000 7fb061419700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb05c0f5410 0x7fb05c105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:47720/0 (socket says 172.21.15.178:47720) 2024-02-11T17:04:50.279 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:50.276+0000 7fb061419700 1 -- 172.21.15.178:0/751843642 learned_addr learned my addr 172.21.15.178:0/751843642 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:04:50.280 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:50.276+0000 7fb061419700 1 -- 172.21.15.178:0/751843642 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb04c005040 con 0x7fb05c0f5410 2024-02-11T17:04:50.280 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:50.276+0000 7fb061419700 1 --2- 172.21.15.178:0/751843642 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb05c0f5410 0x7fb05c105630 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7fb04c013510 tx=0x7fb04c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:04:50.280 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:50.277+0000 7fb051ffb700 1 -- 172.21.15.178:0/751843642 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb04c009320 con 0x7fb05c0f5410 2024-02-11T17:04:50.280 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:50.277+0000 7fb06367d700 1 -- 172.21.15.178:0/751843642 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb05c105d20 con 0x7fb05c0f5410 2024-02-11T17:04:50.280 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:50.277+0000 7fb051ffb700 1 -- 172.21.15.178:0/751843642 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb04c013f00 con 0x7fb05c0f5410 2024-02-11T17:04:50.280 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:50.277+0000 7fb051ffb700 1 -- 172.21.15.178:0/751843642 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb04c014d90 con 0x7fb05c0f5410 2024-02-11T17:04:50.280 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:50.277+0000 7fb06367d700 1 -- 172.21.15.178:0/751843642 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb05c1061b0 con 0x7fb05c0f5410 2024-02-11T17:04:50.281 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:50.278+0000 7fb051ffb700 1 -- 172.21.15.178:0/751843642 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb04c0263a0 con 0x7fb05c0f5410 2024-02-11T17:04:50.281 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:50.278+0000 7fb051ffb700 1 --2- 172.21.15.178:0/751843642 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb04802ff60 0x7fb048032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:04:50.281 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:50.278+0000 7fb051ffb700 1 -- 172.21.15.178:0/751843642 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb04c040ff0 con 0x7fb05c0f5410 2024-02-11T17:04:50.281 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:50.278+0000 7fb06367d700 1 -- 172.21.15.178:0/751843642 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb05c045ad0 con 0x7fb05c0f5410 2024-02-11T17:04:50.281 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:50.278+0000 7fb060c18700 1 --2- 172.21.15.178:0/751843642 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb04802ff60 0x7fb048032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:04:50.282 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:50.278+0000 7fb060c18700 1 --2- 172.21.15.178:0/751843642 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb04802ff60 0x7fb048032420 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fb0580096a0 tx=0x7fb058006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:04:50.284 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:50.283+0000 7fb051ffb700 1 -- 172.21.15.178:0/751843642 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb04c012920 con 0x7fb05c0f5410 2024-02-11T17:04:50.492 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:50.491+0000 7fb06367d700 1 -- 172.21.15.178:0/751843642 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb05c0fae60 con 0x7fb05c0f5410 2024-02-11T17:04:50.493 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:50.491+0000 7fb051ffb700 1 -- 172.21.15.178:0/751843642 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb04c02b020 con 0x7fb05c0f5410 2024-02-11T17:04:50.493 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:04:50.493 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:04:50.495 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:50.493+0000 7fb06367d700 1 -- 172.21.15.178:0/751843642 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb04802ff60 msgr2=0x7fb048032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:04:50.495 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:50.493+0000 7fb06367d700 1 --2- 172.21.15.178:0/751843642 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb04802ff60 0x7fb048032420 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fb0580096a0 tx=0x7fb058006b40).stop 2024-02-11T17:04:50.495 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:50.493+0000 7fb06367d700 1 -- 172.21.15.178:0/751843642 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb05c0f5410 msgr2=0x7fb05c105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:04:50.495 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:50.493+0000 7fb06367d700 1 --2- 172.21.15.178:0/751843642 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb05c0f5410 0x7fb05c105630 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7fb04c013510 tx=0x7fb04c00b040).stop 2024-02-11T17:04:50.495 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:50.493+0000 7fb06367d700 1 -- 172.21.15.178:0/751843642 shutdown_connections 2024-02-11T17:04:50.495 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:50.493+0000 7fb06367d700 1 --2- 172.21.15.178:0/751843642 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb04802ff60 0x7fb048032420 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:04:50.495 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:50.493+0000 7fb06367d700 1 --2- 172.21.15.178:0/751843642 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb05c0f5410 0x7fb05c105630 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:04:50.495 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:50.493+0000 7fb06367d700 1 -- 172.21.15.178:0/751843642 >> 172.21.15.178:0/751843642 conn(0x7fb05c0f0b70 msgr2=0x7fb05c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:04:50.495 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:50.493+0000 7fb06367d700 1 -- 172.21.15.178:0/751843642 shutdown_connections 2024-02-11T17:04:50.495 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:50.493+0000 7fb06367d700 1 -- 172.21.15.178:0/751843642 wait complete. 2024-02-11T17:04:50.496 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:04:50.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:04:50 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:04:49.172686+0000 mgr.smithi037.eofnaq (mgr.14162) 144 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:04:51.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:04:51 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:04:50.492725+0000 mon.smithi037 (mon.0) 192 : audit [DBG] from='client.? 172.21.15.178:0/751843642' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:04:52.155 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:04:52.155 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:04:52.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:04:52 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:04:51.173037+0000 mgr.smithi037.eofnaq (mgr.14162) 145 : cluster [DBG] pgmap v138: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:04:53.970 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:53.967+0000 7fe40c260700 1 -- 172.21.15.178:0/1945692544 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4040f6480 msgr2=0x7fe4040f68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:04:53.970 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:53.967+0000 7fe40c260700 1 --2- 172.21.15.178:0/1945692544 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4040f6480 0x7fe4040f68a0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7fe3f4009fb0 tx=0x7fe3f4009b30).stop 2024-02-11T17:04:53.970 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:53.968+0000 7fe40c260700 1 -- 172.21.15.178:0/1945692544 shutdown_connections 2024-02-11T17:04:53.970 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:53.968+0000 7fe40c260700 1 --2- 172.21.15.178:0/1945692544 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4040f6480 0x7fe4040f68a0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:04:53.970 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:53.968+0000 7fe40c260700 1 -- 172.21.15.178:0/1945692544 >> 172.21.15.178:0/1945692544 conn(0x7fe4040f1c00 msgr2=0x7fe4040f4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:04:53.970 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:53.968+0000 7fe40c260700 1 -- 172.21.15.178:0/1945692544 shutdown_connections 2024-02-11T17:04:53.971 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:53.968+0000 7fe40c260700 1 -- 172.21.15.178:0/1945692544 wait complete. 2024-02-11T17:04:53.971 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:53.968+0000 7fe40c260700 1 Processor -- start 2024-02-11T17:04:53.971 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:53.968+0000 7fe40c260700 1 -- start start 2024-02-11T17:04:53.971 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:53.969+0000 7fe40c260700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4040f6480 0x7fe4041066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:04:53.971 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:53.969+0000 7fe40c260700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe404106bc0 con 0x7fe4040f6480 2024-02-11T17:04:53.973 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:53.969+0000 7fe409ffc700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4040f6480 0x7fe4041066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:04:53.973 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:53.969+0000 7fe409ffc700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4040f6480 0x7fe4041066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:47734/0 (socket says 172.21.15.178:47734) 2024-02-11T17:04:53.973 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:53.969+0000 7fe409ffc700 1 -- 172.21.15.178:0/622631368 learned_addr learned my addr 172.21.15.178:0/622631368 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:04:53.973 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:53.969+0000 7fe409ffc700 1 -- 172.21.15.178:0/622631368 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3f4005040 con 0x7fe4040f6480 2024-02-11T17:04:53.973 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:53.969+0000 7fe409ffc700 1 --2- 172.21.15.178:0/622631368 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4040f6480 0x7fe4041066d0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7fe3f4013510 tx=0x7fe3f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:04:53.973 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:53.970+0000 7fe3fa7fc700 1 -- 172.21.15.178:0/622631368 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe3f4009320 con 0x7fe4040f6480 2024-02-11T17:04:53.973 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:53.970+0000 7fe3fa7fc700 1 -- 172.21.15.178:0/622631368 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe3f4013f00 con 0x7fe4040f6480 2024-02-11T17:04:53.973 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:53.970+0000 7fe40c260700 1 -- 172.21.15.178:0/622631368 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe404106dc0 con 0x7fe4040f6480 2024-02-11T17:04:53.973 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:53.970+0000 7fe3fa7fc700 1 -- 172.21.15.178:0/622631368 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe3f4014d90 con 0x7fe4040f6480 2024-02-11T17:04:53.973 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:53.970+0000 7fe40c260700 1 -- 172.21.15.178:0/622631368 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe404107250 con 0x7fe4040f6480 2024-02-11T17:04:53.973 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:53.970+0000 7fe3fa7fc700 1 -- 172.21.15.178:0/622631368 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe3f40095f0 con 0x7fe4040f6480 2024-02-11T17:04:53.974 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:53.970+0000 7fe40c260700 1 -- 172.21.15.178:0/622631368 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe404045ad0 con 0x7fe4040f6480 2024-02-11T17:04:53.974 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:53.971+0000 7fe3fa7fc700 1 --2- 172.21.15.178:0/622631368 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fe3f002ffb0 0x7fe3f0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:04:53.974 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:53.971+0000 7fe3fa7fc700 1 -- 172.21.15.178:0/622631368 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe3f40410c0 con 0x7fe4040f6480 2024-02-11T17:04:53.974 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:53.971+0000 7fe4097fb700 1 --2- 172.21.15.178:0/622631368 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fe3f002ffb0 0x7fe3f0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:04:53.974 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:53.971+0000 7fe4097fb700 1 --2- 172.21.15.178:0/622631368 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fe3f002ffb0 0x7fe3f0032470 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fe4000096a0 tx=0x7fe400006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:04:53.977 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:53.975+0000 7fe3fa7fc700 1 -- 172.21.15.178:0/622631368 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe3f4014420 con 0x7fe4040f6480 2024-02-11T17:04:54.183 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:54.181+0000 7fe40c260700 1 -- 172.21.15.178:0/622631368 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe404108100 con 0x7fe4040f6480 2024-02-11T17:04:54.183 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:54.182+0000 7fe3fa7fc700 1 -- 172.21.15.178:0/622631368 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe3f4010230 con 0x7fe4040f6480 2024-02-11T17:04:54.184 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:04:54.184 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:04:54.186 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:54.183+0000 7fe40c260700 1 -- 172.21.15.178:0/622631368 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fe3f002ffb0 msgr2=0x7fe3f0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:04:54.186 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:54.183+0000 7fe40c260700 1 --2- 172.21.15.178:0/622631368 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fe3f002ffb0 0x7fe3f0032470 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fe4000096a0 tx=0x7fe400006b40).stop 2024-02-11T17:04:54.186 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:54.183+0000 7fe40c260700 1 -- 172.21.15.178:0/622631368 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4040f6480 msgr2=0x7fe4041066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:04:54.186 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:54.183+0000 7fe40c260700 1 --2- 172.21.15.178:0/622631368 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4040f6480 0x7fe4041066d0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7fe3f4013510 tx=0x7fe3f400b040).stop 2024-02-11T17:04:54.186 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:54.183+0000 7fe40c260700 1 -- 172.21.15.178:0/622631368 shutdown_connections 2024-02-11T17:04:54.186 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:54.183+0000 7fe40c260700 1 --2- 172.21.15.178:0/622631368 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fe3f002ffb0 0x7fe3f0032470 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:04:54.186 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:54.183+0000 7fe40c260700 1 --2- 172.21.15.178:0/622631368 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe4040f6480 0x7fe4041066d0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:04:54.186 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:54.183+0000 7fe40c260700 1 -- 172.21.15.178:0/622631368 >> 172.21.15.178:0/622631368 conn(0x7fe4040f1c00 msgr2=0x7fe4040f2720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:04:54.186 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:54.184+0000 7fe40c260700 1 -- 172.21.15.178:0/622631368 shutdown_connections 2024-02-11T17:04:54.186 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:54.184+0000 7fe40c260700 1 -- 172.21.15.178:0/622631368 wait complete. 2024-02-11T17:04:54.186 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:04:54.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:04:54 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:04:53.173390+0000 mgr.smithi037.eofnaq (mgr.14162) 146 : cluster [DBG] pgmap v139: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:04:54.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:04:54 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:04:54.183356+0000 mon.smithi037 (mon.0) 193 : audit [DBG] from='client.? 172.21.15.178:0/622631368' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:04:55.891 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:04:55.891 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:04:56.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:04:56 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:04:55.173754+0000 mgr.smithi037.eofnaq (mgr.14162) 147 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:04:57.722 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:57.719+0000 7fc6f64f4700 1 -- 172.21.15.178:0/1708243071 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc6f00f6510 msgr2=0x7fc6f00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:04:57.723 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:57.719+0000 7fc6f64f4700 1 --2- 172.21.15.178:0/1708243071 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc6f00f6510 0x7fc6f00f6930 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7fc6e4009fb0 tx=0x7fc6e4009b30).stop 2024-02-11T17:04:57.723 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:57.719+0000 7fc6f64f4700 1 -- 172.21.15.178:0/1708243071 shutdown_connections 2024-02-11T17:04:57.723 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:57.719+0000 7fc6f64f4700 1 --2- 172.21.15.178:0/1708243071 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc6f00f6510 0x7fc6f00f6930 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:04:57.723 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:57.719+0000 7fc6f64f4700 1 -- 172.21.15.178:0/1708243071 >> 172.21.15.178:0/1708243071 conn(0x7fc6f00f1c70 msgr2=0x7fc6f00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:04:57.723 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:57.719+0000 7fc6f64f4700 1 -- 172.21.15.178:0/1708243071 shutdown_connections 2024-02-11T17:04:57.724 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:57.719+0000 7fc6f64f4700 1 -- 172.21.15.178:0/1708243071 wait complete. 2024-02-11T17:04:57.724 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:57.720+0000 7fc6f64f4700 1 Processor -- start 2024-02-11T17:04:57.724 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:57.720+0000 7fc6f64f4700 1 -- start start 2024-02-11T17:04:57.724 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:57.720+0000 7fc6f64f4700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc6f00f6510 0x7fc6f0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:04:57.724 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:57.720+0000 7fc6f64f4700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f0106c20 con 0x7fc6f00f6510 2024-02-11T17:04:57.724 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:57.720+0000 7fc6effff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc6f00f6510 0x7fc6f0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:04:57.724 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:57.720+0000 7fc6effff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc6f00f6510 0x7fc6f0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:47756/0 (socket says 172.21.15.178:47756) 2024-02-11T17:04:57.724 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:57.720+0000 7fc6effff700 1 -- 172.21.15.178:0/3160910208 learned_addr learned my addr 172.21.15.178:0/3160910208 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:04:57.724 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:57.720+0000 7fc6effff700 1 -- 172.21.15.178:0/3160910208 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6e4005040 con 0x7fc6f00f6510 2024-02-11T17:04:57.724 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:57.721+0000 7fc6effff700 1 --2- 172.21.15.178:0/3160910208 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc6f00f6510 0x7fc6f0106730 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7fc6e4013510 tx=0x7fc6e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:04:57.724 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:57.721+0000 7fc6ecff9700 1 -- 172.21.15.178:0/3160910208 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc6e4009320 con 0x7fc6f00f6510 2024-02-11T17:04:57.724 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:57.721+0000 7fc6ecff9700 1 -- 172.21.15.178:0/3160910208 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc6e4013f00 con 0x7fc6f00f6510 2024-02-11T17:04:57.724 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:57.721+0000 7fc6f64f4700 1 -- 172.21.15.178:0/3160910208 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6f0106e20 con 0x7fc6f00f6510 2024-02-11T17:04:57.725 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:57.721+0000 7fc6ecff9700 1 -- 172.21.15.178:0/3160910208 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc6e4014d90 con 0x7fc6f00f6510 2024-02-11T17:04:57.725 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:57.721+0000 7fc6f64f4700 1 -- 172.21.15.178:0/3160910208 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6f01072b0 con 0x7fc6f00f6510 2024-02-11T17:04:57.725 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:57.722+0000 7fc6ecff9700 1 -- 172.21.15.178:0/3160910208 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc6e4014420 con 0x7fc6f00f6510 2024-02-11T17:04:57.725 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:57.722+0000 7fc6ecff9700 1 --2- 172.21.15.178:0/3160910208 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fc6d802ffb0 0x7fc6d8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:04:57.726 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:57.722+0000 7fc6f64f4700 1 -- 172.21.15.178:0/3160910208 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6f0045ad0 con 0x7fc6f00f6510 2024-02-11T17:04:57.726 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:57.722+0000 7fc6ecff9700 1 -- 172.21.15.178:0/3160910208 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc6e4040fd0 con 0x7fc6f00f6510 2024-02-11T17:04:57.726 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:57.722+0000 7fc6ef7fe700 1 --2- 172.21.15.178:0/3160910208 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fc6d802ffb0 0x7fc6d8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:04:57.726 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:57.722+0000 7fc6ef7fe700 1 --2- 172.21.15.178:0/3160910208 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fc6d802ffb0 0x7fc6d8032470 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fc6e00096a0 tx=0x7fc6e0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:04:57.728 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:57.727+0000 7fc6ecff9700 1 -- 172.21.15.178:0/3160910208 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc6e40095f0 con 0x7fc6f00f6510 2024-02-11T17:04:57.933 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:57.931+0000 7fc6f64f4700 1 -- 172.21.15.178:0/3160910208 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc6f00fc050 con 0x7fc6f00f6510 2024-02-11T17:04:57.934 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:57.932+0000 7fc6ecff9700 1 -- 172.21.15.178:0/3160910208 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc6e402b020 con 0x7fc6f00f6510 2024-02-11T17:04:57.934 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:04:57.934 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:04:57.935 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:57.934+0000 7fc6f64f4700 1 -- 172.21.15.178:0/3160910208 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fc6d802ffb0 msgr2=0x7fc6d8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:04:57.935 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:57.934+0000 7fc6f64f4700 1 --2- 172.21.15.178:0/3160910208 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fc6d802ffb0 0x7fc6d8032470 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fc6e00096a0 tx=0x7fc6e0006b40).stop 2024-02-11T17:04:57.935 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:57.934+0000 7fc6f64f4700 1 -- 172.21.15.178:0/3160910208 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc6f00f6510 msgr2=0x7fc6f0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:04:57.936 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:57.934+0000 7fc6f64f4700 1 --2- 172.21.15.178:0/3160910208 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc6f00f6510 0x7fc6f0106730 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7fc6e4013510 tx=0x7fc6e400b040).stop 2024-02-11T17:04:57.936 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:57.934+0000 7fc6f64f4700 1 -- 172.21.15.178:0/3160910208 shutdown_connections 2024-02-11T17:04:57.936 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:57.934+0000 7fc6f64f4700 1 --2- 172.21.15.178:0/3160910208 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fc6d802ffb0 0x7fc6d8032470 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:04:57.936 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:57.934+0000 7fc6f64f4700 1 --2- 172.21.15.178:0/3160910208 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc6f00f6510 0x7fc6f0106730 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:04:57.936 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:57.934+0000 7fc6f64f4700 1 -- 172.21.15.178:0/3160910208 >> 172.21.15.178:0/3160910208 conn(0x7fc6f00f1c70 msgr2=0x7fc6f00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:04:57.936 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:57.934+0000 7fc6f64f4700 1 -- 172.21.15.178:0/3160910208 shutdown_connections 2024-02-11T17:04:57.936 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:04:57.934+0000 7fc6f64f4700 1 -- 172.21.15.178:0/3160910208 wait complete. 2024-02-11T17:04:57.936 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:04:58.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:04:58 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:04:57.174082+0000 mgr.smithi037.eofnaq (mgr.14162) 148 : cluster [DBG] pgmap v141: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:04:58.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:04:58 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:04:57.933436+0000 mon.smithi037 (mon.0) 194 : audit [DBG] from='client.? 172.21.15.178:0/3160910208' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:04:59.585 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:04:59.585 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:05:00.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:00 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:04:59.174350+0000 mgr.smithi037.eofnaq (mgr.14162) 149 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:05:01.479 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:01.441+0000 7f11c5446700 1 -- 172.21.15.178:0/2529845456 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f11c00f6510 msgr2=0x7f11c00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:05:01.480 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:01.441+0000 7f11c5446700 1 --2- 172.21.15.178:0/2529845456 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f11c00f6510 0x7f11c00f6930 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f11b0009fb0 tx=0x7f11b0009b30).stop 2024-02-11T17:05:01.480 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:01.441+0000 7f11c5446700 1 -- 172.21.15.178:0/2529845456 shutdown_connections 2024-02-11T17:05:01.480 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:01.441+0000 7f11c5446700 1 --2- 172.21.15.178:0/2529845456 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f11c00f6510 0x7f11c00f6930 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:05:01.480 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:01.441+0000 7f11c5446700 1 -- 172.21.15.178:0/2529845456 >> 172.21.15.178:0/2529845456 conn(0x7f11c00f1c70 msgr2=0x7f11c00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:05:01.480 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:01.441+0000 7f11c5446700 1 -- 172.21.15.178:0/2529845456 shutdown_connections 2024-02-11T17:05:01.481 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:01.441+0000 7f11c5446700 1 -- 172.21.15.178:0/2529845456 wait complete. 2024-02-11T17:05:01.481 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:01.442+0000 7f11c5446700 1 Processor -- start 2024-02-11T17:05:01.481 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:01.442+0000 7f11c5446700 1 -- start start 2024-02-11T17:05:01.481 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:01.442+0000 7f11c5446700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f11c00f6510 0x7f11c0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:05:01.481 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:01.442+0000 7f11c5446700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11c0106c20 con 0x7f11c00f6510 2024-02-11T17:05:01.481 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:01.442+0000 7f11beffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f11c00f6510 0x7f11c0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:05:01.481 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:01.442+0000 7f11beffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f11c00f6510 0x7f11c0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:38664/0 (socket says 172.21.15.178:38664) 2024-02-11T17:05:01.481 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:01.442+0000 7f11beffd700 1 -- 172.21.15.178:0/1304199099 learned_addr learned my addr 172.21.15.178:0/1304199099 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:05:01.481 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:01.442+0000 7f11beffd700 1 -- 172.21.15.178:0/1304199099 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11b0005040 con 0x7f11c00f6510 2024-02-11T17:05:01.481 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:01.442+0000 7f11beffd700 1 --2- 172.21.15.178:0/1304199099 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f11c00f6510 0x7f11c0106730 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f11b0013510 tx=0x7f11b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:05:01.482 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:01.443+0000 7f11a77fe700 1 -- 172.21.15.178:0/1304199099 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f11b0009320 con 0x7f11c00f6510 2024-02-11T17:05:01.482 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:01.443+0000 7f11a77fe700 1 -- 172.21.15.178:0/1304199099 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f11b0013f00 con 0x7f11c00f6510 2024-02-11T17:05:01.482 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:01.443+0000 7f11c5446700 1 -- 172.21.15.178:0/1304199099 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11c0106e20 con 0x7f11c00f6510 2024-02-11T17:05:01.482 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:01.443+0000 7f11a77fe700 1 -- 172.21.15.178:0/1304199099 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f11b0014d90 con 0x7f11c00f6510 2024-02-11T17:05:01.482 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:01.443+0000 7f11c5446700 1 -- 172.21.15.178:0/1304199099 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11c01072b0 con 0x7f11c00f6510 2024-02-11T17:05:01.482 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:01.443+0000 7f11a77fe700 1 -- 172.21.15.178:0/1304199099 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f11b00263a0 con 0x7f11c00f6510 2024-02-11T17:05:01.482 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:01.444+0000 7f11a77fe700 1 --2- 172.21.15.178:0/1304199099 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f11a802ff60 0x7f11a8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:05:01.483 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:01.444+0000 7f11a77fe700 1 -- 172.21.15.178:0/1304199099 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f11b0041150 con 0x7f11c00f6510 2024-02-11T17:05:01.483 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:01.444+0000 7f11c5446700 1 -- 172.21.15.178:0/1304199099 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11c0045ad0 con 0x7f11c00f6510 2024-02-11T17:05:01.483 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:01.444+0000 7f11be7fc700 1 --2- 172.21.15.178:0/1304199099 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f11a802ff60 0x7f11a8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:05:01.483 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:01.444+0000 7f11be7fc700 1 --2- 172.21.15.178:0/1304199099 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f11a802ff60 0x7f11a8032420 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f11b40096a0 tx=0x7f11b4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:05:01.483 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:01.449+0000 7f11a77fe700 1 -- 172.21.15.178:0/1304199099 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f11b0012920 con 0x7f11c00f6510 2024-02-11T17:05:01.653 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:01.652+0000 7f11c5446700 1 -- 172.21.15.178:0/1304199099 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f11c0107fd0 con 0x7f11c00f6510 2024-02-11T17:05:01.653 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:01.653+0000 7f11a77fe700 1 -- 172.21.15.178:0/1304199099 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f11b002b020 con 0x7f11c00f6510 2024-02-11T17:05:01.654 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:05:01.654 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:05:01.655 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:01.654+0000 7f11c5446700 1 -- 172.21.15.178:0/1304199099 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f11a802ff60 msgr2=0x7f11a8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:05:01.655 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:01.654+0000 7f11c5446700 1 --2- 172.21.15.178:0/1304199099 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f11a802ff60 0x7f11a8032420 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f11b40096a0 tx=0x7f11b4006b40).stop 2024-02-11T17:05:01.655 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:01.654+0000 7f11c5446700 1 -- 172.21.15.178:0/1304199099 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f11c00f6510 msgr2=0x7f11c0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:05:01.655 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:01.654+0000 7f11c5446700 1 --2- 172.21.15.178:0/1304199099 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f11c00f6510 0x7f11c0106730 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f11b0013510 tx=0x7f11b000b040).stop 2024-02-11T17:05:01.655 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:01.654+0000 7f11c5446700 1 -- 172.21.15.178:0/1304199099 shutdown_connections 2024-02-11T17:05:01.655 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:01.654+0000 7f11c5446700 1 --2- 172.21.15.178:0/1304199099 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f11a802ff60 0x7f11a8032420 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:05:01.655 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:01.654+0000 7f11c5446700 1 --2- 172.21.15.178:0/1304199099 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f11c00f6510 0x7f11c0106730 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:05:01.656 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:01.654+0000 7f11c5446700 1 -- 172.21.15.178:0/1304199099 >> 172.21.15.178:0/1304199099 conn(0x7f11c00f1c70 msgr2=0x7f11c00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:05:01.656 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:01.654+0000 7f11c5446700 1 -- 172.21.15.178:0/1304199099 shutdown_connections 2024-02-11T17:05:01.656 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:01.654+0000 7f11c5446700 1 -- 172.21.15.178:0/1304199099 wait complete. 2024-02-11T17:05:01.656 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:05:02.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:02 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:05:01.174642+0000 mgr.smithi037.eofnaq (mgr.14162) 150 : cluster [DBG] pgmap v143: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:05:02.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:02 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:05:01.653023+0000 mon.smithi037 (mon.0) 195 : audit [DBG] from='client.? 172.21.15.178:0/1304199099' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:05:03.350 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:05:03.350 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:05:04.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:04 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:05:03.174971+0000 mgr.smithi037.eofnaq (mgr.14162) 151 : cluster [DBG] pgmap v144: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:05:04.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:04 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:05:03.274804+0000 mon.smithi037 (mon.0) 196 : audit [INF] from='mgr.14162 172.21.15.37:0/2745465012' entity='mgr.smithi037.eofnaq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi037.eofnaq/mirror_snapshot_schedule"}]: dispatch 2024-02-11T17:05:04.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:04 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:05:03.278338+0000 mon.smithi037 (mon.0) 197 : audit [INF] from='mgr.14162 172.21.15.37:0/2745465012' entity='mgr.smithi037.eofnaq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi037.eofnaq/trash_purge_schedule"}]: dispatch 2024-02-11T17:05:05.360 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:05.358+0000 7f5500043700 1 -- 172.21.15.178:0/2345336319 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f54f80f6510 msgr2=0x7f54f80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:05:05.361 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:05.358+0000 7f5500043700 1 --2- 172.21.15.178:0/2345336319 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f54f80f6510 0x7f54f80f6930 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f54ec009fb0 tx=0x7f54ec009b30).stop 2024-02-11T17:05:05.361 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:05.358+0000 7f5500043700 1 -- 172.21.15.178:0/2345336319 shutdown_connections 2024-02-11T17:05:05.361 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:05.358+0000 7f5500043700 1 --2- 172.21.15.178:0/2345336319 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f54f80f6510 0x7f54f80f6930 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:05:05.361 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:05.358+0000 7f5500043700 1 -- 172.21.15.178:0/2345336319 >> 172.21.15.178:0/2345336319 conn(0x7f54f80f1c70 msgr2=0x7f54f80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:05:05.361 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:05.358+0000 7f5500043700 1 -- 172.21.15.178:0/2345336319 shutdown_connections 2024-02-11T17:05:05.361 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:05.358+0000 7f5500043700 1 -- 172.21.15.178:0/2345336319 wait complete. 2024-02-11T17:05:05.361 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:05.358+0000 7f5500043700 1 Processor -- start 2024-02-11T17:05:05.361 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:05.358+0000 7f5500043700 1 -- start start 2024-02-11T17:05:05.361 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:05.358+0000 7f5500043700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f54f80f6510 0x7f54f8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:05:05.361 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:05.358+0000 7f5500043700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54f8106c20 con 0x7f54f80f6510 2024-02-11T17:05:05.361 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:05.359+0000 7f54fdddf700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f54f80f6510 0x7f54f8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:05:05.361 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:05.359+0000 7f54fdddf700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f54f80f6510 0x7f54f8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:38686/0 (socket says 172.21.15.178:38686) 2024-02-11T17:05:05.361 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:05.359+0000 7f54fdddf700 1 -- 172.21.15.178:0/896907308 learned_addr learned my addr 172.21.15.178:0/896907308 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:05:05.361 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:05.359+0000 7f54fdddf700 1 -- 172.21.15.178:0/896907308 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54ec005040 con 0x7f54f80f6510 2024-02-11T17:05:05.362 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:05.359+0000 7f54fdddf700 1 --2- 172.21.15.178:0/896907308 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f54f80f6510 0x7f54f8106730 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f54ec013510 tx=0x7f54ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:05:05.362 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:05.359+0000 7f54ea7fc700 1 -- 172.21.15.178:0/896907308 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f54ec009320 con 0x7f54f80f6510 2024-02-11T17:05:05.362 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:05.359+0000 7f5500043700 1 -- 172.21.15.178:0/896907308 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54f8106e20 con 0x7f54f80f6510 2024-02-11T17:05:05.362 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:05.359+0000 7f54ea7fc700 1 -- 172.21.15.178:0/896907308 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f54ec013f00 con 0x7f54f80f6510 2024-02-11T17:05:05.362 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:05.359+0000 7f54ea7fc700 1 -- 172.21.15.178:0/896907308 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f54ec014d90 con 0x7f54f80f6510 2024-02-11T17:05:05.362 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:05.359+0000 7f5500043700 1 -- 172.21.15.178:0/896907308 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54f81072b0 con 0x7f54f80f6510 2024-02-11T17:05:05.363 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:05.360+0000 7f54ea7fc700 1 -- 172.21.15.178:0/896907308 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f54ec0095f0 con 0x7f54f80f6510 2024-02-11T17:05:05.363 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:05.360+0000 7f5500043700 1 -- 172.21.15.178:0/896907308 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54f8045ad0 con 0x7f54f80f6510 2024-02-11T17:05:05.363 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:05.361+0000 7f54ea7fc700 1 --2- 172.21.15.178:0/896907308 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f54e402ff60 0x7f54e4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:05:05.363 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:05.361+0000 7f54ea7fc700 1 -- 172.21.15.178:0/896907308 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f54ec041030 con 0x7f54f80f6510 2024-02-11T17:05:05.363 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:05.361+0000 7f54fd5de700 1 --2- 172.21.15.178:0/896907308 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f54e402ff60 0x7f54e4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:05:05.363 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:05.361+0000 7f54fd5de700 1 --2- 172.21.15.178:0/896907308 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f54e402ff60 0x7f54e4032420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f54f40096a0 tx=0x7f54f4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:05:05.366 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:05.366+0000 7f54ea7fc700 1 -- 172.21.15.178:0/896907308 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f54ec014420 con 0x7f54f80f6510 2024-02-11T17:05:05.570 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:05.570+0000 7f5500043700 1 -- 172.21.15.178:0/896907308 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f54f80fbf60 con 0x7f54f80f6510 2024-02-11T17:05:05.571 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:05.570+0000 7f54ea7fc700 1 -- 172.21.15.178:0/896907308 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f54ec010230 con 0x7f54f80f6510 2024-02-11T17:05:05.571 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:05:05.571 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:05:05.573 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:05.572+0000 7f5500043700 1 -- 172.21.15.178:0/896907308 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f54e402ff60 msgr2=0x7f54e4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:05:05.573 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:05.572+0000 7f5500043700 1 --2- 172.21.15.178:0/896907308 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f54e402ff60 0x7f54e4032420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f54f40096a0 tx=0x7f54f4006b40).stop 2024-02-11T17:05:05.573 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:05.572+0000 7f5500043700 1 -- 172.21.15.178:0/896907308 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f54f80f6510 msgr2=0x7f54f8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:05:05.573 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:05.572+0000 7f5500043700 1 --2- 172.21.15.178:0/896907308 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f54f80f6510 0x7f54f8106730 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f54ec013510 tx=0x7f54ec00b040).stop 2024-02-11T17:05:05.573 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:05.572+0000 7f5500043700 1 -- 172.21.15.178:0/896907308 shutdown_connections 2024-02-11T17:05:05.573 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:05.572+0000 7f5500043700 1 --2- 172.21.15.178:0/896907308 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f54e402ff60 0x7f54e4032420 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:05:05.573 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:05.572+0000 7f5500043700 1 --2- 172.21.15.178:0/896907308 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f54f80f6510 0x7f54f8106730 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:05:05.573 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:05.572+0000 7f5500043700 1 -- 172.21.15.178:0/896907308 >> 172.21.15.178:0/896907308 conn(0x7f54f80f1c70 msgr2=0x7f54f80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:05:05.573 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:05.572+0000 7f5500043700 1 -- 172.21.15.178:0/896907308 shutdown_connections 2024-02-11T17:05:05.573 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:05.572+0000 7f5500043700 1 -- 172.21.15.178:0/896907308 wait complete. 2024-02-11T17:05:05.574 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:05:06.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:06 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:05:05.175254+0000 mgr.smithi037.eofnaq (mgr.14162) 152 : cluster [DBG] pgmap v145: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:05:06.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:06 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:05:05.570650+0000 mon.smithi037 (mon.0) 198 : audit [DBG] from='client.? 172.21.15.178:0/896907308' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:05:07.212 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:05:07.213 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:05:08.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:08 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:05:07.175591+0000 mgr.smithi037.eofnaq (mgr.14162) 153 : cluster [DBG] pgmap v146: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:05:09.114 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:09.108+0000 7f2174ca0700 1 -- 172.21.15.178:0/2652478421 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f21700f6510 msgr2=0x7f21700f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:05:09.115 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:09.108+0000 7f2174ca0700 1 --2- 172.21.15.178:0/2652478421 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f21700f6510 0x7f21700f6930 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f2160009fb0 tx=0x7f2160009b30).stop 2024-02-11T17:05:09.115 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:09.108+0000 7f2174ca0700 1 -- 172.21.15.178:0/2652478421 shutdown_connections 2024-02-11T17:05:09.115 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:09.108+0000 7f2174ca0700 1 --2- 172.21.15.178:0/2652478421 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f21700f6510 0x7f21700f6930 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:05:09.115 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:09.108+0000 7f2174ca0700 1 -- 172.21.15.178:0/2652478421 >> 172.21.15.178:0/2652478421 conn(0x7f21700f1c70 msgr2=0x7f21700f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:05:09.115 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:09.108+0000 7f2174ca0700 1 -- 172.21.15.178:0/2652478421 shutdown_connections 2024-02-11T17:05:09.115 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:09.108+0000 7f2174ca0700 1 -- 172.21.15.178:0/2652478421 wait complete. 2024-02-11T17:05:09.115 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:09.109+0000 7f2174ca0700 1 Processor -- start 2024-02-11T17:05:09.115 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:09.109+0000 7f2174ca0700 1 -- start start 2024-02-11T17:05:09.115 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:09.109+0000 7f2174ca0700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f21700f6510 0x7f2170106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:05:09.115 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:09.109+0000 7f2174ca0700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2170106c20 con 0x7f21700f6510 2024-02-11T17:05:09.115 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:09.109+0000 7f216e59c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f21700f6510 0x7f2170106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:05:09.115 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:09.109+0000 7f216e59c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f21700f6510 0x7f2170106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:38698/0 (socket says 172.21.15.178:38698) 2024-02-11T17:05:09.115 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:09.109+0000 7f216e59c700 1 -- 172.21.15.178:0/3491466393 learned_addr learned my addr 172.21.15.178:0/3491466393 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:05:09.116 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:09.109+0000 7f216e59c700 1 -- 172.21.15.178:0/3491466393 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2160005040 con 0x7f21700f6510 2024-02-11T17:05:09.116 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:09.110+0000 7f216e59c700 1 --2- 172.21.15.178:0/3491466393 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f21700f6510 0x7f2170106730 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f2160013510 tx=0x7f216000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:05:09.116 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:09.110+0000 7f215effd700 1 -- 172.21.15.178:0/3491466393 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2160009320 con 0x7f21700f6510 2024-02-11T17:05:09.116 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:09.110+0000 7f2174ca0700 1 -- 172.21.15.178:0/3491466393 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2170106e20 con 0x7f21700f6510 2024-02-11T17:05:09.116 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:09.110+0000 7f215effd700 1 -- 172.21.15.178:0/3491466393 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2160013f00 con 0x7f21700f6510 2024-02-11T17:05:09.116 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:09.110+0000 7f215effd700 1 -- 172.21.15.178:0/3491466393 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2160014d90 con 0x7f21700f6510 2024-02-11T17:05:09.116 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:09.110+0000 7f2174ca0700 1 -- 172.21.15.178:0/3491466393 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21701072b0 con 0x7f21700f6510 2024-02-11T17:05:09.116 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:09.111+0000 7f215effd700 1 -- 172.21.15.178:0/3491466393 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f21600095f0 con 0x7f21700f6510 2024-02-11T17:05:09.116 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:09.111+0000 7f2174ca0700 1 -- 172.21.15.178:0/3491466393 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2170045ad0 con 0x7f21700f6510 2024-02-11T17:05:09.116 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:09.111+0000 7f215effd700 1 --2- 172.21.15.178:0/3491466393 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f215802ff60 0x7f2158032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:05:09.116 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:09.111+0000 7f215effd700 1 -- 172.21.15.178:0/3491466393 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2160041000 con 0x7f21700f6510 2024-02-11T17:05:09.116 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:09.111+0000 7f216dd9b700 1 --2- 172.21.15.178:0/3491466393 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f215802ff60 0x7f2158032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:05:09.116 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:09.111+0000 7f216dd9b700 1 --2- 172.21.15.178:0/3491466393 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f215802ff60 0x7f2158032420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f21640096a0 tx=0x7f2164006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:05:09.117 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:09.116+0000 7f215effd700 1 -- 172.21.15.178:0/3491466393 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2160014420 con 0x7f21700f6510 2024-02-11T17:05:09.321 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:09.320+0000 7f2174ca0700 1 -- 172.21.15.178:0/3491466393 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f21700fbf60 con 0x7f21700f6510 2024-02-11T17:05:09.322 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:09.320+0000 7f215effd700 1 -- 172.21.15.178:0/3491466393 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2160010230 con 0x7f21700f6510 2024-02-11T17:05:09.322 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:05:09.322 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:05:09.324 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:09.322+0000 7f2174ca0700 1 -- 172.21.15.178:0/3491466393 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f215802ff60 msgr2=0x7f2158032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:05:09.324 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:09.322+0000 7f2174ca0700 1 --2- 172.21.15.178:0/3491466393 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f215802ff60 0x7f2158032420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f21640096a0 tx=0x7f2164006b40).stop 2024-02-11T17:05:09.324 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:09.322+0000 7f2174ca0700 1 -- 172.21.15.178:0/3491466393 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f21700f6510 msgr2=0x7f2170106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:05:09.324 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:09.322+0000 7f2174ca0700 1 --2- 172.21.15.178:0/3491466393 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f21700f6510 0x7f2170106730 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f2160013510 tx=0x7f216000b040).stop 2024-02-11T17:05:09.324 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:09.322+0000 7f2174ca0700 1 -- 172.21.15.178:0/3491466393 shutdown_connections 2024-02-11T17:05:09.324 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:09.322+0000 7f2174ca0700 1 --2- 172.21.15.178:0/3491466393 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f215802ff60 0x7f2158032420 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:05:09.324 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:09.322+0000 7f2174ca0700 1 --2- 172.21.15.178:0/3491466393 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f21700f6510 0x7f2170106730 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:05:09.324 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:09.322+0000 7f2174ca0700 1 -- 172.21.15.178:0/3491466393 >> 172.21.15.178:0/3491466393 conn(0x7f21700f1c70 msgr2=0x7f21700f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:05:09.324 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:09.322+0000 7f2174ca0700 1 -- 172.21.15.178:0/3491466393 shutdown_connections 2024-02-11T17:05:09.325 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:09.322+0000 7f2174ca0700 1 -- 172.21.15.178:0/3491466393 wait complete. 2024-02-11T17:05:09.325 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:05:10.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:10 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:05:09.175938+0000 mgr.smithi037.eofnaq (mgr.14162) 154 : cluster [DBG] pgmap v147: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:05:10.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:10 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:05:09.321712+0000 mon.smithi037 (mon.0) 199 : audit [DBG] from='client.? 172.21.15.178:0/3491466393' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:05:11.020 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:05:11.021 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:05:12.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:12 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:05:11.176196+0000 mgr.smithi037.eofnaq (mgr.14162) 155 : cluster [DBG] pgmap v148: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:05:12.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:12 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:05:12.057531+0000 mon.smithi037 (mon.0) 200 : audit [INF] from='mgr.14162 172.21.15.37:0/2745465012' entity='mgr.smithi037.eofnaq' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi037", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-11T17:05:12.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:12 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:05:12.059970+0000 mon.smithi037 (mon.0) 201 : audit [INF] from='mgr.14162 172.21.15.37:0/2745465012' entity='mgr.smithi037.eofnaq' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi037", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-02-11T17:05:12.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:12 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:05:12.060883+0000 mon.smithi037 (mon.0) 202 : audit [DBG] from='mgr.14162 172.21.15.37:0/2745465012' entity='mgr.smithi037.eofnaq' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-11T17:05:13.040 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:13.036+0000 7f773bfff700 1 -- 172.21.15.178:0/1797794086 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f773c0f62a0 msgr2=0x7f773c0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:05:13.040 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:13.036+0000 7f773bfff700 1 --2- 172.21.15.178:0/1797794086 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f773c0f62a0 0x7f773c0f66c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f772c009fb0 tx=0x7f772c009b30).stop 2024-02-11T17:05:13.040 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:13.036+0000 7f773bfff700 1 -- 172.21.15.178:0/1797794086 shutdown_connections 2024-02-11T17:05:13.040 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:13.036+0000 7f773bfff700 1 --2- 172.21.15.178:0/1797794086 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f773c0f62a0 0x7f773c0f66c0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:05:13.040 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:13.036+0000 7f773bfff700 1 -- 172.21.15.178:0/1797794086 >> 172.21.15.178:0/1797794086 conn(0x7f773c0f1aa0 msgr2=0x7f773c0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:05:13.040 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:13.036+0000 7f773bfff700 1 -- 172.21.15.178:0/1797794086 shutdown_connections 2024-02-11T17:05:13.040 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:13.036+0000 7f773bfff700 1 -- 172.21.15.178:0/1797794086 wait complete. 2024-02-11T17:05:13.040 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:13.037+0000 7f773bfff700 1 Processor -- start 2024-02-11T17:05:13.040 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:13.037+0000 7f773bfff700 1 -- start start 2024-02-11T17:05:13.041 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:13.037+0000 7f773bfff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f773c06b960 0x7f773c069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:05:13.041 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:13.037+0000 7f773bfff700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f773c06bd80 con 0x7f773c06b960 2024-02-11T17:05:13.041 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:13.037+0000 7f773affd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f773c06b960 0x7f773c069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:05:13.041 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:13.037+0000 7f773affd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f773c06b960 0x7f773c069e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:35858/0 (socket says 172.21.15.178:35858) 2024-02-11T17:05:13.041 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:13.037+0000 7f773affd700 1 -- 172.21.15.178:0/2798424642 learned_addr learned my addr 172.21.15.178:0/2798424642 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:05:13.041 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:13.037+0000 7f773affd700 1 -- 172.21.15.178:0/2798424642 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f772c005040 con 0x7f773c06b960 2024-02-11T17:05:13.041 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:13.038+0000 7f773affd700 1 --2- 172.21.15.178:0/2798424642 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f773c06b960 0x7f773c069e20 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f772c009f80 tx=0x7f772c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:05:13.041 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:13.038+0000 7f77337fe700 1 -- 172.21.15.178:0/2798424642 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f772c0136c0 con 0x7f773c06b960 2024-02-11T17:05:13.041 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:13.038+0000 7f773bfff700 1 -- 172.21.15.178:0/2798424642 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f773c06a310 con 0x7f773c06b960 2024-02-11T17:05:13.041 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:13.038+0000 7f77337fe700 1 -- 172.21.15.178:0/2798424642 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f772c0095f0 con 0x7f773c06b960 2024-02-11T17:05:13.041 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:13.038+0000 7f77337fe700 1 -- 172.21.15.178:0/2798424642 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f772c014420 con 0x7f773c06b960 2024-02-11T17:05:13.041 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:13.038+0000 7f773bfff700 1 -- 172.21.15.178:0/2798424642 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f773c06a800 con 0x7f773c06b960 2024-02-11T17:05:13.042 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:13.038+0000 7f77337fe700 1 -- 172.21.15.178:0/2798424642 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f772c025060 con 0x7f773c06b960 2024-02-11T17:05:13.042 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:13.039+0000 7f773bfff700 1 -- 172.21.15.178:0/2798424642 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f773c059100 con 0x7f773c06b960 2024-02-11T17:05:13.042 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:13.039+0000 7f77337fe700 1 --2- 172.21.15.178:0/2798424642 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f772802ff60 0x7f7728032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:05:13.042 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:13.039+0000 7f77337fe700 1 -- 172.21.15.178:0/2798424642 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f772c040e60 con 0x7f773c06b960 2024-02-11T17:05:13.042 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:13.039+0000 7f773a7fc700 1 --2- 172.21.15.178:0/2798424642 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f772802ff60 0x7f7728032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:05:13.043 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:13.039+0000 7f773a7fc700 1 --2- 172.21.15.178:0/2798424642 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f772802ff60 0x7f7728032420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f77240096a0 tx=0x7f7724006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:05:13.045 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:13.044+0000 7f77337fe700 1 -- 172.21.15.178:0/2798424642 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f772c01c950 con 0x7f773c06b960 2024-02-11T17:05:13.250 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:13.249+0000 7f773bfff700 1 -- 172.21.15.178:0/2798424642 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f773c06abf0 con 0x7f773c06b960 2024-02-11T17:05:13.251 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:13.249+0000 7f77337fe700 1 -- 172.21.15.178:0/2798424642 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f772c01cb60 con 0x7f773c06b960 2024-02-11T17:05:13.251 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:05:13.251 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:05:13.253 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:13.251+0000 7f773bfff700 1 -- 172.21.15.178:0/2798424642 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f772802ff60 msgr2=0x7f7728032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:05:13.253 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:13.251+0000 7f773bfff700 1 --2- 172.21.15.178:0/2798424642 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f772802ff60 0x7f7728032420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f77240096a0 tx=0x7f7724006b40).stop 2024-02-11T17:05:13.253 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:13.251+0000 7f773bfff700 1 -- 172.21.15.178:0/2798424642 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f773c06b960 msgr2=0x7f773c069e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:05:13.253 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:13.251+0000 7f773bfff700 1 --2- 172.21.15.178:0/2798424642 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f773c06b960 0x7f773c069e20 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f772c009f80 tx=0x7f772c009b30).stop 2024-02-11T17:05:13.253 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:13.251+0000 7f773bfff700 1 -- 172.21.15.178:0/2798424642 shutdown_connections 2024-02-11T17:05:13.253 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:13.251+0000 7f773bfff700 1 --2- 172.21.15.178:0/2798424642 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f772802ff60 0x7f7728032420 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:05:13.253 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:13.251+0000 7f773bfff700 1 --2- 172.21.15.178:0/2798424642 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f773c06b960 0x7f773c069e20 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:05:13.253 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:13.251+0000 7f773bfff700 1 -- 172.21.15.178:0/2798424642 >> 172.21.15.178:0/2798424642 conn(0x7f773c0f1aa0 msgr2=0x7f773c0f25b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:05:13.253 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:13.251+0000 7f773bfff700 1 -- 172.21.15.178:0/2798424642 shutdown_connections 2024-02-11T17:05:13.253 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:13.251+0000 7f773bfff700 1 -- 172.21.15.178:0/2798424642 wait complete. 2024-02-11T17:05:13.254 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:05:13.434 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cephadm 2024-02-11T17:05:12.056259+0000 mgr.smithi037.eofnaq (mgr.14162) 156 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi037 ... 2024-02-11T17:05:13.434 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi037 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2024-02-11T17:05:13.434 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-11T17:05:13.434 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: stat: stderr time="2024-02-11T17:00:50Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-11T17:05:13.434 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: stat: stderr time="2024-02-11T17:01:27Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-11T17:05:13.434 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: stat: stderr time="2024-02-11T17:02:05Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-11T17:05:13.434 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-11T17:05:13.434 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi037 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2024-02-11T17:05:13.434 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-11T17:05:13.435 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: stat: stderr time="2024-02-11T17:03:19Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-11T17:05:13.435 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: stat: stderr time="2024-02-11T17:03:57Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-11T17:05:13.435 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: stat: stderr time="2024-02-11T17:04:34Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-11T17:05:13.435 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-11T17:05:13.435 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: Traceback (most recent call last): 2024-02-11T17:05:13.435 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: File "/var/lib/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-11T17:05:13.435 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: main() 2024-02-11T17:05:13.435 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: File "/var/lib/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-11T17:05:13.435 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: r = ctx.func(ctx) 2024-02-11T17:05:13.435 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: File "/var/lib/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-11T17:05:13.435 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: return func(ctx) 2024-02-11T17:05:13.435 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: File "/var/lib/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-02-11T17:05:13.435 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-11T17:05:13.435 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: File "/var/lib/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2024-02-11T17:05:13.436 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2024-02-11T17:05:13.436 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: File "/var/lib/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-02-11T17:05:13.436 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: raise RuntimeError('uid/gid not found') 2024-02-11T17:05:13.436 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: RuntimeError: uid/gid not found 2024-02-11T17:05:13.436 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: Traceback (most recent call last): 2024-02-11T17:05:13.436 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-02-11T17:05:13.436 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: yield (conn, connr) 2024-02-11T17:05:13.436 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-02-11T17:05:13.436 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: code, '\n'.join(err))) 2024-02-11T17:05:13.436 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi037 ... 2024-02-11T17:05:13.436 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi037 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2024-02-11T17:05:13.436 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-11T17:05:13.436 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: stat: stderr time="2024-02-11T17:00:50Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-11T17:05:13.437 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: stat: stderr time="2024-02-11T17:01:27Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-11T17:05:13.437 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: stat: stderr time="2024-02-11T17:02:05Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-11T17:05:13.437 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-11T17:05:13.437 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi037 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2024-02-11T17:05:13.437 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-11T17:05:13.437 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: stat: stderr time="2024-02-11T17:03:19Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-11T17:05:13.437 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: stat: stderr time="2024-02-11T17:03:57Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-11T17:05:13.437 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: stat: stderr time="2024-02-11T17:04:34Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-11T17:05:13.437 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-11T17:05:13.437 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: Traceback (most recent call last): 2024-02-11T17:05:13.437 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: File "/var/lib/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-11T17:05:13.437 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: main() 2024-02-11T17:05:13.437 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: File "/var/lib/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-11T17:05:13.437 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: r = ctx.func(ctx) 2024-02-11T17:05:13.437 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: File "/var/lib/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-11T17:05:13.438 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: return func(ctx) 2024-02-11T17:05:13.438 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: File "/var/lib/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-02-11T17:05:13.438 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-11T17:05:13.438 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: File "/var/lib/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2024-02-11T17:05:13.438 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2024-02-11T17:05:13.438 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: File "/var/lib/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-02-11T17:05:13.438 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: raise RuntimeError('uid/gid not found') 2024-02-11T17:05:13.438 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: RuntimeError: uid/gid not found 2024-02-11T17:05:13.438 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cephadm 2024-02-11T17:05:12.061635+0000 mgr.smithi037.eofnaq (mgr.14162) 157 : cephadm [INF] Deploying daemon crash.smithi037 on smithi037 2024-02-11T17:05:14.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:14 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:05:13.176437+0000 mgr.smithi037.eofnaq (mgr.14162) 158 : cluster [DBG] pgmap v149: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:05:14.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:14 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:05:13.250690+0000 mon.smithi037 (mon.0) 203 : audit [DBG] from='client.? 172.21.15.178:0/2798424642' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:05:14.969 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:05:14.970 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:05:15.933 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:15 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:05:14.466005+0000 mon.smithi037 (mon.0) 204 : audit [INF] from='mgr.14162 172.21.15.37:0/2745465012' entity='mgr.smithi037.eofnaq' 2024-02-11T17:05:15.933 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:15 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:05:14.568059+0000 mon.smithi037 (mon.0) 205 : audit [INF] from='mgr.14162 172.21.15.37:0/2745465012' entity='mgr.smithi037.eofnaq' 2024-02-11T17:05:15.933 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:15 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:05:14.570246+0000 mon.smithi037 (mon.0) 206 : audit [INF] from='mgr.14162 172.21.15.37:0/2745465012' entity='mgr.smithi037.eofnaq' 2024-02-11T17:05:15.933 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:15 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:05:14.570635+0000 mon.smithi037 (mon.0) 207 : audit [INF] from='mgr.14162 172.21.15.37:0/2745465012' entity='mgr.smithi037.eofnaq' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-11T17:05:15.934 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:15 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:05:14.571075+0000 mgr.smithi037.eofnaq (mgr.14162) 159 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-11T17:05:15.934 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:15 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:05:14.576383+0000 mon.smithi037 (mon.0) 208 : audit [INF] from='mgr.14162 172.21.15.37:0/2745465012' entity='mgr.smithi037.eofnaq' 2024-02-11T17:05:15.934 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:15 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cephadm 2024-02-11T17:05:14.579190+0000 mgr.smithi037.eofnaq (mgr.14162) 160 : cephadm [INF] Deploying daemon grafana.smithi037 on smithi037 2024-02-11T17:05:16.780 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:16.777+0000 7f11e3fff700 1 -- 172.21.15.178:0/1527066210 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f11e40f51a0 msgr2=0x7f11e40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:05:16.780 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:16.777+0000 7f11e3fff700 1 --2- 172.21.15.178:0/1527066210 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f11e40f51a0 0x7f11e40f55c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f11d8009fb0 tx=0x7f11d8009b30).stop 2024-02-11T17:05:16.780 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:16.777+0000 7f11e3fff700 1 -- 172.21.15.178:0/1527066210 shutdown_connections 2024-02-11T17:05:16.780 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:16.777+0000 7f11e3fff700 1 --2- 172.21.15.178:0/1527066210 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f11e40f51a0 0x7f11e40f55c0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:05:16.780 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:16.777+0000 7f11e3fff700 1 -- 172.21.15.178:0/1527066210 >> 172.21.15.178:0/1527066210 conn(0x7f11e40f09a0 msgr2=0x7f11e40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:05:16.780 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:16.777+0000 7f11e3fff700 1 -- 172.21.15.178:0/1527066210 shutdown_connections 2024-02-11T17:05:16.780 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:16.777+0000 7f11e3fff700 1 -- 172.21.15.178:0/1527066210 wait complete. 2024-02-11T17:05:16.781 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:16.778+0000 7f11e3fff700 1 Processor -- start 2024-02-11T17:05:16.781 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:16.778+0000 7f11e3fff700 1 -- start start 2024-02-11T17:05:16.781 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:16.778+0000 7f11e3fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f11e40f51a0 0x7f11e41091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:05:16.781 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:16.778+0000 7f11e3fff700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11e4060220 con 0x7f11e40f51a0 2024-02-11T17:05:16.781 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:16.778+0000 7f11e2ffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f11e40f51a0 0x7f11e41091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:05:16.781 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:16.778+0000 7f11e2ffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f11e40f51a0 0x7f11e41091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:35882/0 (socket says 172.21.15.178:35882) 2024-02-11T17:05:16.781 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:16.778+0000 7f11e2ffd700 1 -- 172.21.15.178:0/710733379 learned_addr learned my addr 172.21.15.178:0/710733379 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:05:16.782 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:16.779+0000 7f11e2ffd700 1 -- 172.21.15.178:0/710733379 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11d8005040 con 0x7f11e40f51a0 2024-02-11T17:05:16.783 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:16.779+0000 7f11e2ffd700 1 --2- 172.21.15.178:0/710733379 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f11e40f51a0 0x7f11e41091d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f11d8013510 tx=0x7f11d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:05:16.783 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:16.779+0000 7f11cb7fe700 1 -- 172.21.15.178:0/710733379 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f11d8009320 con 0x7f11e40f51a0 2024-02-11T17:05:16.783 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:16.779+0000 7f11e3fff700 1 -- 172.21.15.178:0/710733379 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11e4060420 con 0x7f11e40f51a0 2024-02-11T17:05:16.783 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:16.779+0000 7f11cb7fe700 1 -- 172.21.15.178:0/710733379 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f11d8013f00 con 0x7f11e40f51a0 2024-02-11T17:05:16.783 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:16.779+0000 7f11cb7fe700 1 -- 172.21.15.178:0/710733379 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f11d8014dc0 con 0x7f11e40f51a0 2024-02-11T17:05:16.783 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:16.779+0000 7f11e3fff700 1 -- 172.21.15.178:0/710733379 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11e4109880 con 0x7f11e40f51a0 2024-02-11T17:05:16.783 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:16.780+0000 7f11cb7fe700 1 -- 172.21.15.178:0/710733379 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f11d801d3d0 con 0x7f11e40f51a0 2024-02-11T17:05:16.783 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:16.780+0000 7f11e3fff700 1 -- 172.21.15.178:0/710733379 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11e4059100 con 0x7f11e40f51a0 2024-02-11T17:05:16.783 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:16.780+0000 7f11cb7fe700 1 --2- 172.21.15.178:0/710733379 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f11cc02ff60 0x7f11cc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:05:16.783 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:16.780+0000 7f11cb7fe700 1 -- 172.21.15.178:0/710733379 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f11d8041060 con 0x7f11e40f51a0 2024-02-11T17:05:16.783 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:16.780+0000 7f11e27fc700 1 --2- 172.21.15.178:0/710733379 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f11cc02ff60 0x7f11cc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:05:16.783 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:16.781+0000 7f11e27fc700 1 --2- 172.21.15.178:0/710733379 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f11cc02ff60 0x7f11cc032420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f11d40096a0 tx=0x7f11d4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:05:16.786 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:16.785+0000 7f11cb7fe700 1 -- 172.21.15.178:0/710733379 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f11d80095f0 con 0x7f11e40f51a0 2024-02-11T17:05:16.933 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:16 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:05:15.176830+0000 mgr.smithi037.eofnaq (mgr.14162) 161 : cluster [DBG] pgmap v150: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:05:16.991 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:16.989+0000 7f11e3fff700 1 -- 172.21.15.178:0/710733379 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f11e4045ad0 con 0x7f11e40f51a0 2024-02-11T17:05:16.992 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:16.990+0000 7f11cb7fe700 1 -- 172.21.15.178:0/710733379 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f11d801cb10 con 0x7f11e40f51a0 2024-02-11T17:05:16.992 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:05:16.992 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:05:16.994 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:16.992+0000 7f11e3fff700 1 -- 172.21.15.178:0/710733379 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f11cc02ff60 msgr2=0x7f11cc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:05:16.994 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:16.992+0000 7f11e3fff700 1 --2- 172.21.15.178:0/710733379 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f11cc02ff60 0x7f11cc032420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f11d40096a0 tx=0x7f11d4006b40).stop 2024-02-11T17:05:16.994 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:16.992+0000 7f11e3fff700 1 -- 172.21.15.178:0/710733379 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f11e40f51a0 msgr2=0x7f11e41091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:05:16.994 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:16.992+0000 7f11e3fff700 1 --2- 172.21.15.178:0/710733379 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f11e40f51a0 0x7f11e41091d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f11d8013510 tx=0x7f11d800b040).stop 2024-02-11T17:05:16.995 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:16.992+0000 7f11e3fff700 1 -- 172.21.15.178:0/710733379 shutdown_connections 2024-02-11T17:05:16.995 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:16.992+0000 7f11e3fff700 1 --2- 172.21.15.178:0/710733379 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f11cc02ff60 0x7f11cc032420 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:05:16.995 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:16.992+0000 7f11e3fff700 1 --2- 172.21.15.178:0/710733379 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f11e40f51a0 0x7f11e41091d0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:05:16.995 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:16.992+0000 7f11e3fff700 1 -- 172.21.15.178:0/710733379 >> 172.21.15.178:0/710733379 conn(0x7f11e40f09a0 msgr2=0x7f11e40f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:05:16.995 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:16.992+0000 7f11e3fff700 1 -- 172.21.15.178:0/710733379 shutdown_connections 2024-02-11T17:05:16.995 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:16.992+0000 7f11e3fff700 1 -- 172.21.15.178:0/710733379 wait complete. 2024-02-11T17:05:16.995 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:05:17.933 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:17 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:05:16.991387+0000 mon.smithi037 (mon.0) 209 : audit [DBG] from='client.? 172.21.15.178:0/710733379' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:05:18.694 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:05:18.695 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:05:18.933 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:18 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:05:17.177142+0000 mgr.smithi037.eofnaq (mgr.14162) 162 : cluster [DBG] pgmap v151: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:05:20.474 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:20.471+0000 7f11f8aae700 1 -- 172.21.15.178:0/2708614070 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f11f40f5410 msgr2=0x7f11f40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:05:20.475 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:20.471+0000 7f11f8aae700 1 --2- 172.21.15.178:0/2708614070 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f11f40f5410 0x7f11f40f5830 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f11e8009fb0 tx=0x7f11e8009b30).stop 2024-02-11T17:05:20.475 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:20.471+0000 7f11f8aae700 1 -- 172.21.15.178:0/2708614070 shutdown_connections 2024-02-11T17:05:20.475 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:20.471+0000 7f11f8aae700 1 --2- 172.21.15.178:0/2708614070 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f11f40f5410 0x7f11f40f5830 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:05:20.475 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:20.471+0000 7f11f8aae700 1 -- 172.21.15.178:0/2708614070 >> 172.21.15.178:0/2708614070 conn(0x7f11f40f0b70 msgr2=0x7f11f40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:05:20.475 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:20.471+0000 7f11f8aae700 1 -- 172.21.15.178:0/2708614070 shutdown_connections 2024-02-11T17:05:20.475 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:20.471+0000 7f11f8aae700 1 -- 172.21.15.178:0/2708614070 wait complete. 2024-02-11T17:05:20.475 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:20.472+0000 7f11f8aae700 1 Processor -- start 2024-02-11T17:05:20.475 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:20.472+0000 7f11f8aae700 1 -- start start 2024-02-11T17:05:20.475 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:20.472+0000 7f11f8aae700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f11f40f5410 0x7f11f410b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:05:20.475 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:20.472+0000 7f11f8aae700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11f41071e0 con 0x7f11f40f5410 2024-02-11T17:05:20.475 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:20.472+0000 7f11f259c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f11f40f5410 0x7f11f410b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:05:20.475 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:20.472+0000 7f11f259c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f11f40f5410 0x7f11f410b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:40836/0 (socket says 172.21.15.178:40836) 2024-02-11T17:05:20.475 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:20.472+0000 7f11f259c700 1 -- 172.21.15.178:0/3791073448 learned_addr learned my addr 172.21.15.178:0/3791073448 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:05:20.475 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:20.472+0000 7f11f259c700 1 -- 172.21.15.178:0/3791073448 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11e8005040 con 0x7f11f40f5410 2024-02-11T17:05:20.476 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:20.473+0000 7f11f259c700 1 --2- 172.21.15.178:0/3791073448 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f11f40f5410 0x7f11f410b7f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f11e8013510 tx=0x7f11e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:05:20.476 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:20.473+0000 7f11e2ffd700 1 -- 172.21.15.178:0/3791073448 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f11e8009320 con 0x7f11f40f5410 2024-02-11T17:05:20.476 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:20.473+0000 7f11f8aae700 1 -- 172.21.15.178:0/3791073448 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11f4060280 con 0x7f11f40f5410 2024-02-11T17:05:20.476 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:20.473+0000 7f11e2ffd700 1 -- 172.21.15.178:0/3791073448 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f11e8013f00 con 0x7f11f40f5410 2024-02-11T17:05:20.476 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:20.473+0000 7f11e2ffd700 1 -- 172.21.15.178:0/3791073448 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f11e8014d90 con 0x7f11f40f5410 2024-02-11T17:05:20.476 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:20.473+0000 7f11f8aae700 1 -- 172.21.15.178:0/3791073448 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11f410bce0 con 0x7f11f40f5410 2024-02-11T17:05:20.477 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:20.474+0000 7f11e2ffd700 1 -- 172.21.15.178:0/3791073448 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f11e80095f0 con 0x7f11f40f5410 2024-02-11T17:05:20.477 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:20.474+0000 7f11f8aae700 1 -- 172.21.15.178:0/3791073448 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11f4045ad0 con 0x7f11f40f5410 2024-02-11T17:05:20.477 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:20.474+0000 7f11e2ffd700 1 --2- 172.21.15.178:0/3791073448 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f11dc02ff60 0x7f11dc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:05:20.477 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:20.474+0000 7f11e2ffd700 1 -- 172.21.15.178:0/3791073448 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f11e8041000 con 0x7f11f40f5410 2024-02-11T17:05:20.477 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:20.474+0000 7f11f1d9b700 1 --2- 172.21.15.178:0/3791073448 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f11dc02ff60 0x7f11dc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:05:20.477 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:20.474+0000 7f11f1d9b700 1 --2- 172.21.15.178:0/3791073448 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f11dc02ff60 0x7f11dc032420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f11e40096a0 tx=0x7f11e4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:05:20.480 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:20.479+0000 7f11e2ffd700 1 -- 172.21.15.178:0/3791073448 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f11e8014420 con 0x7f11f40f5410 2024-02-11T17:05:20.686 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:20.684+0000 7f11f8aae700 1 -- 172.21.15.178:0/3791073448 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f11f40fae60 con 0x7f11f40f5410 2024-02-11T17:05:20.687 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:20.685+0000 7f11e2ffd700 1 -- 172.21.15.178:0/3791073448 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f11e8010230 con 0x7f11f40f5410 2024-02-11T17:05:20.687 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:05:20.687 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:05:20.688 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:20.687+0000 7f11f8aae700 1 -- 172.21.15.178:0/3791073448 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f11dc02ff60 msgr2=0x7f11dc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:05:20.688 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:20.687+0000 7f11f8aae700 1 --2- 172.21.15.178:0/3791073448 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f11dc02ff60 0x7f11dc032420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f11e40096a0 tx=0x7f11e4006b40).stop 2024-02-11T17:05:20.688 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:20.687+0000 7f11f8aae700 1 -- 172.21.15.178:0/3791073448 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f11f40f5410 msgr2=0x7f11f410b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:05:20.688 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:20.687+0000 7f11f8aae700 1 --2- 172.21.15.178:0/3791073448 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f11f40f5410 0x7f11f410b7f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f11e8013510 tx=0x7f11e800b040).stop 2024-02-11T17:05:20.689 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:20.687+0000 7f11f8aae700 1 -- 172.21.15.178:0/3791073448 shutdown_connections 2024-02-11T17:05:20.689 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:20.687+0000 7f11f8aae700 1 --2- 172.21.15.178:0/3791073448 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f11dc02ff60 0x7f11dc032420 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:05:20.689 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:20.687+0000 7f11f8aae700 1 --2- 172.21.15.178:0/3791073448 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f11f40f5410 0x7f11f410b7f0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:05:20.689 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:20.687+0000 7f11f8aae700 1 -- 172.21.15.178:0/3791073448 >> 172.21.15.178:0/3791073448 conn(0x7f11f40f0b70 msgr2=0x7f11f40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:05:20.689 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:20.687+0000 7f11f8aae700 1 -- 172.21.15.178:0/3791073448 shutdown_connections 2024-02-11T17:05:20.689 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:20.687+0000 7f11f8aae700 1 -- 172.21.15.178:0/3791073448 wait complete. 2024-02-11T17:05:20.689 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:05:20.933 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:20 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:05:19.177483+0000 mgr.smithi037.eofnaq (mgr.14162) 163 : cluster [DBG] pgmap v152: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:05:21.933 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:21 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:05:20.686433+0000 mon.smithi037 (mon.0) 210 : audit [DBG] from='client.? 172.21.15.178:0/3791073448' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:05:22.311 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:05:22.311 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:05:22.933 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:22 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:05:21.177748+0000 mgr.smithi037.eofnaq (mgr.14162) 164 : cluster [DBG] pgmap v153: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:05:24.172 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:24.169+0000 7f814e77c700 1 -- 172.21.15.178:0/556374888 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f81480f5410 msgr2=0x7f81480f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:05:24.173 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:24.169+0000 7f814e77c700 1 --2- 172.21.15.178:0/556374888 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f81480f5410 0x7f81480f5830 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f813c009fb0 tx=0x7f813c009b30).stop 2024-02-11T17:05:24.173 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:24.169+0000 7f814e77c700 1 -- 172.21.15.178:0/556374888 shutdown_connections 2024-02-11T17:05:24.173 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:24.169+0000 7f814e77c700 1 --2- 172.21.15.178:0/556374888 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f81480f5410 0x7f81480f5830 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:05:24.173 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:24.169+0000 7f814e77c700 1 -- 172.21.15.178:0/556374888 >> 172.21.15.178:0/556374888 conn(0x7f81480f0b70 msgr2=0x7f81480f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:05:24.173 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:24.169+0000 7f814e77c700 1 -- 172.21.15.178:0/556374888 shutdown_connections 2024-02-11T17:05:24.173 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:24.169+0000 7f814e77c700 1 -- 172.21.15.178:0/556374888 wait complete. 2024-02-11T17:05:24.173 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:24.169+0000 7f814e77c700 1 Processor -- start 2024-02-11T17:05:24.173 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:24.169+0000 7f814e77c700 1 -- start start 2024-02-11T17:05:24.173 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:24.170+0000 7f814e77c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f81480f5410 0x7f81481093d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:05:24.173 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:24.170+0000 7f814e77c700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8148060240 con 0x7f81480f5410 2024-02-11T17:05:24.173 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:24.170+0000 7f8147fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f81480f5410 0x7f81481093d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:05:24.173 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:24.170+0000 7f8147fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f81480f5410 0x7f81481093d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:40854/0 (socket says 172.21.15.178:40854) 2024-02-11T17:05:24.173 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:24.170+0000 7f8147fff700 1 -- 172.21.15.178:0/1568605640 learned_addr learned my addr 172.21.15.178:0/1568605640 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:05:24.174 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:24.170+0000 7f8147fff700 1 -- 172.21.15.178:0/1568605640 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f813c005040 con 0x7f81480f5410 2024-02-11T17:05:24.174 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:24.170+0000 7f8147fff700 1 --2- 172.21.15.178:0/1568605640 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f81480f5410 0x7f81481093d0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f813c013510 tx=0x7f813c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:05:24.174 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:24.170+0000 7f8144ff9700 1 -- 172.21.15.178:0/1568605640 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f813c009320 con 0x7f81480f5410 2024-02-11T17:05:24.174 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:24.170+0000 7f8144ff9700 1 -- 172.21.15.178:0/1568605640 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f813c013f00 con 0x7f81480f5410 2024-02-11T17:05:24.174 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:24.171+0000 7f814e77c700 1 -- 172.21.15.178:0/1568605640 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8148060440 con 0x7f81480f5410 2024-02-11T17:05:24.174 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:24.171+0000 7f8144ff9700 1 -- 172.21.15.178:0/1568605640 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f813c014d90 con 0x7f81480f5410 2024-02-11T17:05:24.174 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:24.171+0000 7f814e77c700 1 -- 172.21.15.178:0/1568605640 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8148109a80 con 0x7f81480f5410 2024-02-11T17:05:24.175 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:24.171+0000 7f8144ff9700 1 -- 172.21.15.178:0/1568605640 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f813c0095f0 con 0x7f81480f5410 2024-02-11T17:05:24.175 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:24.171+0000 7f814e77c700 1 -- 172.21.15.178:0/1568605640 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8148045ad0 con 0x7f81480f5410 2024-02-11T17:05:24.175 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:24.172+0000 7f8144ff9700 1 --2- 172.21.15.178:0/1568605640 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f813002ff60 0x7f8130032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:05:24.175 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:24.172+0000 7f8144ff9700 1 -- 172.21.15.178:0/1568605640 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f813c0410c0 con 0x7f81480f5410 2024-02-11T17:05:24.175 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:24.172+0000 7f81477fe700 1 --2- 172.21.15.178:0/1568605640 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f813002ff60 0x7f8130032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:05:24.175 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:24.172+0000 7f81477fe700 1 --2- 172.21.15.178:0/1568605640 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f813002ff60 0x7f8130032420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f81380096a0 tx=0x7f8138006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:05:24.177 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:24.176+0000 7f8144ff9700 1 -- 172.21.15.178:0/1568605640 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f813c01d7f0 con 0x7f81480f5410 2024-02-11T17:05:24.380 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:24.379+0000 7f814e77c700 1 -- 172.21.15.178:0/1568605640 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f814810a2a0 con 0x7f81480f5410 2024-02-11T17:05:24.381 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:24.379+0000 7f8144ff9700 1 -- 172.21.15.178:0/1568605640 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f813c010230 con 0x7f81480f5410 2024-02-11T17:05:24.381 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:05:24.382 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:05:24.383 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:24.381+0000 7f814e77c700 1 -- 172.21.15.178:0/1568605640 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f813002ff60 msgr2=0x7f8130032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:05:24.383 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:24.381+0000 7f814e77c700 1 --2- 172.21.15.178:0/1568605640 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f813002ff60 0x7f8130032420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f81380096a0 tx=0x7f8138006b40).stop 2024-02-11T17:05:24.383 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:24.381+0000 7f814e77c700 1 -- 172.21.15.178:0/1568605640 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f81480f5410 msgr2=0x7f81481093d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:05:24.383 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:24.381+0000 7f814e77c700 1 --2- 172.21.15.178:0/1568605640 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f81480f5410 0x7f81481093d0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f813c013510 tx=0x7f813c00b040).stop 2024-02-11T17:05:24.383 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:24.381+0000 7f814e77c700 1 -- 172.21.15.178:0/1568605640 shutdown_connections 2024-02-11T17:05:24.383 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:24.381+0000 7f814e77c700 1 --2- 172.21.15.178:0/1568605640 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f813002ff60 0x7f8130032420 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:05:24.383 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:24.381+0000 7f814e77c700 1 --2- 172.21.15.178:0/1568605640 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f81480f5410 0x7f81481093d0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:05:24.383 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:24.381+0000 7f814e77c700 1 -- 172.21.15.178:0/1568605640 >> 172.21.15.178:0/1568605640 conn(0x7f81480f0b70 msgr2=0x7f81480f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:05:24.383 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:24.381+0000 7f814e77c700 1 -- 172.21.15.178:0/1568605640 shutdown_connections 2024-02-11T17:05:24.383 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:24.381+0000 7f814e77c700 1 -- 172.21.15.178:0/1568605640 wait complete. 2024-02-11T17:05:24.384 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:05:24.933 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:24 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:05:23.177948+0000 mgr.smithi037.eofnaq (mgr.14162) 165 : cluster [DBG] pgmap v154: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:05:24.933 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:24 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:05:24.380828+0000 mon.smithi037 (mon.0) 211 : audit [DBG] from='client.? 172.21.15.178:0/1568605640' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:05:26.075 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:05:26.076 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:05:26.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:26 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:05:25.178154+0000 mgr.smithi037.eofnaq (mgr.14162) 166 : cluster [DBG] pgmap v155: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:05:27.964 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:27.961+0000 7f52bfb0d700 1 -- 172.21.15.178:0/943208697 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f52b80f6510 msgr2=0x7f52b80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:05:27.965 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:27.961+0000 7f52bfb0d700 1 --2- 172.21.15.178:0/943208697 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f52b80f6510 0x7f52b80f6930 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f52a8009fb0 tx=0x7f52a8009b30).stop 2024-02-11T17:05:27.965 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:27.961+0000 7f52bfb0d700 1 -- 172.21.15.178:0/943208697 shutdown_connections 2024-02-11T17:05:27.965 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:27.961+0000 7f52bfb0d700 1 --2- 172.21.15.178:0/943208697 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f52b80f6510 0x7f52b80f6930 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:05:27.965 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:27.961+0000 7f52bfb0d700 1 -- 172.21.15.178:0/943208697 >> 172.21.15.178:0/943208697 conn(0x7f52b80f1c70 msgr2=0x7f52b80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:05:27.965 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:27.961+0000 7f52bfb0d700 1 -- 172.21.15.178:0/943208697 shutdown_connections 2024-02-11T17:05:27.965 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:27.961+0000 7f52bfb0d700 1 -- 172.21.15.178:0/943208697 wait complete. 2024-02-11T17:05:27.965 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:27.962+0000 7f52bfb0d700 1 Processor -- start 2024-02-11T17:05:27.965 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:27.962+0000 7f52bfb0d700 1 -- start start 2024-02-11T17:05:27.965 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:27.962+0000 7f52bfb0d700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f52b80f6510 0x7f52b8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:05:27.965 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:27.962+0000 7f52bfb0d700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52b8106c20 con 0x7f52b80f6510 2024-02-11T17:05:27.965 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:27.962+0000 7f52bd8a9700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f52b80f6510 0x7f52b8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:05:27.965 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:27.962+0000 7f52bd8a9700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f52b80f6510 0x7f52b8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:40872/0 (socket says 172.21.15.178:40872) 2024-02-11T17:05:27.966 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:27.962+0000 7f52bd8a9700 1 -- 172.21.15.178:0/741054647 learned_addr learned my addr 172.21.15.178:0/741054647 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:05:27.966 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:27.962+0000 7f52bd8a9700 1 -- 172.21.15.178:0/741054647 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52a8005040 con 0x7f52b80f6510 2024-02-11T17:05:27.966 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:27.963+0000 7f52bd8a9700 1 --2- 172.21.15.178:0/741054647 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f52b80f6510 0x7f52b8106730 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f52a8013510 tx=0x7f52a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:05:27.966 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:27.963+0000 7f52ae7fc700 1 -- 172.21.15.178:0/741054647 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f52a8009320 con 0x7f52b80f6510 2024-02-11T17:05:27.966 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:27.963+0000 7f52ae7fc700 1 -- 172.21.15.178:0/741054647 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f52a8013f00 con 0x7f52b80f6510 2024-02-11T17:05:27.966 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:27.963+0000 7f52bfb0d700 1 -- 172.21.15.178:0/741054647 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52b8106e20 con 0x7f52b80f6510 2024-02-11T17:05:27.966 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:27.963+0000 7f52ae7fc700 1 -- 172.21.15.178:0/741054647 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f52a8014d90 con 0x7f52b80f6510 2024-02-11T17:05:27.966 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:27.963+0000 7f52bfb0d700 1 -- 172.21.15.178:0/741054647 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52b81072b0 con 0x7f52b80f6510 2024-02-11T17:05:27.967 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:27.963+0000 7f52ae7fc700 1 -- 172.21.15.178:0/741054647 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f52a8014420 con 0x7f52b80f6510 2024-02-11T17:05:27.967 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:27.963+0000 7f52bfb0d700 1 -- 172.21.15.178:0/741054647 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52b8045ad0 con 0x7f52b80f6510 2024-02-11T17:05:27.967 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:27.964+0000 7f52ae7fc700 1 --2- 172.21.15.178:0/741054647 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f52a402ff60 0x7f52a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:05:27.967 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:27.964+0000 7f52ae7fc700 1 -- 172.21.15.178:0/741054647 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f52a8041000 con 0x7f52b80f6510 2024-02-11T17:05:27.967 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:27.964+0000 7f52bd0a8700 1 --2- 172.21.15.178:0/741054647 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f52a402ff60 0x7f52a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:05:27.967 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:27.964+0000 7f52bd0a8700 1 --2- 172.21.15.178:0/741054647 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f52a402ff60 0x7f52a4032420 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f52b40096a0 tx=0x7f52b4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:05:27.970 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:27.968+0000 7f52ae7fc700 1 -- 172.21.15.178:0/741054647 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f52a80095f0 con 0x7f52b80f6510 2024-02-11T17:05:28.175 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:28.173+0000 7f52bfb0d700 1 -- 172.21.15.178:0/741054647 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f52b80fbf60 con 0x7f52b80f6510 2024-02-11T17:05:28.175 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:28.174+0000 7f52ae7fc700 1 -- 172.21.15.178:0/741054647 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f52a801db70 con 0x7f52b80f6510 2024-02-11T17:05:28.176 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:05:28.176 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:05:28.177 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:28.175+0000 7f52bfb0d700 1 -- 172.21.15.178:0/741054647 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f52a402ff60 msgr2=0x7f52a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:05:28.177 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:28.175+0000 7f52bfb0d700 1 --2- 172.21.15.178:0/741054647 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f52a402ff60 0x7f52a4032420 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f52b40096a0 tx=0x7f52b4006b40).stop 2024-02-11T17:05:28.177 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:28.175+0000 7f52bfb0d700 1 -- 172.21.15.178:0/741054647 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f52b80f6510 msgr2=0x7f52b8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:05:28.177 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:28.175+0000 7f52bfb0d700 1 --2- 172.21.15.178:0/741054647 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f52b80f6510 0x7f52b8106730 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f52a8013510 tx=0x7f52a800b040).stop 2024-02-11T17:05:28.177 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:28.175+0000 7f52bfb0d700 1 -- 172.21.15.178:0/741054647 shutdown_connections 2024-02-11T17:05:28.177 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:28.175+0000 7f52bfb0d700 1 --2- 172.21.15.178:0/741054647 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f52a402ff60 0x7f52a4032420 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:05:28.177 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:28.175+0000 7f52bfb0d700 1 --2- 172.21.15.178:0/741054647 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f52b80f6510 0x7f52b8106730 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:05:28.178 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:28.175+0000 7f52bfb0d700 1 -- 172.21.15.178:0/741054647 >> 172.21.15.178:0/741054647 conn(0x7f52b80f1c70 msgr2=0x7f52b80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:05:28.178 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:28.176+0000 7f52bfb0d700 1 -- 172.21.15.178:0/741054647 shutdown_connections 2024-02-11T17:05:28.178 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:28.176+0000 7f52bfb0d700 1 -- 172.21.15.178:0/741054647 wait complete. 2024-02-11T17:05:28.178 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:05:28.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:28 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:05:27.178412+0000 mgr.smithi037.eofnaq (mgr.14162) 167 : cluster [DBG] pgmap v156: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:05:28.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:28 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:05:28.175117+0000 mon.smithi037 (mon.0) 212 : audit [DBG] from='client.? 172.21.15.178:0/741054647' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:05:29.889 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:05:29.890 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:05:30.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:30 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:05:29.178633+0000 mgr.smithi037.eofnaq (mgr.14162) 168 : cluster [DBG] pgmap v157: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:05:31.752 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:31.748+0000 7f30ab6cb700 1 -- 172.21.15.178:0/3670601105 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f30a40f5410 msgr2=0x7f30a40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:05:31.752 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:31.748+0000 7f30ab6cb700 1 --2- 172.21.15.178:0/3670601105 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f30a40f5410 0x7f30a40f5830 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f3098009fb0 tx=0x7f3098009b30).stop 2024-02-11T17:05:31.752 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:31.748+0000 7f30ab6cb700 1 -- 172.21.15.178:0/3670601105 shutdown_connections 2024-02-11T17:05:31.752 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:31.748+0000 7f30ab6cb700 1 --2- 172.21.15.178:0/3670601105 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f30a40f5410 0x7f30a40f5830 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:05:31.752 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:31.748+0000 7f30ab6cb700 1 -- 172.21.15.178:0/3670601105 >> 172.21.15.178:0/3670601105 conn(0x7f30a40f0b70 msgr2=0x7f30a40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:05:31.752 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:31.748+0000 7f30ab6cb700 1 -- 172.21.15.178:0/3670601105 shutdown_connections 2024-02-11T17:05:31.752 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:31.748+0000 7f30ab6cb700 1 -- 172.21.15.178:0/3670601105 wait complete. 2024-02-11T17:05:31.753 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:31.749+0000 7f30ab6cb700 1 Processor -- start 2024-02-11T17:05:31.753 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:31.749+0000 7f30ab6cb700 1 -- start start 2024-02-11T17:05:31.753 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:31.749+0000 7f30ab6cb700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f30a40f5410 0x7f30a4105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:05:31.753 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:31.749+0000 7f30ab6cb700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30a4105b20 con 0x7f30a40f5410 2024-02-11T17:05:31.753 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:31.749+0000 7f30a9467700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f30a40f5410 0x7f30a4105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:05:31.753 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:31.749+0000 7f30a9467700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f30a40f5410 0x7f30a4105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:41238/0 (socket says 172.21.15.178:41238) 2024-02-11T17:05:31.753 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:31.749+0000 7f30a9467700 1 -- 172.21.15.178:0/1080795961 learned_addr learned my addr 172.21.15.178:0/1080795961 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:05:31.753 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:31.749+0000 7f30a9467700 1 -- 172.21.15.178:0/1080795961 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3098005040 con 0x7f30a40f5410 2024-02-11T17:05:31.753 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:31.750+0000 7f30a9467700 1 --2- 172.21.15.178:0/1080795961 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f30a40f5410 0x7f30a4105630 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f3098014000 tx=0x7f309800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:05:31.753 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:31.750+0000 7f3095ffb700 1 -- 172.21.15.178:0/1080795961 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f30980095f0 con 0x7f30a40f5410 2024-02-11T17:05:31.753 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:31.750+0000 7f30ab6cb700 1 -- 172.21.15.178:0/1080795961 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30a4105d20 con 0x7f30a40f5410 2024-02-11T17:05:31.753 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:31.750+0000 7f3095ffb700 1 -- 172.21.15.178:0/1080795961 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3098007e70 con 0x7f30a40f5410 2024-02-11T17:05:31.753 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:31.750+0000 7f3095ffb700 1 -- 172.21.15.178:0/1080795961 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3098014ea0 con 0x7f30a40f5410 2024-02-11T17:05:31.753 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:31.750+0000 7f30ab6cb700 1 -- 172.21.15.178:0/1080795961 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30a41061b0 con 0x7f30a40f5410 2024-02-11T17:05:31.754 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:31.751+0000 7f3095ffb700 1 -- 172.21.15.178:0/1080795961 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3098025070 con 0x7f30a40f5410 2024-02-11T17:05:31.754 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:31.751+0000 7f30ab6cb700 1 -- 172.21.15.178:0/1080795961 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30a4045ad0 con 0x7f30a40f5410 2024-02-11T17:05:31.754 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:31.751+0000 7f3095ffb700 1 --2- 172.21.15.178:0/1080795961 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f309002ff60 0x7f3090032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:05:31.754 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:31.751+0000 7f3095ffb700 1 -- 172.21.15.178:0/1080795961 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f30980409f0 con 0x7f30a40f5410 2024-02-11T17:05:31.754 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:31.751+0000 7f30a8c66700 1 --2- 172.21.15.178:0/1080795961 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f309002ff60 0x7f3090032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:05:31.754 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:31.751+0000 7f30a8c66700 1 --2- 172.21.15.178:0/1080795961 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f309002ff60 0x7f3090032420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f30a00096a0 tx=0x7f30a0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:05:31.757 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:31.756+0000 7f3095ffb700 1 -- 172.21.15.178:0/1080795961 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f309800f4e0 con 0x7f30a40f5410 2024-02-11T17:05:31.963 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:31.961+0000 7f30ab6cb700 1 -- 172.21.15.178:0/1080795961 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f30a40fae60 con 0x7f30a40f5410 2024-02-11T17:05:31.963 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:31.962+0000 7f3095ffb700 1 -- 172.21.15.178:0/1080795961 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f309800f300 con 0x7f30a40f5410 2024-02-11T17:05:31.963 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:05:31.964 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:05:31.965 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:31.963+0000 7f30ab6cb700 1 -- 172.21.15.178:0/1080795961 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f309002ff60 msgr2=0x7f3090032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:05:31.965 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:31.963+0000 7f30ab6cb700 1 --2- 172.21.15.178:0/1080795961 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f309002ff60 0x7f3090032420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f30a00096a0 tx=0x7f30a0006b40).stop 2024-02-11T17:05:31.965 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:31.963+0000 7f30ab6cb700 1 -- 172.21.15.178:0/1080795961 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f30a40f5410 msgr2=0x7f30a4105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:05:31.965 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:31.963+0000 7f30ab6cb700 1 --2- 172.21.15.178:0/1080795961 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f30a40f5410 0x7f30a4105630 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f3098014000 tx=0x7f309800b040).stop 2024-02-11T17:05:31.965 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:31.963+0000 7f30ab6cb700 1 -- 172.21.15.178:0/1080795961 shutdown_connections 2024-02-11T17:05:31.966 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:31.963+0000 7f30ab6cb700 1 --2- 172.21.15.178:0/1080795961 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f309002ff60 0x7f3090032420 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:05:31.966 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:31.963+0000 7f30ab6cb700 1 --2- 172.21.15.178:0/1080795961 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f30a40f5410 0x7f30a4105630 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:05:31.966 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:31.963+0000 7f30ab6cb700 1 -- 172.21.15.178:0/1080795961 >> 172.21.15.178:0/1080795961 conn(0x7f30a40f0b70 msgr2=0x7f30a40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:05:31.966 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:31.963+0000 7f30ab6cb700 1 -- 172.21.15.178:0/1080795961 shutdown_connections 2024-02-11T17:05:31.966 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:31.963+0000 7f30ab6cb700 1 -- 172.21.15.178:0/1080795961 wait complete. 2024-02-11T17:05:31.966 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:05:32.482 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:32 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:05:31.178897+0000 mgr.smithi037.eofnaq (mgr.14162) 169 : cluster [DBG] pgmap v158: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:05:32.482 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:32 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:05:31.963020+0000 mon.smithi037 (mon.0) 213 : audit [DBG] from='client.? 172.21.15.178:0/1080795961' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:05:33.584 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:05:33.584 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:05:33.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:33 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:05:32.394239+0000 mon.smithi037 (mon.0) 214 : audit [INF] from='mgr.14162 172.21.15.37:0/2745465012' entity='mgr.smithi037.eofnaq' 2024-02-11T17:05:33.684 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:33 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cephadm 2024-02-11T17:05:32.395566+0000 mgr.smithi037.eofnaq (mgr.14162) 170 : cephadm [INF] Deploying daemon node-exporter.smithi037 on smithi037 2024-02-11T17:05:34.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:34 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:05:33.179223+0000 mgr.smithi037.eofnaq (mgr.14162) 171 : cluster [DBG] pgmap v159: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:05:35.520 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:35.498+0000 7fb51d393700 1 -- 172.21.15.178:0/1296292157 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb5180f5410 msgr2=0x7fb5180f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:05:35.521 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:35.498+0000 7fb51d393700 1 --2- 172.21.15.178:0/1296292157 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb5180f5410 0x7fb5180f5830 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7fb500009fb0 tx=0x7fb500009b30).stop 2024-02-11T17:05:35.521 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:35.498+0000 7fb51d393700 1 -- 172.21.15.178:0/1296292157 shutdown_connections 2024-02-11T17:05:35.521 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:35.498+0000 7fb51d393700 1 --2- 172.21.15.178:0/1296292157 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb5180f5410 0x7fb5180f5830 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:05:35.521 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:35.499+0000 7fb51d393700 1 -- 172.21.15.178:0/1296292157 >> 172.21.15.178:0/1296292157 conn(0x7fb5180f0b70 msgr2=0x7fb5180f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:05:35.521 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:35.499+0000 7fb51d393700 1 -- 172.21.15.178:0/1296292157 shutdown_connections 2024-02-11T17:05:35.521 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:35.499+0000 7fb51d393700 1 -- 172.21.15.178:0/1296292157 wait complete. 2024-02-11T17:05:35.521 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:35.499+0000 7fb51d393700 1 Processor -- start 2024-02-11T17:05:35.521 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:35.499+0000 7fb51d393700 1 -- start start 2024-02-11T17:05:35.521 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:35.499+0000 7fb51d393700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb5180f5410 0x7fb518105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:05:35.521 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:35.499+0000 7fb51d393700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb518105af0 con 0x7fb5180f5410 2024-02-11T17:05:35.521 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:35.500+0000 7fb516ffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb5180f5410 0x7fb518105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:05:35.521 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:35.500+0000 7fb516ffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb5180f5410 0x7fb518105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:41262/0 (socket says 172.21.15.178:41262) 2024-02-11T17:05:35.521 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:35.500+0000 7fb516ffd700 1 -- 172.21.15.178:0/3039221217 learned_addr learned my addr 172.21.15.178:0/3039221217 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:05:35.522 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:35.500+0000 7fb516ffd700 1 -- 172.21.15.178:0/3039221217 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb500005040 con 0x7fb5180f5410 2024-02-11T17:05:35.522 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:35.500+0000 7fb516ffd700 1 --2- 172.21.15.178:0/3039221217 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb5180f5410 0x7fb518105600 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7fb500013510 tx=0x7fb50000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:05:35.522 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:35.500+0000 7fb50f7fe700 1 -- 172.21.15.178:0/3039221217 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb500009320 con 0x7fb5180f5410 2024-02-11T17:05:35.522 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:35.500+0000 7fb51d393700 1 -- 172.21.15.178:0/3039221217 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb518105cf0 con 0x7fb5180f5410 2024-02-11T17:05:35.522 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:35.500+0000 7fb50f7fe700 1 -- 172.21.15.178:0/3039221217 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb500013f00 con 0x7fb5180f5410 2024-02-11T17:05:35.522 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:35.500+0000 7fb50f7fe700 1 -- 172.21.15.178:0/3039221217 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb500014d90 con 0x7fb5180f5410 2024-02-11T17:05:35.522 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:35.500+0000 7fb51d393700 1 -- 172.21.15.178:0/3039221217 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb518106180 con 0x7fb5180f5410 2024-02-11T17:05:35.522 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:35.501+0000 7fb50f7fe700 1 -- 172.21.15.178:0/3039221217 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb5000095f0 con 0x7fb5180f5410 2024-02-11T17:05:35.522 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:35.501+0000 7fb51d393700 1 -- 172.21.15.178:0/3039221217 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb518045ad0 con 0x7fb5180f5410 2024-02-11T17:05:35.522 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:35.501+0000 7fb50f7fe700 1 --2- 172.21.15.178:0/3039221217 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb50402ff60 0x7fb504032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:05:35.522 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:35.501+0000 7fb50f7fe700 1 -- 172.21.15.178:0/3039221217 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb500041000 con 0x7fb5180f5410 2024-02-11T17:05:35.522 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:35.502+0000 7fb5167fc700 1 --2- 172.21.15.178:0/3039221217 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb50402ff60 0x7fb504032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:05:35.522 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:35.502+0000 7fb5167fc700 1 --2- 172.21.15.178:0/3039221217 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb50402ff60 0x7fb504032420 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7fb5080096a0 tx=0x7fb508006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:05:35.522 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:35.506+0000 7fb50f7fe700 1 -- 172.21.15.178:0/3039221217 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb500014420 con 0x7fb5180f5410 2024-02-11T17:05:35.710 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:35.708+0000 7fb51d393700 1 -- 172.21.15.178:0/3039221217 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb5180fae60 con 0x7fb5180f5410 2024-02-11T17:05:35.711 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:35.709+0000 7fb50f7fe700 1 -- 172.21.15.178:0/3039221217 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb500010230 con 0x7fb5180f5410 2024-02-11T17:05:35.711 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:05:35.711 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:05:35.713 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:35.711+0000 7fb51d393700 1 -- 172.21.15.178:0/3039221217 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb50402ff60 msgr2=0x7fb504032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:05:35.713 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:35.711+0000 7fb51d393700 1 --2- 172.21.15.178:0/3039221217 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb50402ff60 0x7fb504032420 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7fb5080096a0 tx=0x7fb508006b40).stop 2024-02-11T17:05:35.713 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:35.711+0000 7fb51d393700 1 -- 172.21.15.178:0/3039221217 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb5180f5410 msgr2=0x7fb518105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:05:35.713 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:35.711+0000 7fb51d393700 1 --2- 172.21.15.178:0/3039221217 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb5180f5410 0x7fb518105600 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7fb500013510 tx=0x7fb50000b040).stop 2024-02-11T17:05:35.713 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:35.711+0000 7fb51d393700 1 -- 172.21.15.178:0/3039221217 shutdown_connections 2024-02-11T17:05:35.713 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:35.711+0000 7fb51d393700 1 --2- 172.21.15.178:0/3039221217 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb50402ff60 0x7fb504032420 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:05:35.713 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:35.711+0000 7fb51d393700 1 --2- 172.21.15.178:0/3039221217 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb5180f5410 0x7fb518105600 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:05:35.713 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:35.711+0000 7fb51d393700 1 -- 172.21.15.178:0/3039221217 >> 172.21.15.178:0/3039221217 conn(0x7fb5180f0b70 msgr2=0x7fb5180f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:05:35.713 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:35.711+0000 7fb51d393700 1 -- 172.21.15.178:0/3039221217 shutdown_connections 2024-02-11T17:05:35.713 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:35.711+0000 7fb51d393700 1 -- 172.21.15.178:0/3039221217 wait complete. 2024-02-11T17:05:35.713 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:05:36.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:36 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:05:35.179545+0000 mgr.smithi037.eofnaq (mgr.14162) 172 : cluster [DBG] pgmap v160: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:05:36.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:36 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:05:35.710482+0000 mon.smithi037 (mon.0) 215 : audit [DBG] from='client.? 172.21.15.178:0/3039221217' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:05:37.483 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:05:37.483 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:05:38.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:38 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:05:37.179882+0000 mgr.smithi037.eofnaq (mgr.14162) 173 : cluster [DBG] pgmap v161: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:05:39.616 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:39.607+0000 7f8a8ccb5700 1 -- 172.21.15.178:0/1054134652 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a880f51a0 msgr2=0x7f8a880f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:05:39.616 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:39.607+0000 7f8a8ccb5700 1 --2- 172.21.15.178:0/1054134652 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a880f51a0 0x7f8a880f55c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7f8a78009fb0 tx=0x7f8a78009b30).stop 2024-02-11T17:05:39.616 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:39.608+0000 7f8a8ccb5700 1 -- 172.21.15.178:0/1054134652 shutdown_connections 2024-02-11T17:05:39.616 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:39.608+0000 7f8a8ccb5700 1 --2- 172.21.15.178:0/1054134652 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a880f51a0 0x7f8a880f55c0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:05:39.617 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:39.608+0000 7f8a8ccb5700 1 -- 172.21.15.178:0/1054134652 >> 172.21.15.178:0/1054134652 conn(0x7f8a880f09a0 msgr2=0x7f8a880f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:05:39.617 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:39.608+0000 7f8a8ccb5700 1 -- 172.21.15.178:0/1054134652 shutdown_connections 2024-02-11T17:05:39.617 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:39.608+0000 7f8a8ccb5700 1 -- 172.21.15.178:0/1054134652 wait complete. 2024-02-11T17:05:39.617 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:39.608+0000 7f8a8ccb5700 1 Processor -- start 2024-02-11T17:05:39.617 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:39.608+0000 7f8a8ccb5700 1 -- start start 2024-02-11T17:05:39.617 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:39.608+0000 7f8a8ccb5700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a880f51a0 0x7f8a881064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:05:39.617 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:39.608+0000 7f8a8ccb5700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a881069d0 con 0x7f8a880f51a0 2024-02-11T17:05:39.617 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:39.609+0000 7f8a877fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a880f51a0 0x7f8a881064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:05:39.617 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:39.609+0000 7f8a877fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a880f51a0 0x7f8a881064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:37826/0 (socket says 172.21.15.178:37826) 2024-02-11T17:05:39.617 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:39.609+0000 7f8a877fe700 1 -- 172.21.15.178:0/1300953989 learned_addr learned my addr 172.21.15.178:0/1300953989 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:05:39.617 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:39.609+0000 7f8a877fe700 1 -- 172.21.15.178:0/1300953989 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a78005040 con 0x7f8a880f51a0 2024-02-11T17:05:39.617 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:39.609+0000 7f8a877fe700 1 --2- 172.21.15.178:0/1300953989 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a880f51a0 0x7f8a881064e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f8a78013510 tx=0x7f8a7800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:05:39.617 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:39.609+0000 7f8a6ffff700 1 -- 172.21.15.178:0/1300953989 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a78009320 con 0x7f8a880f51a0 2024-02-11T17:05:39.617 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:39.610+0000 7f8a8ccb5700 1 -- 172.21.15.178:0/1300953989 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a88106bd0 con 0x7f8a880f51a0 2024-02-11T17:05:39.617 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:39.610+0000 7f8a6ffff700 1 -- 172.21.15.178:0/1300953989 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8a78013f00 con 0x7f8a880f51a0 2024-02-11T17:05:39.617 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:39.610+0000 7f8a6ffff700 1 -- 172.21.15.178:0/1300953989 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a78014dc0 con 0x7f8a880f51a0 2024-02-11T17:05:39.618 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:39.610+0000 7f8a8ccb5700 1 -- 172.21.15.178:0/1300953989 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a88107060 con 0x7f8a880f51a0 2024-02-11T17:05:39.618 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:39.610+0000 7f8a6ffff700 1 -- 172.21.15.178:0/1300953989 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8a780095f0 con 0x7f8a880f51a0 2024-02-11T17:05:39.618 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:39.610+0000 7f8a8ccb5700 1 -- 172.21.15.178:0/1300953989 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a88045ad0 con 0x7f8a880f51a0 2024-02-11T17:05:39.618 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:39.611+0000 7f8a6ffff700 1 --2- 172.21.15.178:0/1300953989 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f8a7002ff60 0x7f8a70032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:05:39.618 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:39.611+0000 7f8a6ffff700 1 -- 172.21.15.178:0/1300953989 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8a780410c0 con 0x7f8a880f51a0 2024-02-11T17:05:39.618 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:39.611+0000 7f8a86ffd700 1 --2- 172.21.15.178:0/1300953989 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f8a7002ff60 0x7f8a70032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:05:39.618 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:39.611+0000 7f8a86ffd700 1 --2- 172.21.15.178:0/1300953989 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f8a7002ff60 0x7f8a70032420 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f8a7c0096a0 tx=0x7f8a7c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:05:39.618 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:39.615+0000 7f8a6ffff700 1 -- 172.21.15.178:0/1300953989 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8a7801c790 con 0x7f8a880f51a0 2024-02-11T17:05:39.823 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:39.821+0000 7f8a8ccb5700 1 -- 172.21.15.178:0/1300953989 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8a88107f10 con 0x7f8a880f51a0 2024-02-11T17:05:39.823 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:39.822+0000 7f8a6ffff700 1 -- 172.21.15.178:0/1300953989 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8a78010230 con 0x7f8a880f51a0 2024-02-11T17:05:39.824 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:05:39.824 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:05:39.825 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:39.823+0000 7f8a8ccb5700 1 -- 172.21.15.178:0/1300953989 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f8a7002ff60 msgr2=0x7f8a70032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:05:39.825 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:39.823+0000 7f8a8ccb5700 1 --2- 172.21.15.178:0/1300953989 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f8a7002ff60 0x7f8a70032420 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f8a7c0096a0 tx=0x7f8a7c006b40).stop 2024-02-11T17:05:39.825 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:39.823+0000 7f8a8ccb5700 1 -- 172.21.15.178:0/1300953989 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a880f51a0 msgr2=0x7f8a881064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:05:39.826 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:39.823+0000 7f8a8ccb5700 1 --2- 172.21.15.178:0/1300953989 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a880f51a0 0x7f8a881064e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f8a78013510 tx=0x7f8a7800b040).stop 2024-02-11T17:05:39.826 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:39.824+0000 7f8a8ccb5700 1 -- 172.21.15.178:0/1300953989 shutdown_connections 2024-02-11T17:05:39.826 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:39.824+0000 7f8a8ccb5700 1 --2- 172.21.15.178:0/1300953989 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f8a7002ff60 0x7f8a70032420 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:05:39.826 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:39.824+0000 7f8a8ccb5700 1 --2- 172.21.15.178:0/1300953989 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8a880f51a0 0x7f8a881064e0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:05:39.826 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:39.824+0000 7f8a8ccb5700 1 -- 172.21.15.178:0/1300953989 >> 172.21.15.178:0/1300953989 conn(0x7f8a880f09a0 msgr2=0x7f8a8805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:05:39.826 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:39.824+0000 7f8a8ccb5700 1 -- 172.21.15.178:0/1300953989 shutdown_connections 2024-02-11T17:05:39.826 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:39.824+0000 7f8a8ccb5700 1 -- 172.21.15.178:0/1300953989 wait complete. 2024-02-11T17:05:39.826 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:05:40.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:40 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:05:39.180165+0000 mgr.smithi037.eofnaq (mgr.14162) 174 : cluster [DBG] pgmap v162: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:05:40.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:40 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:05:39.823043+0000 mon.smithi037 (mon.0) 216 : audit [DBG] from='client.? 172.21.15.178:0/1300953989' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:05:41.525 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:05:41.525 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:05:42.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:42 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:05:41.180484+0000 mgr.smithi037.eofnaq (mgr.14162) 175 : cluster [DBG] pgmap v163: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:05:43.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:43.312+0000 7fab5b5b7700 1 -- 172.21.15.178:0/1221699995 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fab540f6510 msgr2=0x7fab540f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:05:43.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:43.312+0000 7fab5b5b7700 1 --2- 172.21.15.178:0/1221699995 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fab540f6510 0x7fab540f6930 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7fab44009fb0 tx=0x7fab44009b30).stop 2024-02-11T17:05:43.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:43.313+0000 7fab5b5b7700 1 -- 172.21.15.178:0/1221699995 shutdown_connections 2024-02-11T17:05:43.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:43.313+0000 7fab5b5b7700 1 --2- 172.21.15.178:0/1221699995 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fab540f6510 0x7fab540f6930 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:05:43.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:43.313+0000 7fab5b5b7700 1 -- 172.21.15.178:0/1221699995 >> 172.21.15.178:0/1221699995 conn(0x7fab540f1c70 msgr2=0x7fab540f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:05:43.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:43.313+0000 7fab5b5b7700 1 -- 172.21.15.178:0/1221699995 shutdown_connections 2024-02-11T17:05:43.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:43.313+0000 7fab5b5b7700 1 -- 172.21.15.178:0/1221699995 wait complete. 2024-02-11T17:05:43.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:43.313+0000 7fab5b5b7700 1 Processor -- start 2024-02-11T17:05:43.316 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:43.313+0000 7fab5b5b7700 1 -- start start 2024-02-11T17:05:43.317 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:43.313+0000 7fab5b5b7700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fab540f6510 0x7fab54106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:05:43.317 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:43.313+0000 7fab5b5b7700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab54106c20 con 0x7fab540f6510 2024-02-11T17:05:43.317 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:43.314+0000 7fab59353700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fab540f6510 0x7fab54106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:05:43.317 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:43.314+0000 7fab59353700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fab540f6510 0x7fab54106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:37852/0 (socket says 172.21.15.178:37852) 2024-02-11T17:05:43.317 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:43.314+0000 7fab59353700 1 -- 172.21.15.178:0/1651782089 learned_addr learned my addr 172.21.15.178:0/1651782089 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:05:43.317 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:43.314+0000 7fab59353700 1 -- 172.21.15.178:0/1651782089 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab44005040 con 0x7fab540f6510 2024-02-11T17:05:43.317 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:43.314+0000 7fab59353700 1 --2- 172.21.15.178:0/1651782089 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fab540f6510 0x7fab54106730 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7fab44014000 tx=0x7fab4400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:05:43.317 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:43.314+0000 7fab49ffb700 1 -- 172.21.15.178:0/1651782089 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab440095f0 con 0x7fab540f6510 2024-02-11T17:05:43.317 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:43.314+0000 7fab5b5b7700 1 -- 172.21.15.178:0/1651782089 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab54106e20 con 0x7fab540f6510 2024-02-11T17:05:43.317 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:43.314+0000 7fab49ffb700 1 -- 172.21.15.178:0/1651782089 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fab44007e70 con 0x7fab540f6510 2024-02-11T17:05:43.317 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:43.314+0000 7fab49ffb700 1 -- 172.21.15.178:0/1651782089 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab44014ea0 con 0x7fab540f6510 2024-02-11T17:05:43.317 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:43.314+0000 7fab5b5b7700 1 -- 172.21.15.178:0/1651782089 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab541072b0 con 0x7fab540f6510 2024-02-11T17:05:43.318 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:43.315+0000 7fab5b5b7700 1 -- 172.21.15.178:0/1651782089 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab54045ad0 con 0x7fab540f6510 2024-02-11T17:05:43.318 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:43.315+0000 7fab49ffb700 1 -- 172.21.15.178:0/1651782089 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fab44025070 con 0x7fab540f6510 2024-02-11T17:05:43.318 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:43.315+0000 7fab49ffb700 1 --2- 172.21.15.178:0/1651782089 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fab4002ff60 0x7fab40032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:05:43.318 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:43.315+0000 7fab49ffb700 1 -- 172.21.15.178:0/1651782089 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fab44040a30 con 0x7fab540f6510 2024-02-11T17:05:43.318 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:43.316+0000 7fab58b52700 1 --2- 172.21.15.178:0/1651782089 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fab4002ff60 0x7fab40032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:05:43.319 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:43.316+0000 7fab58b52700 1 --2- 172.21.15.178:0/1651782089 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fab4002ff60 0x7fab40032420 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fab500096a0 tx=0x7fab50006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:05:43.322 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:43.320+0000 7fab49ffb700 1 -- 172.21.15.178:0/1651782089 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fab440079c0 con 0x7fab540f6510 2024-02-11T17:05:43.526 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:43.525+0000 7fab5b5b7700 1 -- 172.21.15.178:0/1651782089 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fab540fbf60 con 0x7fab540f6510 2024-02-11T17:05:43.527 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:43.525+0000 7fab49ffb700 1 -- 172.21.15.178:0/1651782089 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fab4400f300 con 0x7fab540f6510 2024-02-11T17:05:43.527 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:05:43.527 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:05:43.529 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:43.527+0000 7fab5b5b7700 1 -- 172.21.15.178:0/1651782089 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fab4002ff60 msgr2=0x7fab40032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:05:43.529 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:43.527+0000 7fab5b5b7700 1 --2- 172.21.15.178:0/1651782089 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fab4002ff60 0x7fab40032420 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fab500096a0 tx=0x7fab50006b40).stop 2024-02-11T17:05:43.529 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:43.527+0000 7fab5b5b7700 1 -- 172.21.15.178:0/1651782089 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fab540f6510 msgr2=0x7fab54106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:05:43.529 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:43.527+0000 7fab5b5b7700 1 --2- 172.21.15.178:0/1651782089 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fab540f6510 0x7fab54106730 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7fab44014000 tx=0x7fab4400b040).stop 2024-02-11T17:05:43.529 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:43.527+0000 7fab5b5b7700 1 -- 172.21.15.178:0/1651782089 shutdown_connections 2024-02-11T17:05:43.529 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:43.527+0000 7fab5b5b7700 1 --2- 172.21.15.178:0/1651782089 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fab4002ff60 0x7fab40032420 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:05:43.529 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:43.527+0000 7fab5b5b7700 1 --2- 172.21.15.178:0/1651782089 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fab540f6510 0x7fab54106730 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:05:43.529 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:43.527+0000 7fab5b5b7700 1 -- 172.21.15.178:0/1651782089 >> 172.21.15.178:0/1651782089 conn(0x7fab540f1c70 msgr2=0x7fab540f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:05:43.529 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:43.527+0000 7fab5b5b7700 1 -- 172.21.15.178:0/1651782089 shutdown_connections 2024-02-11T17:05:43.529 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:43.527+0000 7fab5b5b7700 1 -- 172.21.15.178:0/1651782089 wait complete. 2024-02-11T17:05:43.530 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:05:44.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:44 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:05:43.180813+0000 mgr.smithi037.eofnaq (mgr.14162) 176 : cluster [DBG] pgmap v164: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:05:44.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:44 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:05:43.526849+0000 mon.smithi037 (mon.0) 217 : audit [DBG] from='client.? 172.21.15.178:0/1651782089' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:05:45.328 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:05:45.328 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:05:46.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:46 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:05:45.181207+0000 mgr.smithi037.eofnaq (mgr.14162) 177 : cluster [DBG] pgmap v165: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:05:47.275 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:47.272+0000 7fdc81f39700 1 -- 172.21.15.178:0/4163381380 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdc7c0f5410 msgr2=0x7fdc7c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:05:47.275 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:47.272+0000 7fdc81f39700 1 --2- 172.21.15.178:0/4163381380 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdc7c0f5410 0x7fdc7c0f5830 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7fdc70009fb0 tx=0x7fdc70009b30).stop 2024-02-11T17:05:47.275 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:47.272+0000 7fdc81f39700 1 -- 172.21.15.178:0/4163381380 shutdown_connections 2024-02-11T17:05:47.275 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:47.272+0000 7fdc81f39700 1 --2- 172.21.15.178:0/4163381380 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdc7c0f5410 0x7fdc7c0f5830 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:05:47.275 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:47.272+0000 7fdc81f39700 1 -- 172.21.15.178:0/4163381380 >> 172.21.15.178:0/4163381380 conn(0x7fdc7c0f0b70 msgr2=0x7fdc7c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:05:47.276 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:47.272+0000 7fdc81f39700 1 -- 172.21.15.178:0/4163381380 shutdown_connections 2024-02-11T17:05:47.276 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:47.272+0000 7fdc81f39700 1 -- 172.21.15.178:0/4163381380 wait complete. 2024-02-11T17:05:47.276 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:47.273+0000 7fdc81f39700 1 Processor -- start 2024-02-11T17:05:47.276 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:47.273+0000 7fdc81f39700 1 -- start start 2024-02-11T17:05:47.276 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:47.273+0000 7fdc81f39700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdc7c0f5410 0x7fdc7c109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:05:47.276 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:47.273+0000 7fdc81f39700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7c060240 con 0x7fdc7c0f5410 2024-02-11T17:05:47.276 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:47.273+0000 7fdc7b7fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdc7c0f5410 0x7fdc7c109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:05:47.276 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:47.273+0000 7fdc7b7fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdc7c0f5410 0x7fdc7c109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:37878/0 (socket says 172.21.15.178:37878) 2024-02-11T17:05:47.276 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:47.273+0000 7fdc7b7fe700 1 -- 172.21.15.178:0/1597821303 learned_addr learned my addr 172.21.15.178:0/1597821303 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:05:47.276 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:47.274+0000 7fdc7b7fe700 1 -- 172.21.15.178:0/1597821303 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc70005040 con 0x7fdc7c0f5410 2024-02-11T17:05:47.276 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:47.274+0000 7fdc7b7fe700 1 --2- 172.21.15.178:0/1597821303 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdc7c0f5410 0x7fdc7c109430 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7fdc70014000 tx=0x7fdc7000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:05:47.278 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:47.274+0000 7fdc63fff700 1 -- 172.21.15.178:0/1597821303 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc700095f0 con 0x7fdc7c0f5410 2024-02-11T17:05:47.278 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:47.274+0000 7fdc81f39700 1 -- 172.21.15.178:0/1597821303 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc7c060440 con 0x7fdc7c0f5410 2024-02-11T17:05:47.278 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:47.274+0000 7fdc63fff700 1 -- 172.21.15.178:0/1597821303 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdc70007e70 con 0x7fdc7c0f5410 2024-02-11T17:05:47.278 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:47.274+0000 7fdc63fff700 1 -- 172.21.15.178:0/1597821303 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc70014ea0 con 0x7fdc7c0f5410 2024-02-11T17:05:47.278 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:47.274+0000 7fdc81f39700 1 -- 172.21.15.178:0/1597821303 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc7c109ae0 con 0x7fdc7c0f5410 2024-02-11T17:05:47.278 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:47.275+0000 7fdc63fff700 1 -- 172.21.15.178:0/1597821303 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdc70025070 con 0x7fdc7c0f5410 2024-02-11T17:05:47.278 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:47.275+0000 7fdc81f39700 1 -- 172.21.15.178:0/1597821303 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc7c045ad0 con 0x7fdc7c0f5410 2024-02-11T17:05:47.278 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:47.275+0000 7fdc63fff700 1 --2- 172.21.15.178:0/1597821303 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fdc6402ff60 0x7fdc64032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:05:47.278 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:47.275+0000 7fdc63fff700 1 -- 172.21.15.178:0/1597821303 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdc700409b0 con 0x7fdc7c0f5410 2024-02-11T17:05:47.278 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:47.275+0000 7fdc7affd700 1 --2- 172.21.15.178:0/1597821303 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fdc6402ff60 0x7fdc64032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:05:47.278 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:47.276+0000 7fdc7affd700 1 --2- 172.21.15.178:0/1597821303 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fdc6402ff60 0x7fdc64032420 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fdc6c0096a0 tx=0x7fdc6c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:05:47.281 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:47.280+0000 7fdc63fff700 1 -- 172.21.15.178:0/1597821303 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdc7000f4e0 con 0x7fdc7c0f5410 2024-02-11T17:05:47.486 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:47.485+0000 7fdc81f39700 1 -- 172.21.15.178:0/1597821303 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdc7c0fae60 con 0x7fdc7c0f5410 2024-02-11T17:05:47.487 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:47.485+0000 7fdc63fff700 1 -- 172.21.15.178:0/1597821303 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdc7000f300 con 0x7fdc7c0f5410 2024-02-11T17:05:47.487 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:05:47.487 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:05:47.489 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:47.487+0000 7fdc81f39700 1 -- 172.21.15.178:0/1597821303 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fdc6402ff60 msgr2=0x7fdc64032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:05:47.489 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:47.487+0000 7fdc81f39700 1 --2- 172.21.15.178:0/1597821303 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fdc6402ff60 0x7fdc64032420 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fdc6c0096a0 tx=0x7fdc6c006b40).stop 2024-02-11T17:05:47.489 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:47.487+0000 7fdc81f39700 1 -- 172.21.15.178:0/1597821303 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdc7c0f5410 msgr2=0x7fdc7c109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:05:47.489 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:47.487+0000 7fdc81f39700 1 --2- 172.21.15.178:0/1597821303 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdc7c0f5410 0x7fdc7c109430 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7fdc70014000 tx=0x7fdc7000b040).stop 2024-02-11T17:05:47.489 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:47.487+0000 7fdc81f39700 1 -- 172.21.15.178:0/1597821303 shutdown_connections 2024-02-11T17:05:47.489 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:47.487+0000 7fdc81f39700 1 --2- 172.21.15.178:0/1597821303 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fdc6402ff60 0x7fdc64032420 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:05:47.489 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:47.487+0000 7fdc81f39700 1 --2- 172.21.15.178:0/1597821303 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdc7c0f5410 0x7fdc7c109430 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:05:47.490 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:47.487+0000 7fdc81f39700 1 -- 172.21.15.178:0/1597821303 >> 172.21.15.178:0/1597821303 conn(0x7fdc7c0f0b70 msgr2=0x7fdc7c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:05:47.490 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:47.487+0000 7fdc81f39700 1 -- 172.21.15.178:0/1597821303 shutdown_connections 2024-02-11T17:05:47.490 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:47.487+0000 7fdc81f39700 1 -- 172.21.15.178:0/1597821303 wait complete. 2024-02-11T17:05:47.490 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:05:48.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:48 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:05:47.181470+0000 mgr.smithi037.eofnaq (mgr.14162) 178 : cluster [DBG] pgmap v166: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:05:48.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:48 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:05:47.486879+0000 mon.smithi037 (mon.0) 218 : audit [DBG] from='client.? 172.21.15.178:0/1597821303' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:05:49.137 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:05:49.138 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:05:50.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:50 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:05:49.181740+0000 mgr.smithi037.eofnaq (mgr.14162) 179 : cluster [DBG] pgmap v167: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:05:51.122 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:51.118+0000 7f3a191ef700 1 -- 172.21.15.178:0/544181889 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3a140f5410 msgr2=0x7f3a140f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:05:51.122 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:51.118+0000 7f3a191ef700 1 --2- 172.21.15.178:0/544181889 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3a140f5410 0x7f3a140f5830 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7f3a08009fb0 tx=0x7f3a08009b30).stop 2024-02-11T17:05:51.122 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:51.118+0000 7f3a191ef700 1 -- 172.21.15.178:0/544181889 shutdown_connections 2024-02-11T17:05:51.122 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:51.118+0000 7f3a191ef700 1 --2- 172.21.15.178:0/544181889 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3a140f5410 0x7f3a140f5830 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:05:51.122 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:51.118+0000 7f3a191ef700 1 -- 172.21.15.178:0/544181889 >> 172.21.15.178:0/544181889 conn(0x7f3a140f0b70 msgr2=0x7f3a140f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:05:51.123 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:51.118+0000 7f3a191ef700 1 -- 172.21.15.178:0/544181889 shutdown_connections 2024-02-11T17:05:51.123 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:51.118+0000 7f3a191ef700 1 -- 172.21.15.178:0/544181889 wait complete. 2024-02-11T17:05:51.123 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:51.119+0000 7f3a191ef700 1 Processor -- start 2024-02-11T17:05:51.123 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:51.119+0000 7f3a191ef700 1 -- start start 2024-02-11T17:05:51.123 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:51.119+0000 7f3a191ef700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3a140f5410 0x7f3a14105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:05:51.123 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:51.119+0000 7f3a191ef700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a14105b20 con 0x7f3a140f5410 2024-02-11T17:05:51.124 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:51.119+0000 7f3a12d9d700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3a140f5410 0x7f3a14105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:05:51.124 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:51.119+0000 7f3a12d9d700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3a140f5410 0x7f3a14105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:42458/0 (socket says 172.21.15.178:42458) 2024-02-11T17:05:51.124 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:51.119+0000 7f3a12d9d700 1 -- 172.21.15.178:0/2130459038 learned_addr learned my addr 172.21.15.178:0/2130459038 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:05:51.124 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:51.119+0000 7f3a12d9d700 1 -- 172.21.15.178:0/2130459038 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a08005040 con 0x7f3a140f5410 2024-02-11T17:05:51.124 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:51.120+0000 7f3a12d9d700 1 --2- 172.21.15.178:0/2130459038 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3a140f5410 0x7f3a14105630 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f3a08014000 tx=0x7f3a0800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:05:51.124 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:51.120+0000 7f39fb7fe700 1 -- 172.21.15.178:0/2130459038 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a080095f0 con 0x7f3a140f5410 2024-02-11T17:05:51.124 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:51.120+0000 7f3a191ef700 1 -- 172.21.15.178:0/2130459038 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a14105d20 con 0x7f3a140f5410 2024-02-11T17:05:51.124 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:51.120+0000 7f39fb7fe700 1 -- 172.21.15.178:0/2130459038 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3a08007e70 con 0x7f3a140f5410 2024-02-11T17:05:51.124 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:51.120+0000 7f39fb7fe700 1 -- 172.21.15.178:0/2130459038 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a08014ea0 con 0x7f3a140f5410 2024-02-11T17:05:51.124 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:51.120+0000 7f3a191ef700 1 -- 172.21.15.178:0/2130459038 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a141061b0 con 0x7f3a140f5410 2024-02-11T17:05:51.125 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:51.121+0000 7f3a191ef700 1 -- 172.21.15.178:0/2130459038 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a14045ad0 con 0x7f3a140f5410 2024-02-11T17:05:51.125 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:51.121+0000 7f39fb7fe700 1 -- 172.21.15.178:0/2130459038 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3a08025070 con 0x7f3a140f5410 2024-02-11T17:05:51.125 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:51.121+0000 7f39fb7fe700 1 --2- 172.21.15.178:0/2130459038 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f39fc02ff60 0x7f39fc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:05:51.125 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:51.121+0000 7f39fb7fe700 1 -- 172.21.15.178:0/2130459038 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3a08040970 con 0x7f3a140f5410 2024-02-11T17:05:51.125 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:51.121+0000 7f3a1259c700 1 --2- 172.21.15.178:0/2130459038 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f39fc02ff60 0x7f39fc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:05:51.125 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:51.121+0000 7f3a1259c700 1 --2- 172.21.15.178:0/2130459038 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f39fc02ff60 0x7f39fc032420 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f3a040096a0 tx=0x7f3a04006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:05:51.127 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:51.125+0000 7f39fb7fe700 1 -- 172.21.15.178:0/2130459038 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3a080079c0 con 0x7f3a140f5410 2024-02-11T17:05:51.329 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:51.327+0000 7f3a191ef700 1 -- 172.21.15.178:0/2130459038 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3a140fae60 con 0x7f3a140f5410 2024-02-11T17:05:51.329 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:51.328+0000 7f39fb7fe700 1 -- 172.21.15.178:0/2130459038 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3a0800f300 con 0x7f3a140f5410 2024-02-11T17:05:51.330 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:05:51.330 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:05:51.331 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:51.329+0000 7f3a191ef700 1 -- 172.21.15.178:0/2130459038 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f39fc02ff60 msgr2=0x7f39fc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:05:51.331 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:51.329+0000 7f3a191ef700 1 --2- 172.21.15.178:0/2130459038 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f39fc02ff60 0x7f39fc032420 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f3a040096a0 tx=0x7f3a04006b40).stop 2024-02-11T17:05:51.331 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:51.329+0000 7f3a191ef700 1 -- 172.21.15.178:0/2130459038 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3a140f5410 msgr2=0x7f3a14105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:05:51.331 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:51.329+0000 7f3a191ef700 1 --2- 172.21.15.178:0/2130459038 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3a140f5410 0x7f3a14105630 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f3a08014000 tx=0x7f3a0800b040).stop 2024-02-11T17:05:51.331 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:51.329+0000 7f3a191ef700 1 -- 172.21.15.178:0/2130459038 shutdown_connections 2024-02-11T17:05:51.331 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:51.329+0000 7f3a191ef700 1 --2- 172.21.15.178:0/2130459038 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f39fc02ff60 0x7f39fc032420 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:05:51.331 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:51.329+0000 7f3a191ef700 1 --2- 172.21.15.178:0/2130459038 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3a140f5410 0x7f3a14105630 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:05:51.332 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:51.329+0000 7f3a191ef700 1 -- 172.21.15.178:0/2130459038 >> 172.21.15.178:0/2130459038 conn(0x7f3a140f0b70 msgr2=0x7f3a140f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:05:51.332 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:51.329+0000 7f3a191ef700 1 -- 172.21.15.178:0/2130459038 shutdown_connections 2024-02-11T17:05:51.332 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:51.329+0000 7f3a191ef700 1 -- 172.21.15.178:0/2130459038 wait complete. 2024-02-11T17:05:51.332 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:05:52.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:52 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:05:51.182084+0000 mgr.smithi037.eofnaq (mgr.14162) 180 : cluster [DBG] pgmap v168: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:05:52.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:52 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:05:51.329229+0000 mon.smithi037 (mon.0) 219 : audit [DBG] from='client.? 172.21.15.178:0/2130459038' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:05:53.048 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:05:53.049 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:05:54.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:54 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:05:53.182421+0000 mgr.smithi037.eofnaq (mgr.14162) 181 : cluster [DBG] pgmap v169: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:05:55.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:55.055+0000 7f6434e47700 1 -- 172.21.15.178:0/4110169490 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f64300f5410 msgr2=0x7f64300f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:05:55.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:55.055+0000 7f6434e47700 1 --2- 172.21.15.178:0/4110169490 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f64300f5410 0x7f64300f5830 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f6418009fb0 tx=0x7f6418009b30).stop 2024-02-11T17:05:55.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:55.055+0000 7f6434e47700 1 -- 172.21.15.178:0/4110169490 shutdown_connections 2024-02-11T17:05:55.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:55.055+0000 7f6434e47700 1 --2- 172.21.15.178:0/4110169490 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f64300f5410 0x7f64300f5830 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:05:55.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:55.055+0000 7f6434e47700 1 -- 172.21.15.178:0/4110169490 >> 172.21.15.178:0/4110169490 conn(0x7f64300f0b70 msgr2=0x7f64300f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:05:55.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:55.055+0000 7f6434e47700 1 -- 172.21.15.178:0/4110169490 shutdown_connections 2024-02-11T17:05:55.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:55.055+0000 7f6434e47700 1 -- 172.21.15.178:0/4110169490 wait complete. 2024-02-11T17:05:55.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:55.056+0000 7f6434e47700 1 Processor -- start 2024-02-11T17:05:55.059 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:55.056+0000 7f6434e47700 1 -- start start 2024-02-11T17:05:55.059 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:55.056+0000 7f6434e47700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f64300f5410 0x7f6430109400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:05:55.059 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:55.056+0000 7f6434e47700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6430060240 con 0x7f64300f5410 2024-02-11T17:05:55.059 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:55.056+0000 7f642e59c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f64300f5410 0x7f6430109400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:05:55.059 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:55.056+0000 7f642e59c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f64300f5410 0x7f6430109400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:42488/0 (socket says 172.21.15.178:42488) 2024-02-11T17:05:55.059 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:55.056+0000 7f642e59c700 1 -- 172.21.15.178:0/3747840242 learned_addr learned my addr 172.21.15.178:0/3747840242 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:05:55.059 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:55.057+0000 7f642e59c700 1 -- 172.21.15.178:0/3747840242 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6418005040 con 0x7f64300f5410 2024-02-11T17:05:55.059 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:55.057+0000 7f642e59c700 1 --2- 172.21.15.178:0/3747840242 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f64300f5410 0x7f6430109400 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f6418013510 tx=0x7f641800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:05:55.059 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:55.057+0000 7f6426ffd700 1 -- 172.21.15.178:0/3747840242 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6418009320 con 0x7f64300f5410 2024-02-11T17:05:55.059 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:55.057+0000 7f6434e47700 1 -- 172.21.15.178:0/3747840242 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6430060440 con 0x7f64300f5410 2024-02-11T17:05:55.059 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:55.057+0000 7f6426ffd700 1 -- 172.21.15.178:0/3747840242 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6418013f00 con 0x7f64300f5410 2024-02-11T17:05:55.059 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:55.057+0000 7f6426ffd700 1 -- 172.21.15.178:0/3747840242 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6418014d90 con 0x7f64300f5410 2024-02-11T17:05:55.059 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:55.057+0000 7f6434e47700 1 -- 172.21.15.178:0/3747840242 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6430109ab0 con 0x7f64300f5410 2024-02-11T17:05:55.060 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:55.058+0000 7f6426ffd700 1 -- 172.21.15.178:0/3747840242 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f64180095f0 con 0x7f64300f5410 2024-02-11T17:05:55.061 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:55.058+0000 7f6434e47700 1 -- 172.21.15.178:0/3747840242 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6430045ad0 con 0x7f64300f5410 2024-02-11T17:05:55.061 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:55.058+0000 7f6426ffd700 1 --2- 172.21.15.178:0/3747840242 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f641c02ff60 0x7f641c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:05:55.061 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:55.058+0000 7f6426ffd700 1 -- 172.21.15.178:0/3747840242 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6418041000 con 0x7f64300f5410 2024-02-11T17:05:55.061 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:55.058+0000 7f642dd9b700 1 --2- 172.21.15.178:0/3747840242 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f641c02ff60 0x7f641c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:05:55.061 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:55.059+0000 7f642dd9b700 1 --2- 172.21.15.178:0/3747840242 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f641c02ff60 0x7f641c032420 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f64200096a0 tx=0x7f6420006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:05:55.063 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:55.063+0000 7f6426ffd700 1 -- 172.21.15.178:0/3747840242 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f64180268c0 con 0x7f64300f5410 2024-02-11T17:05:55.274 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:55.273+0000 7f6434e47700 1 -- 172.21.15.178:0/3747840242 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f64300fae60 con 0x7f64300f5410 2024-02-11T17:05:55.274 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:55.274+0000 7f6426ffd700 1 -- 172.21.15.178:0/3747840242 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6418010230 con 0x7f64300f5410 2024-02-11T17:05:55.275 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:05:55.275 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:05:55.276 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:55.276+0000 7f6434e47700 1 -- 172.21.15.178:0/3747840242 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f641c02ff60 msgr2=0x7f641c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:05:55.276 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:55.276+0000 7f6434e47700 1 --2- 172.21.15.178:0/3747840242 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f641c02ff60 0x7f641c032420 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f64200096a0 tx=0x7f6420006b40).stop 2024-02-11T17:05:55.276 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:55.276+0000 7f6434e47700 1 -- 172.21.15.178:0/3747840242 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f64300f5410 msgr2=0x7f6430109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:05:55.277 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:55.276+0000 7f6434e47700 1 --2- 172.21.15.178:0/3747840242 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f64300f5410 0x7f6430109400 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f6418013510 tx=0x7f641800b040).stop 2024-02-11T17:05:55.277 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:55.276+0000 7f6434e47700 1 -- 172.21.15.178:0/3747840242 shutdown_connections 2024-02-11T17:05:55.277 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:55.276+0000 7f6434e47700 1 --2- 172.21.15.178:0/3747840242 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f641c02ff60 0x7f641c032420 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:05:55.277 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:55.276+0000 7f6434e47700 1 --2- 172.21.15.178:0/3747840242 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f64300f5410 0x7f6430109400 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:05:55.277 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:55.276+0000 7f6434e47700 1 -- 172.21.15.178:0/3747840242 >> 172.21.15.178:0/3747840242 conn(0x7f64300f0b70 msgr2=0x7f64300f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:05:55.277 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:55.276+0000 7f6434e47700 1 -- 172.21.15.178:0/3747840242 shutdown_connections 2024-02-11T17:05:55.277 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:55.276+0000 7f6434e47700 1 -- 172.21.15.178:0/3747840242 wait complete. 2024-02-11T17:05:55.277 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:05:56.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:56 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:05:55.182721+0000 mgr.smithi037.eofnaq (mgr.14162) 182 : cluster [DBG] pgmap v170: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:05:56.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:56 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:05:55.274243+0000 mon.smithi037 (mon.0) 220 : audit [DBG] from='client.? 172.21.15.178:0/3747840242' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:05:57.062 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:05:57.062 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:05:58.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:05:58 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:05:57.182994+0000 mgr.smithi037.eofnaq (mgr.14162) 183 : cluster [DBG] pgmap v171: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:05:59.141 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:59.141+0000 7f1c01925700 1 -- 172.21.15.178:0/4042381138 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1bfc0f47c0 msgr2=0x7f1bfc0f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:05:59.142 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:59.141+0000 7f1c01925700 1 --2- 172.21.15.178:0/4042381138 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1bfc0f47c0 0x7f1bfc0f4be0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7f1bf0009fb0 tx=0x7f1bf0009b30).stop 2024-02-11T17:05:59.142 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:59.141+0000 7f1c01925700 1 -- 172.21.15.178:0/4042381138 shutdown_connections 2024-02-11T17:05:59.142 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:59.141+0000 7f1c01925700 1 --2- 172.21.15.178:0/4042381138 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1bfc0f47c0 0x7f1bfc0f4be0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:05:59.142 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:59.141+0000 7f1c01925700 1 -- 172.21.15.178:0/4042381138 >> 172.21.15.178:0/4042381138 conn(0x7f1bfc0f02e0 msgr2=0x7f1bfc0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:05:59.142 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:59.141+0000 7f1c01925700 1 -- 172.21.15.178:0/4042381138 shutdown_connections 2024-02-11T17:05:59.142 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:59.141+0000 7f1c01925700 1 -- 172.21.15.178:0/4042381138 wait complete. 2024-02-11T17:05:59.143 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:59.141+0000 7f1c01925700 1 Processor -- start 2024-02-11T17:05:59.144 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:59.142+0000 7f1c01925700 1 -- start start 2024-02-11T17:05:59.144 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:59.142+0000 7f1c01925700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1bfc0f47c0 0x7f1bfc106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:05:59.144 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:59.142+0000 7f1c01925700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bfc106bf0 con 0x7f1bfc0f47c0 2024-02-11T17:05:59.144 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:59.142+0000 7f1bfaffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1bfc0f47c0 0x7f1bfc106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:05:59.144 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:59.142+0000 7f1bfaffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1bfc0f47c0 0x7f1bfc106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:42500/0 (socket says 172.21.15.178:42500) 2024-02-11T17:05:59.144 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:59.142+0000 7f1bfaffd700 1 -- 172.21.15.178:0/1004619123 learned_addr learned my addr 172.21.15.178:0/1004619123 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:05:59.144 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:59.142+0000 7f1bfaffd700 1 -- 172.21.15.178:0/1004619123 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bf0005040 con 0x7f1bfc0f47c0 2024-02-11T17:05:59.144 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:59.143+0000 7f1bfaffd700 1 --2- 172.21.15.178:0/1004619123 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1bfc0f47c0 0x7f1bfc106700 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f1bf0013510 tx=0x7f1bf000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:05:59.144 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:59.143+0000 7f1be37fe700 1 -- 172.21.15.178:0/1004619123 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1bf0009320 con 0x7f1bfc0f47c0 2024-02-11T17:05:59.144 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:59.143+0000 7f1be37fe700 1 -- 172.21.15.178:0/1004619123 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1bf0013f00 con 0x7f1bfc0f47c0 2024-02-11T17:05:59.144 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:59.143+0000 7f1c01925700 1 -- 172.21.15.178:0/1004619123 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bfc106df0 con 0x7f1bfc0f47c0 2024-02-11T17:05:59.144 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:59.143+0000 7f1be37fe700 1 -- 172.21.15.178:0/1004619123 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1bf0014d90 con 0x7f1bfc0f47c0 2024-02-11T17:05:59.145 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:59.143+0000 7f1c01925700 1 -- 172.21.15.178:0/1004619123 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bfc107280 con 0x7f1bfc0f47c0 2024-02-11T17:05:59.145 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:59.144+0000 7f1be37fe700 1 -- 172.21.15.178:0/1004619123 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1bf0014420 con 0x7f1bfc0f47c0 2024-02-11T17:05:59.146 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:59.144+0000 7f1c01925700 1 -- 172.21.15.178:0/1004619123 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bfc045ad0 con 0x7f1bfc0f47c0 2024-02-11T17:05:59.146 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:59.144+0000 7f1be37fe700 1 --2- 172.21.15.178:0/1004619123 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f1be402ff60 0x7f1be4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:05:59.146 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:59.144+0000 7f1be37fe700 1 -- 172.21.15.178:0/1004619123 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1bf0041020 con 0x7f1bfc0f47c0 2024-02-11T17:05:59.146 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:59.144+0000 7f1bfa7fc700 1 --2- 172.21.15.178:0/1004619123 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f1be402ff60 0x7f1be4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:05:59.146 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:59.144+0000 7f1bfa7fc700 1 --2- 172.21.15.178:0/1004619123 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f1be402ff60 0x7f1be4032420 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f1bec0096a0 tx=0x7f1bec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:05:59.150 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:59.149+0000 7f1be37fe700 1 -- 172.21.15.178:0/1004619123 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1bf00095f0 con 0x7f1bfc0f47c0 2024-02-11T17:05:59.355 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:59.354+0000 7f1c01925700 1 -- 172.21.15.178:0/1004619123 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1bfc108130 con 0x7f1bfc0f47c0 2024-02-11T17:05:59.355 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:59.355+0000 7f1be37fe700 1 -- 172.21.15.178:0/1004619123 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1bf001db70 con 0x7f1bfc0f47c0 2024-02-11T17:05:59.356 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:05:59.356 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:05:59.357 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:59.356+0000 7f1c01925700 1 -- 172.21.15.178:0/1004619123 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f1be402ff60 msgr2=0x7f1be4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:05:59.357 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:59.356+0000 7f1c01925700 1 --2- 172.21.15.178:0/1004619123 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f1be402ff60 0x7f1be4032420 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f1bec0096a0 tx=0x7f1bec006b40).stop 2024-02-11T17:05:59.357 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:59.356+0000 7f1c01925700 1 -- 172.21.15.178:0/1004619123 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1bfc0f47c0 msgr2=0x7f1bfc106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:05:59.357 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:59.356+0000 7f1c01925700 1 --2- 172.21.15.178:0/1004619123 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1bfc0f47c0 0x7f1bfc106700 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f1bf0013510 tx=0x7f1bf000b040).stop 2024-02-11T17:05:59.357 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:59.356+0000 7f1c01925700 1 -- 172.21.15.178:0/1004619123 shutdown_connections 2024-02-11T17:05:59.358 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:59.356+0000 7f1c01925700 1 --2- 172.21.15.178:0/1004619123 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f1be402ff60 0x7f1be4032420 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:05:59.358 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:59.356+0000 7f1c01925700 1 --2- 172.21.15.178:0/1004619123 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1bfc0f47c0 0x7f1bfc106700 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:05:59.358 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:59.356+0000 7f1c01925700 1 -- 172.21.15.178:0/1004619123 >> 172.21.15.178:0/1004619123 conn(0x7f1bfc0f02e0 msgr2=0x7f1bfc0f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:05:59.358 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:59.356+0000 7f1c01925700 1 -- 172.21.15.178:0/1004619123 shutdown_connections 2024-02-11T17:05:59.358 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:05:59.356+0000 7f1c01925700 1 -- 172.21.15.178:0/1004619123 wait complete. 2024-02-11T17:05:59.358 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:06:00.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:06:00 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:05:59.183324+0000 mgr.smithi037.eofnaq (mgr.14162) 184 : cluster [DBG] pgmap v172: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:06:00.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:06:00 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:05:59.354989+0000 mon.smithi037 (mon.0) 221 : audit [DBG] from='client.? 172.21.15.178:0/1004619123' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:06:01.044 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:06:01.045 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:06:02.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:06:02 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:06:01.183580+0000 mgr.smithi037.eofnaq (mgr.14162) 185 : cluster [DBG] pgmap v173: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:06:03.026 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:03.023+0000 7f356b58f700 1 -- 172.21.15.178:0/2962691348 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f35640f6510 msgr2=0x7f35640f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:06:03.026 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:03.023+0000 7f356b58f700 1 --2- 172.21.15.178:0/2962691348 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f35640f6510 0x7f35640f6930 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f3554009fb0 tx=0x7f3554009b30).stop 2024-02-11T17:06:03.026 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:03.023+0000 7f356b58f700 1 -- 172.21.15.178:0/2962691348 shutdown_connections 2024-02-11T17:06:03.026 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:03.023+0000 7f356b58f700 1 --2- 172.21.15.178:0/2962691348 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f35640f6510 0x7f35640f6930 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:06:03.027 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:03.023+0000 7f356b58f700 1 -- 172.21.15.178:0/2962691348 >> 172.21.15.178:0/2962691348 conn(0x7f35640f1c70 msgr2=0x7f35640f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:06:03.027 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:03.023+0000 7f356b58f700 1 -- 172.21.15.178:0/2962691348 shutdown_connections 2024-02-11T17:06:03.027 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:03.023+0000 7f356b58f700 1 -- 172.21.15.178:0/2962691348 wait complete. 2024-02-11T17:06:03.027 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:03.023+0000 7f356b58f700 1 Processor -- start 2024-02-11T17:06:03.027 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:03.023+0000 7f356b58f700 1 -- start start 2024-02-11T17:06:03.027 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:03.023+0000 7f356b58f700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f35640f6510 0x7f3564106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:06:03.027 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:03.023+0000 7f356b58f700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3564106c20 con 0x7f35640f6510 2024-02-11T17:06:03.027 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:03.024+0000 7f356932b700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f35640f6510 0x7f3564106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:06:03.027 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:03.024+0000 7f356932b700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f35640f6510 0x7f3564106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:53618/0 (socket says 172.21.15.178:53618) 2024-02-11T17:06:03.027 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:03.024+0000 7f356932b700 1 -- 172.21.15.178:0/1031419555 learned_addr learned my addr 172.21.15.178:0/1031419555 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:06:03.027 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:03.024+0000 7f356932b700 1 -- 172.21.15.178:0/1031419555 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3554005040 con 0x7f35640f6510 2024-02-11T17:06:03.027 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:03.024+0000 7f356932b700 1 --2- 172.21.15.178:0/1031419555 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f35640f6510 0x7f3564106730 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f3554013510 tx=0x7f355400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:06:03.027 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:03.024+0000 7f3559ffb700 1 -- 172.21.15.178:0/1031419555 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3554009320 con 0x7f35640f6510 2024-02-11T17:06:03.028 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:03.024+0000 7f3559ffb700 1 -- 172.21.15.178:0/1031419555 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3554013f00 con 0x7f35640f6510 2024-02-11T17:06:03.028 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:03.024+0000 7f356b58f700 1 -- 172.21.15.178:0/1031419555 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3564106e20 con 0x7f35640f6510 2024-02-11T17:06:03.028 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:03.024+0000 7f3559ffb700 1 -- 172.21.15.178:0/1031419555 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3554014d90 con 0x7f35640f6510 2024-02-11T17:06:03.028 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:03.025+0000 7f356b58f700 1 -- 172.21.15.178:0/1031419555 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35641072b0 con 0x7f35640f6510 2024-02-11T17:06:03.029 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:03.025+0000 7f356b58f700 1 -- 172.21.15.178:0/1031419555 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3564045ad0 con 0x7f35640f6510 2024-02-11T17:06:03.029 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:03.025+0000 7f3559ffb700 1 -- 172.21.15.178:0/1031419555 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f35540095f0 con 0x7f35640f6510 2024-02-11T17:06:03.029 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:03.025+0000 7f3559ffb700 1 --2- 172.21.15.178:0/1031419555 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f355002ff60 0x7f3550032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:06:03.029 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:03.026+0000 7f3559ffb700 1 -- 172.21.15.178:0/1031419555 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3554041000 con 0x7f35640f6510 2024-02-11T17:06:03.029 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:03.026+0000 7f3568b2a700 1 --2- 172.21.15.178:0/1031419555 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f355002ff60 0x7f3550032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:06:03.029 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:03.026+0000 7f3568b2a700 1 --2- 172.21.15.178:0/1031419555 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f355002ff60 0x7f3550032420 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f35600096a0 tx=0x7f3560006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:06:03.032 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:03.030+0000 7f3559ffb700 1 -- 172.21.15.178:0/1031419555 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3554014420 con 0x7f35640f6510 2024-02-11T17:06:03.236 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:03.234+0000 7f356b58f700 1 -- 172.21.15.178:0/1031419555 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f35640fbf60 con 0x7f35640f6510 2024-02-11T17:06:03.237 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:03.235+0000 7f3559ffb700 1 -- 172.21.15.178:0/1031419555 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f355401d7f0 con 0x7f35640f6510 2024-02-11T17:06:03.237 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:06:03.237 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:06:03.239 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:03.237+0000 7f356b58f700 1 -- 172.21.15.178:0/1031419555 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f355002ff60 msgr2=0x7f3550032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:06:03.239 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:03.237+0000 7f356b58f700 1 --2- 172.21.15.178:0/1031419555 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f355002ff60 0x7f3550032420 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f35600096a0 tx=0x7f3560006b40).stop 2024-02-11T17:06:03.240 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:03.237+0000 7f356b58f700 1 -- 172.21.15.178:0/1031419555 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f35640f6510 msgr2=0x7f3564106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:06:03.240 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:03.237+0000 7f356b58f700 1 --2- 172.21.15.178:0/1031419555 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f35640f6510 0x7f3564106730 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f3554013510 tx=0x7f355400b040).stop 2024-02-11T17:06:03.240 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:03.237+0000 7f356b58f700 1 -- 172.21.15.178:0/1031419555 shutdown_connections 2024-02-11T17:06:03.240 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:03.237+0000 7f356b58f700 1 --2- 172.21.15.178:0/1031419555 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f355002ff60 0x7f3550032420 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:06:03.240 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:03.237+0000 7f356b58f700 1 --2- 172.21.15.178:0/1031419555 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f35640f6510 0x7f3564106730 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:06:03.240 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:03.237+0000 7f356b58f700 1 -- 172.21.15.178:0/1031419555 >> 172.21.15.178:0/1031419555 conn(0x7f35640f1c70 msgr2=0x7f35640f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:06:03.240 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:03.237+0000 7f356b58f700 1 -- 172.21.15.178:0/1031419555 shutdown_connections 2024-02-11T17:06:03.240 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:03.237+0000 7f356b58f700 1 -- 172.21.15.178:0/1031419555 wait complete. 2024-02-11T17:06:03.241 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:06:03.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:06:03 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:06:03.236453+0000 mon.smithi037 (mon.0) 222 : audit [DBG] from='client.? 172.21.15.178:0/1031419555' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:06:04.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:06:04 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:06:03.183951+0000 mgr.smithi037.eofnaq (mgr.14162) 186 : cluster [DBG] pgmap v174: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:06:04.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:06:04 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:06:03.276046+0000 mon.smithi037 (mon.0) 223 : audit [INF] from='mgr.14162 172.21.15.37:0/2745465012' entity='mgr.smithi037.eofnaq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi037.eofnaq/mirror_snapshot_schedule"}]: dispatch 2024-02-11T17:06:04.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:06:04 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:06:03.279246+0000 mon.smithi037 (mon.0) 224 : audit [INF] from='mgr.14162 172.21.15.37:0/2745465012' entity='mgr.smithi037.eofnaq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi037.eofnaq/trash_purge_schedule"}]: dispatch 2024-02-11T17:06:04.857 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:06:04.857 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:06:06.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:06:06 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:06:05.184145+0000 mgr.smithi037.eofnaq (mgr.14162) 187 : cluster [DBG] pgmap v175: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:06:06.807 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:06.803+0000 7f39aff7b700 1 -- 172.21.15.178:0/2005786928 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f39a80f5410 msgr2=0x7f39a80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:06:06.807 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:06.803+0000 7f39aff7b700 1 --2- 172.21.15.178:0/2005786928 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f39a80f5410 0x7f39a80f5830 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f3998009fb0 tx=0x7f3998009b30).stop 2024-02-11T17:06:06.807 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:06.803+0000 7f39aff7b700 1 -- 172.21.15.178:0/2005786928 shutdown_connections 2024-02-11T17:06:06.807 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:06.803+0000 7f39aff7b700 1 --2- 172.21.15.178:0/2005786928 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f39a80f5410 0x7f39a80f5830 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:06:06.807 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:06.803+0000 7f39aff7b700 1 -- 172.21.15.178:0/2005786928 >> 172.21.15.178:0/2005786928 conn(0x7f39a80f0b70 msgr2=0x7f39a80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:06:06.807 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:06.803+0000 7f39aff7b700 1 -- 172.21.15.178:0/2005786928 shutdown_connections 2024-02-11T17:06:06.807 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:06.803+0000 7f39aff7b700 1 -- 172.21.15.178:0/2005786928 wait complete. 2024-02-11T17:06:06.807 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:06.804+0000 7f39aff7b700 1 Processor -- start 2024-02-11T17:06:06.807 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:06.804+0000 7f39aff7b700 1 -- start start 2024-02-11T17:06:06.807 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:06.804+0000 7f39aff7b700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f39a80f5410 0x7f39a8106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:06:06.807 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:06.804+0000 7f39aff7b700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39a8106c10 con 0x7f39a80f5410 2024-02-11T17:06:06.808 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:06.804+0000 7f39add17700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f39a80f5410 0x7f39a8106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:06:06.808 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:06.804+0000 7f39add17700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f39a80f5410 0x7f39a8106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:53634/0 (socket says 172.21.15.178:53634) 2024-02-11T17:06:06.808 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:06.804+0000 7f39add17700 1 -- 172.21.15.178:0/971508676 learned_addr learned my addr 172.21.15.178:0/971508676 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:06:06.808 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:06.805+0000 7f39add17700 1 -- 172.21.15.178:0/971508676 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3998005040 con 0x7f39a80f5410 2024-02-11T17:06:06.808 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:06.805+0000 7f39add17700 1 --2- 172.21.15.178:0/971508676 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f39a80f5410 0x7f39a8106720 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f3998013510 tx=0x7f399800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:06:06.808 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:06.805+0000 7f399e7fc700 1 -- 172.21.15.178:0/971508676 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3998009320 con 0x7f39a80f5410 2024-02-11T17:06:06.808 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:06.805+0000 7f39aff7b700 1 -- 172.21.15.178:0/971508676 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39a8106e10 con 0x7f39a80f5410 2024-02-11T17:06:06.808 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:06.805+0000 7f399e7fc700 1 -- 172.21.15.178:0/971508676 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3998013f00 con 0x7f39a80f5410 2024-02-11T17:06:06.808 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:06.805+0000 7f399e7fc700 1 -- 172.21.15.178:0/971508676 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3998014d90 con 0x7f39a80f5410 2024-02-11T17:06:06.808 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:06.805+0000 7f39aff7b700 1 -- 172.21.15.178:0/971508676 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39a81072a0 con 0x7f39a80f5410 2024-02-11T17:06:06.809 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:06.806+0000 7f399e7fc700 1 -- 172.21.15.178:0/971508676 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3998014420 con 0x7f39a80f5410 2024-02-11T17:06:06.809 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:06.806+0000 7f39aff7b700 1 -- 172.21.15.178:0/971508676 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39a8045ad0 con 0x7f39a80f5410 2024-02-11T17:06:06.809 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:06.806+0000 7f399e7fc700 1 --2- 172.21.15.178:0/971508676 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f399402ff60 0x7f3994032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:06:06.809 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:06.806+0000 7f399e7fc700 1 -- 172.21.15.178:0/971508676 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3998040ff0 con 0x7f39a80f5410 2024-02-11T17:06:06.809 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:06.806+0000 7f39ad516700 1 --2- 172.21.15.178:0/971508676 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f399402ff60 0x7f3994032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:06:06.809 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:06.807+0000 7f39ad516700 1 --2- 172.21.15.178:0/971508676 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f399402ff60 0x7f3994032420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f39a40096a0 tx=0x7f39a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:06:06.812 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:06.811+0000 7f399e7fc700 1 -- 172.21.15.178:0/971508676 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f39980095f0 con 0x7f39a80f5410 2024-02-11T17:06:07.017 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:07.015+0000 7f39aff7b700 1 -- 172.21.15.178:0/971508676 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f39a8108150 con 0x7f39a80f5410 2024-02-11T17:06:07.017 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:07.016+0000 7f399e7fc700 1 -- 172.21.15.178:0/971508676 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f399801db70 con 0x7f39a80f5410 2024-02-11T17:06:07.018 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:06:07.018 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:06:07.019 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:07.018+0000 7f39aff7b700 1 -- 172.21.15.178:0/971508676 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f399402ff60 msgr2=0x7f3994032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:06:07.019 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:07.018+0000 7f39aff7b700 1 --2- 172.21.15.178:0/971508676 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f399402ff60 0x7f3994032420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f39a40096a0 tx=0x7f39a4006b40).stop 2024-02-11T17:06:07.020 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:07.018+0000 7f39aff7b700 1 -- 172.21.15.178:0/971508676 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f39a80f5410 msgr2=0x7f39a8106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:06:07.020 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:07.018+0000 7f39aff7b700 1 --2- 172.21.15.178:0/971508676 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f39a80f5410 0x7f39a8106720 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f3998013510 tx=0x7f399800b040).stop 2024-02-11T17:06:07.020 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:07.018+0000 7f39aff7b700 1 -- 172.21.15.178:0/971508676 shutdown_connections 2024-02-11T17:06:07.020 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:07.018+0000 7f39aff7b700 1 --2- 172.21.15.178:0/971508676 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f399402ff60 0x7f3994032420 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:06:07.020 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:07.018+0000 7f39aff7b700 1 --2- 172.21.15.178:0/971508676 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f39a80f5410 0x7f39a8106720 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:06:07.020 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:07.018+0000 7f39aff7b700 1 -- 172.21.15.178:0/971508676 >> 172.21.15.178:0/971508676 conn(0x7f39a80f0b70 msgr2=0x7f39a805fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:06:07.020 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:07.018+0000 7f39aff7b700 1 -- 172.21.15.178:0/971508676 shutdown_connections 2024-02-11T17:06:07.020 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:07.018+0000 7f39aff7b700 1 -- 172.21.15.178:0/971508676 wait complete. 2024-02-11T17:06:07.020 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:06:07.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:06:07 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:06:07.017323+0000 mon.smithi037 (mon.0) 225 : audit [DBG] from='client.? 172.21.15.178:0/971508676' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:06:08.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:06:08 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:06:07.184437+0000 mgr.smithi037.eofnaq (mgr.14162) 188 : cluster [DBG] pgmap v176: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:06:08.667 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:06:08.668 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:06:10.642 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:10.639+0000 7fa21d833700 1 -- 172.21.15.178:0/4106422480 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa2180f5410 msgr2=0x7fa2180f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:06:10.643 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:10.639+0000 7fa21d833700 1 --2- 172.21.15.178:0/4106422480 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa2180f5410 0x7fa2180f5830 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7fa20c009fb0 tx=0x7fa20c009b30).stop 2024-02-11T17:06:10.643 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:10.639+0000 7fa21d833700 1 -- 172.21.15.178:0/4106422480 shutdown_connections 2024-02-11T17:06:10.643 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:10.639+0000 7fa21d833700 1 --2- 172.21.15.178:0/4106422480 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa2180f5410 0x7fa2180f5830 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:06:10.643 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:10.639+0000 7fa21d833700 1 -- 172.21.15.178:0/4106422480 >> 172.21.15.178:0/4106422480 conn(0x7fa2180f0b70 msgr2=0x7fa2180f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:06:10.643 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:10.639+0000 7fa21d833700 1 -- 172.21.15.178:0/4106422480 shutdown_connections 2024-02-11T17:06:10.643 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:10.639+0000 7fa21d833700 1 -- 172.21.15.178:0/4106422480 wait complete. 2024-02-11T17:06:10.643 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:10.640+0000 7fa21d833700 1 Processor -- start 2024-02-11T17:06:10.643 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:10.640+0000 7fa21d833700 1 -- start start 2024-02-11T17:06:10.643 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:10.640+0000 7fa21d833700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa2180f5410 0x7fa218109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:06:10.643 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:10.640+0000 7fa21d833700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa218060240 con 0x7fa2180f5410 2024-02-11T17:06:10.643 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:10.640+0000 7fa216ffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa2180f5410 0x7fa218109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:06:10.643 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:10.640+0000 7fa216ffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa2180f5410 0x7fa218109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:50236/0 (socket says 172.21.15.178:50236) 2024-02-11T17:06:10.644 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:10.640+0000 7fa216ffd700 1 -- 172.21.15.178:0/3822714459 learned_addr learned my addr 172.21.15.178:0/3822714459 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:06:10.644 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:10.641+0000 7fa216ffd700 1 -- 172.21.15.178:0/3822714459 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa20c005040 con 0x7fa2180f5410 2024-02-11T17:06:10.644 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:10.641+0000 7fa216ffd700 1 --2- 172.21.15.178:0/3822714459 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa2180f5410 0x7fa218109430 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7fa20c013510 tx=0x7fa20c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:06:10.644 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:10.641+0000 7fa1ff7fe700 1 -- 172.21.15.178:0/3822714459 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa20c009320 con 0x7fa2180f5410 2024-02-11T17:06:10.644 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:10.641+0000 7fa21d833700 1 -- 172.21.15.178:0/3822714459 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa218060440 con 0x7fa2180f5410 2024-02-11T17:06:10.644 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:10.641+0000 7fa1ff7fe700 1 -- 172.21.15.178:0/3822714459 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa20c013f00 con 0x7fa2180f5410 2024-02-11T17:06:10.644 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:10.641+0000 7fa1ff7fe700 1 -- 172.21.15.178:0/3822714459 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa20c014d90 con 0x7fa2180f5410 2024-02-11T17:06:10.644 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:10.641+0000 7fa21d833700 1 -- 172.21.15.178:0/3822714459 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa218109ae0 con 0x7fa2180f5410 2024-02-11T17:06:10.645 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:10.642+0000 7fa1ff7fe700 1 -- 172.21.15.178:0/3822714459 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa20c014420 con 0x7fa2180f5410 2024-02-11T17:06:10.645 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:10.642+0000 7fa1ff7fe700 1 --2- 172.21.15.178:0/3822714459 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fa20002ff60 0x7fa200032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:06:10.645 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:10.642+0000 7fa21d833700 1 -- 172.21.15.178:0/3822714459 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa218045ad0 con 0x7fa2180f5410 2024-02-11T17:06:10.645 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:10.642+0000 7fa1ff7fe700 1 -- 172.21.15.178:0/3822714459 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa20c041020 con 0x7fa2180f5410 2024-02-11T17:06:10.645 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:10.642+0000 7fa2167fc700 1 --2- 172.21.15.178:0/3822714459 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fa20002ff60 0x7fa200032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:06:10.645 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:10.643+0000 7fa2167fc700 1 --2- 172.21.15.178:0/3822714459 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fa20002ff60 0x7fa200032420 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fa2080096a0 tx=0x7fa208006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:06:10.649 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:10.647+0000 7fa1ff7fe700 1 -- 172.21.15.178:0/3822714459 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa20c0095f0 con 0x7fa2180f5410 2024-02-11T17:06:10.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:06:10 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:06:09.184764+0000 mgr.smithi037.eofnaq (mgr.14162) 189 : cluster [DBG] pgmap v177: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:06:10.851 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:10.850+0000 7fa21d833700 1 -- 172.21.15.178:0/3822714459 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa2180faf50 con 0x7fa2180f5410 2024-02-11T17:06:10.852 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:10.850+0000 7fa1ff7fe700 1 -- 172.21.15.178:0/3822714459 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa20c02b020 con 0x7fa2180f5410 2024-02-11T17:06:10.852 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:06:10.852 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:06:10.854 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:10.852+0000 7fa21d833700 1 -- 172.21.15.178:0/3822714459 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fa20002ff60 msgr2=0x7fa200032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:06:10.854 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:10.852+0000 7fa21d833700 1 --2- 172.21.15.178:0/3822714459 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fa20002ff60 0x7fa200032420 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fa2080096a0 tx=0x7fa208006b40).stop 2024-02-11T17:06:10.854 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:10.852+0000 7fa21d833700 1 -- 172.21.15.178:0/3822714459 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa2180f5410 msgr2=0x7fa218109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:06:10.854 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:10.852+0000 7fa21d833700 1 --2- 172.21.15.178:0/3822714459 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa2180f5410 0x7fa218109430 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7fa20c013510 tx=0x7fa20c00b040).stop 2024-02-11T17:06:10.854 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:10.852+0000 7fa21d833700 1 -- 172.21.15.178:0/3822714459 shutdown_connections 2024-02-11T17:06:10.854 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:10.852+0000 7fa21d833700 1 --2- 172.21.15.178:0/3822714459 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fa20002ff60 0x7fa200032420 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:06:10.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:10.852+0000 7fa21d833700 1 --2- 172.21.15.178:0/3822714459 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa2180f5410 0x7fa218109430 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:06:10.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:10.852+0000 7fa21d833700 1 -- 172.21.15.178:0/3822714459 >> 172.21.15.178:0/3822714459 conn(0x7fa2180f0b70 msgr2=0x7fa2180f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:06:10.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:10.852+0000 7fa21d833700 1 -- 172.21.15.178:0/3822714459 shutdown_connections 2024-02-11T17:06:10.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:10.852+0000 7fa21d833700 1 -- 172.21.15.178:0/3822714459 wait complete. 2024-02-11T17:06:10.855 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:06:11.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:06:11 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:06:10.851840+0000 mon.smithi037 (mon.0) 226 : audit [DBG] from='client.? 172.21.15.178:0/3822714459' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:06:12.653 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:06:12.653 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:06:12.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:06:12 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:06:11.185120+0000 mgr.smithi037.eofnaq (mgr.14162) 190 : cluster [DBG] pgmap v178: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:06:14.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:06:14 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:06:13.185487+0000 mgr.smithi037.eofnaq (mgr.14162) 191 : cluster [DBG] pgmap v179: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:06:14.765 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:14.762+0000 7fa34e83c700 1 -- 172.21.15.178:0/2310190898 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa3480f5410 msgr2=0x7fa3480f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:06:14.765 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:14.762+0000 7fa34e83c700 1 --2- 172.21.15.178:0/2310190898 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa3480f5410 0x7fa3480f5830 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7fa338009fb0 tx=0x7fa338009b30).stop 2024-02-11T17:06:14.765 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:14.762+0000 7fa34e83c700 1 -- 172.21.15.178:0/2310190898 shutdown_connections 2024-02-11T17:06:14.765 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:14.763+0000 7fa34e83c700 1 --2- 172.21.15.178:0/2310190898 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa3480f5410 0x7fa3480f5830 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:06:14.765 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:14.763+0000 7fa34e83c700 1 -- 172.21.15.178:0/2310190898 >> 172.21.15.178:0/2310190898 conn(0x7fa3480f0b70 msgr2=0x7fa3480f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:06:14.765 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:14.763+0000 7fa34e83c700 1 -- 172.21.15.178:0/2310190898 shutdown_connections 2024-02-11T17:06:14.765 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:14.763+0000 7fa34e83c700 1 -- 172.21.15.178:0/2310190898 wait complete. 2024-02-11T17:06:14.765 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:14.763+0000 7fa34e83c700 1 Processor -- start 2024-02-11T17:06:14.765 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:14.763+0000 7fa34e83c700 1 -- start start 2024-02-11T17:06:14.766 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:14.764+0000 7fa34e83c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa3480f5410 0x7fa348108920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:06:14.767 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:14.764+0000 7fa34e83c700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa348108e10 con 0x7fa3480f5410 2024-02-11T17:06:14.767 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:14.764+0000 7fa347fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa3480f5410 0x7fa348108920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:06:14.767 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:14.764+0000 7fa347fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa3480f5410 0x7fa348108920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:50246/0 (socket says 172.21.15.178:50246) 2024-02-11T17:06:14.767 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:14.764+0000 7fa347fff700 1 -- 172.21.15.178:0/3180945796 learned_addr learned my addr 172.21.15.178:0/3180945796 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:06:14.767 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:14.764+0000 7fa347fff700 1 -- 172.21.15.178:0/3180945796 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa338005040 con 0x7fa3480f5410 2024-02-11T17:06:14.767 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:14.764+0000 7fa347fff700 1 --2- 172.21.15.178:0/3180945796 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa3480f5410 0x7fa348108920 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7fa338013510 tx=0x7fa33800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:06:14.767 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:14.765+0000 7fa344ff9700 1 -- 172.21.15.178:0/3180945796 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa338009320 con 0x7fa3480f5410 2024-02-11T17:06:14.767 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:14.765+0000 7fa34e83c700 1 -- 172.21.15.178:0/3180945796 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa348109010 con 0x7fa3480f5410 2024-02-11T17:06:14.767 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:14.765+0000 7fa344ff9700 1 -- 172.21.15.178:0/3180945796 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa338013f00 con 0x7fa3480f5410 2024-02-11T17:06:14.767 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:14.765+0000 7fa344ff9700 1 -- 172.21.15.178:0/3180945796 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa338014d90 con 0x7fa3480f5410 2024-02-11T17:06:14.767 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:14.765+0000 7fa34e83c700 1 -- 172.21.15.178:0/3180945796 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3481094a0 con 0x7fa3480f5410 2024-02-11T17:06:14.768 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:14.765+0000 7fa344ff9700 1 -- 172.21.15.178:0/3180945796 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa3380263a0 con 0x7fa3480f5410 2024-02-11T17:06:14.768 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:14.765+0000 7fa34e83c700 1 -- 172.21.15.178:0/3180945796 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa348045ad0 con 0x7fa3480f5410 2024-02-11T17:06:14.768 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:14.766+0000 7fa344ff9700 1 --2- 172.21.15.178:0/3180945796 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fa33402ff60 0x7fa334032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:06:14.768 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:14.766+0000 7fa344ff9700 1 -- 172.21.15.178:0/3180945796 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa3380412e0 con 0x7fa3480f5410 2024-02-11T17:06:14.768 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:14.766+0000 7fa3477fe700 1 --2- 172.21.15.178:0/3180945796 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fa33402ff60 0x7fa334032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:06:14.769 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:14.766+0000 7fa3477fe700 1 --2- 172.21.15.178:0/3180945796 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fa33402ff60 0x7fa334032420 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fa3300096a0 tx=0x7fa330006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:06:14.772 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:14.770+0000 7fa344ff9700 1 -- 172.21.15.178:0/3180945796 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa33801cb10 con 0x7fa3480f5410 2024-02-11T17:06:14.980 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:14.978+0000 7fa34e83c700 1 -- 172.21.15.178:0/3180945796 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa34810a350 con 0x7fa3480f5410 2024-02-11T17:06:14.980 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:14.979+0000 7fa344ff9700 1 -- 172.21.15.178:0/3180945796 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa33801cb10 con 0x7fa3480f5410 2024-02-11T17:06:14.980 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:06:14.981 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:06:14.982 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:14.980+0000 7fa34e83c700 1 -- 172.21.15.178:0/3180945796 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fa33402ff60 msgr2=0x7fa334032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:06:14.982 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:14.980+0000 7fa34e83c700 1 --2- 172.21.15.178:0/3180945796 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fa33402ff60 0x7fa334032420 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fa3300096a0 tx=0x7fa330006b40).stop 2024-02-11T17:06:14.982 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:14.980+0000 7fa34e83c700 1 -- 172.21.15.178:0/3180945796 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa3480f5410 msgr2=0x7fa348108920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:06:14.982 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:14.980+0000 7fa34e83c700 1 --2- 172.21.15.178:0/3180945796 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa3480f5410 0x7fa348108920 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7fa338013510 tx=0x7fa33800b040).stop 2024-02-11T17:06:14.982 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:14.980+0000 7fa34e83c700 1 -- 172.21.15.178:0/3180945796 shutdown_connections 2024-02-11T17:06:14.982 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:14.980+0000 7fa34e83c700 1 --2- 172.21.15.178:0/3180945796 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fa33402ff60 0x7fa334032420 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:06:14.982 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:14.980+0000 7fa34e83c700 1 --2- 172.21.15.178:0/3180945796 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa3480f5410 0x7fa348108920 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:06:14.983 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:14.980+0000 7fa34e83c700 1 -- 172.21.15.178:0/3180945796 >> 172.21.15.178:0/3180945796 conn(0x7fa3480f0b70 msgr2=0x7fa34805fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:06:14.983 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:14.980+0000 7fa34e83c700 1 -- 172.21.15.178:0/3180945796 shutdown_connections 2024-02-11T17:06:14.983 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:14.980+0000 7fa34e83c700 1 -- 172.21.15.178:0/3180945796 wait complete. 2024-02-11T17:06:14.983 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:06:15.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:06:15 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:06:14.979971+0000 mon.smithi037 (mon.0) 227 : audit [DBG] from='client.? 172.21.15.178:0/3180945796' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:06:16.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:06:16 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:06:15.185831+0000 mgr.smithi037.eofnaq (mgr.14162) 192 : cluster [DBG] pgmap v180: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:06:16.760 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:06:16.761 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:06:18.587 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:18.576+0000 7f3de9481700 1 -- 172.21.15.178:0/1018732562 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3de40f5410 msgr2=0x7f3de40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:06:18.587 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:18.576+0000 7f3de9481700 1 --2- 172.21.15.178:0/1018732562 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3de40f5410 0x7f3de40f5830 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7f3dd4009fb0 tx=0x7f3dd4009b30).stop 2024-02-11T17:06:18.587 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:18.576+0000 7f3de9481700 1 -- 172.21.15.178:0/1018732562 shutdown_connections 2024-02-11T17:06:18.588 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:18.576+0000 7f3de9481700 1 --2- 172.21.15.178:0/1018732562 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3de40f5410 0x7f3de40f5830 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:06:18.588 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:18.576+0000 7f3de9481700 1 -- 172.21.15.178:0/1018732562 >> 172.21.15.178:0/1018732562 conn(0x7f3de40f0b70 msgr2=0x7f3de40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:06:18.588 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:18.576+0000 7f3de9481700 1 -- 172.21.15.178:0/1018732562 shutdown_connections 2024-02-11T17:06:18.588 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:18.576+0000 7f3de9481700 1 -- 172.21.15.178:0/1018732562 wait complete. 2024-02-11T17:06:18.588 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:18.577+0000 7f3de9481700 1 Processor -- start 2024-02-11T17:06:18.588 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:18.577+0000 7f3de9481700 1 -- start start 2024-02-11T17:06:18.588 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:18.577+0000 7f3de9481700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3de40f5410 0x7f3de41055d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:06:18.588 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:18.577+0000 7f3de9481700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3de4105ac0 con 0x7f3de40f5410 2024-02-11T17:06:18.588 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:18.577+0000 7f3de2ffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3de40f5410 0x7f3de41055d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:06:18.588 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:18.577+0000 7f3de2ffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3de40f5410 0x7f3de41055d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:50262/0 (socket says 172.21.15.178:50262) 2024-02-11T17:06:18.588 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:18.578+0000 7f3de2ffd700 1 -- 172.21.15.178:0/2115220596 learned_addr learned my addr 172.21.15.178:0/2115220596 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:06:18.588 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:18.578+0000 7f3de2ffd700 1 -- 172.21.15.178:0/2115220596 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dd4005040 con 0x7f3de40f5410 2024-02-11T17:06:18.588 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:18.578+0000 7f3de2ffd700 1 --2- 172.21.15.178:0/2115220596 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3de40f5410 0x7f3de41055d0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f3dd4013510 tx=0x7f3dd400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:06:18.589 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:18.578+0000 7f3dcb7fe700 1 -- 172.21.15.178:0/2115220596 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3dd4009320 con 0x7f3de40f5410 2024-02-11T17:06:18.589 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:18.578+0000 7f3de9481700 1 -- 172.21.15.178:0/2115220596 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3de4105cc0 con 0x7f3de40f5410 2024-02-11T17:06:18.589 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:18.578+0000 7f3dcb7fe700 1 -- 172.21.15.178:0/2115220596 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3dd4013f00 con 0x7f3de40f5410 2024-02-11T17:06:18.589 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:18.578+0000 7f3dcb7fe700 1 -- 172.21.15.178:0/2115220596 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3dd4014d90 con 0x7f3de40f5410 2024-02-11T17:06:18.589 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:18.578+0000 7f3de9481700 1 -- 172.21.15.178:0/2115220596 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3de4106150 con 0x7f3de40f5410 2024-02-11T17:06:18.589 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:18.579+0000 7f3dcb7fe700 1 -- 172.21.15.178:0/2115220596 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3dd40095f0 con 0x7f3de40f5410 2024-02-11T17:06:18.589 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:18.579+0000 7f3de9481700 1 -- 172.21.15.178:0/2115220596 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3de4045ad0 con 0x7f3de40f5410 2024-02-11T17:06:18.589 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:18.579+0000 7f3dcb7fe700 1 --2- 172.21.15.178:0/2115220596 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f3dcc02ff60 0x7f3dcc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:06:18.589 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:18.579+0000 7f3dcb7fe700 1 -- 172.21.15.178:0/2115220596 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3dd4040fe0 con 0x7f3de40f5410 2024-02-11T17:06:18.589 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:18.580+0000 7f3de27fc700 1 --2- 172.21.15.178:0/2115220596 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f3dcc02ff60 0x7f3dcc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:06:18.589 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:18.580+0000 7f3de27fc700 1 --2- 172.21.15.178:0/2115220596 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f3dcc02ff60 0x7f3dcc032420 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f3dd80096a0 tx=0x7f3dd8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:06:18.589 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:18.584+0000 7f3dcb7fe700 1 -- 172.21.15.178:0/2115220596 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3dd401d7f0 con 0x7f3de40f5410 2024-02-11T17:06:18.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:06:18 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:06:17.186200+0000 mgr.smithi037.eofnaq (mgr.14162) 193 : cluster [DBG] pgmap v181: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:06:18.790 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:18.788+0000 7f3de9481700 1 -- 172.21.15.178:0/2115220596 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3de4107000 con 0x7f3de40f5410 2024-02-11T17:06:18.791 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:18.789+0000 7f3dcb7fe700 1 -- 172.21.15.178:0/2115220596 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3dd4010230 con 0x7f3de40f5410 2024-02-11T17:06:18.791 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:06:18.791 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:06:18.793 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:18.791+0000 7f3de9481700 1 -- 172.21.15.178:0/2115220596 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f3dcc02ff60 msgr2=0x7f3dcc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:06:18.793 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:18.791+0000 7f3de9481700 1 --2- 172.21.15.178:0/2115220596 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f3dcc02ff60 0x7f3dcc032420 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f3dd80096a0 tx=0x7f3dd8006b40).stop 2024-02-11T17:06:18.793 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:18.791+0000 7f3de9481700 1 -- 172.21.15.178:0/2115220596 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3de40f5410 msgr2=0x7f3de41055d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:06:18.793 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:18.791+0000 7f3de9481700 1 --2- 172.21.15.178:0/2115220596 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3de40f5410 0x7f3de41055d0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f3dd4013510 tx=0x7f3dd400b040).stop 2024-02-11T17:06:18.793 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:18.791+0000 7f3de9481700 1 -- 172.21.15.178:0/2115220596 shutdown_connections 2024-02-11T17:06:18.793 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:18.791+0000 7f3de9481700 1 --2- 172.21.15.178:0/2115220596 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f3dcc02ff60 0x7f3dcc032420 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:06:18.793 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:18.791+0000 7f3de9481700 1 --2- 172.21.15.178:0/2115220596 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3de40f5410 0x7f3de41055d0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:06:18.793 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:18.791+0000 7f3de9481700 1 -- 172.21.15.178:0/2115220596 >> 172.21.15.178:0/2115220596 conn(0x7f3de40f0b70 msgr2=0x7f3de40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:06:18.793 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:18.791+0000 7f3de9481700 1 -- 172.21.15.178:0/2115220596 shutdown_connections 2024-02-11T17:06:18.793 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:18.791+0000 7f3de9481700 1 -- 172.21.15.178:0/2115220596 wait complete. 2024-02-11T17:06:18.794 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:06:19.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:06:19 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:06:18.790447+0000 mon.smithi037 (mon.0) 228 : audit [DBG] from='client.? 172.21.15.178:0/2115220596' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:06:20.573 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:06:20.573 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:06:20.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:06:20 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:06:19.186525+0000 mgr.smithi037.eofnaq (mgr.14162) 194 : cluster [DBG] pgmap v182: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:06:22.642 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:22.639+0000 7f16be317700 1 -- 172.21.15.178:0/2418976389 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f16b80f51a0 msgr2=0x7f16b80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:06:22.642 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:22.639+0000 7f16be317700 1 --2- 172.21.15.178:0/2418976389 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f16b80f51a0 0x7f16b80f55c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f16a8009fb0 tx=0x7f16a8009b30).stop 2024-02-11T17:06:22.642 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:22.639+0000 7f16be317700 1 -- 172.21.15.178:0/2418976389 shutdown_connections 2024-02-11T17:06:22.642 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:22.639+0000 7f16be317700 1 --2- 172.21.15.178:0/2418976389 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f16b80f51a0 0x7f16b80f55c0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:06:22.642 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:22.639+0000 7f16be317700 1 -- 172.21.15.178:0/2418976389 >> 172.21.15.178:0/2418976389 conn(0x7f16b80f09a0 msgr2=0x7f16b80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:06:22.642 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:22.639+0000 7f16be317700 1 -- 172.21.15.178:0/2418976389 shutdown_connections 2024-02-11T17:06:22.642 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:22.639+0000 7f16be317700 1 -- 172.21.15.178:0/2418976389 wait complete. 2024-02-11T17:06:22.642 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:22.640+0000 7f16be317700 1 Processor -- start 2024-02-11T17:06:22.642 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:22.640+0000 7f16be317700 1 -- start start 2024-02-11T17:06:22.642 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:22.640+0000 7f16be317700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f16b80f51a0 0x7f16b81091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:06:22.643 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:22.640+0000 7f16be317700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16b8060220 con 0x7f16b80f51a0 2024-02-11T17:06:22.644 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:22.641+0000 7f16bd315700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f16b80f51a0 0x7f16b81091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:06:22.644 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:22.641+0000 7f16bd315700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f16b80f51a0 0x7f16b81091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:56190/0 (socket says 172.21.15.178:56190) 2024-02-11T17:06:22.644 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:22.641+0000 7f16bd315700 1 -- 172.21.15.178:0/2607149352 learned_addr learned my addr 172.21.15.178:0/2607149352 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:06:22.644 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:22.641+0000 7f16bd315700 1 -- 172.21.15.178:0/2607149352 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16a8005040 con 0x7f16b80f51a0 2024-02-11T17:06:22.644 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:22.641+0000 7f16bd315700 1 --2- 172.21.15.178:0/2607149352 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f16b80f51a0 0x7f16b81091d0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f16a8013510 tx=0x7f16a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:06:22.644 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:22.641+0000 7f16adffb700 1 -- 172.21.15.178:0/2607149352 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f16a8009320 con 0x7f16b80f51a0 2024-02-11T17:06:22.645 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:22.641+0000 7f16adffb700 1 -- 172.21.15.178:0/2607149352 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f16a8013f00 con 0x7f16b80f51a0 2024-02-11T17:06:22.645 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:22.641+0000 7f16be317700 1 -- 172.21.15.178:0/2607149352 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16b8060420 con 0x7f16b80f51a0 2024-02-11T17:06:22.645 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:22.641+0000 7f16adffb700 1 -- 172.21.15.178:0/2607149352 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f16a8014dc0 con 0x7f16b80f51a0 2024-02-11T17:06:22.645 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:22.641+0000 7f16be317700 1 -- 172.21.15.178:0/2607149352 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16b8109880 con 0x7f16b80f51a0 2024-02-11T17:06:22.645 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:22.642+0000 7f16adffb700 1 -- 172.21.15.178:0/2607149352 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f16a8014420 con 0x7f16b80f51a0 2024-02-11T17:06:22.645 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:22.642+0000 7f16be317700 1 -- 172.21.15.178:0/2607149352 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16b8059100 con 0x7f16b80f51a0 2024-02-11T17:06:22.645 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:22.642+0000 7f16adffb700 1 --2- 172.21.15.178:0/2607149352 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f16a402ff60 0x7f16a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:06:22.645 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:22.642+0000 7f16adffb700 1 -- 172.21.15.178:0/2607149352 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f16a80410f0 con 0x7f16b80f51a0 2024-02-11T17:06:22.645 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:22.643+0000 7f16bcb14700 1 --2- 172.21.15.178:0/2607149352 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f16a402ff60 0x7f16a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:06:22.645 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:22.643+0000 7f16bcb14700 1 --2- 172.21.15.178:0/2607149352 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f16a402ff60 0x7f16a4032420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f16b40096a0 tx=0x7f16b4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:06:22.649 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:22.647+0000 7f16adffb700 1 -- 172.21.15.178:0/2607149352 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f16a80095f0 con 0x7f16b80f51a0 2024-02-11T17:06:22.684 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:06:22 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:06:21.186853+0000 mgr.smithi037.eofnaq (mgr.14162) 195 : cluster [DBG] pgmap v183: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:06:22.853 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:22.852+0000 7f16be317700 1 -- 172.21.15.178:0/2607149352 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f16b8045ad0 con 0x7f16b80f51a0 2024-02-11T17:06:22.854 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:22.852+0000 7f16adffb700 1 -- 172.21.15.178:0/2607149352 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f16a801cb10 con 0x7f16b80f51a0 2024-02-11T17:06:22.854 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:06:22.854 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:06:22.856 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:22.854+0000 7f16be317700 1 -- 172.21.15.178:0/2607149352 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f16a402ff60 msgr2=0x7f16a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:06:22.856 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:22.854+0000 7f16be317700 1 --2- 172.21.15.178:0/2607149352 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f16a402ff60 0x7f16a4032420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f16b40096a0 tx=0x7f16b4006b40).stop 2024-02-11T17:06:22.856 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:22.854+0000 7f16be317700 1 -- 172.21.15.178:0/2607149352 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f16b80f51a0 msgr2=0x7f16b81091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:06:22.856 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:22.854+0000 7f16be317700 1 --2- 172.21.15.178:0/2607149352 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f16b80f51a0 0x7f16b81091d0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f16a8013510 tx=0x7f16a800b040).stop 2024-02-11T17:06:22.856 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:22.854+0000 7f16be317700 1 -- 172.21.15.178:0/2607149352 shutdown_connections 2024-02-11T17:06:22.856 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:22.854+0000 7f16be317700 1 --2- 172.21.15.178:0/2607149352 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f16a402ff60 0x7f16a4032420 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:06:22.856 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:22.854+0000 7f16be317700 1 --2- 172.21.15.178:0/2607149352 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f16b80f51a0 0x7f16b81091d0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:06:22.857 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:22.854+0000 7f16be317700 1 -- 172.21.15.178:0/2607149352 >> 172.21.15.178:0/2607149352 conn(0x7f16b80f09a0 msgr2=0x7f16b80f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:06:22.857 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:22.854+0000 7f16be317700 1 -- 172.21.15.178:0/2607149352 shutdown_connections 2024-02-11T17:06:22.857 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:22.854+0000 7f16be317700 1 -- 172.21.15.178:0/2607149352 wait complete. 2024-02-11T17:06:22.857 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:06:23.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:06:23 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:06:22.853816+0000 mon.smithi037 (mon.0) 229 : audit [DBG] from='client.? 172.21.15.178:0/2607149352' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:06:24.518 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:06:24.519 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:06:24.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:06:24 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:06:23.187144+0000 mgr.smithi037.eofnaq (mgr.14162) 196 : cluster [DBG] pgmap v184: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:06:26.387 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:26.382+0000 7f982388c700 1 -- 172.21.15.178:0/3445671304 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f981c0f5410 msgr2=0x7f981c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:06:26.387 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:26.382+0000 7f982388c700 1 --2- 172.21.15.178:0/3445671304 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f981c0f5410 0x7f981c0f5830 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f980c009fb0 tx=0x7f980c009b30).stop 2024-02-11T17:06:26.388 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:26.382+0000 7f982388c700 1 -- 172.21.15.178:0/3445671304 shutdown_connections 2024-02-11T17:06:26.388 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:26.382+0000 7f982388c700 1 --2- 172.21.15.178:0/3445671304 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f981c0f5410 0x7f981c0f5830 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:06:26.388 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:26.382+0000 7f982388c700 1 -- 172.21.15.178:0/3445671304 >> 172.21.15.178:0/3445671304 conn(0x7f981c0f0b70 msgr2=0x7f981c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:06:26.388 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:26.382+0000 7f982388c700 1 -- 172.21.15.178:0/3445671304 shutdown_connections 2024-02-11T17:06:26.388 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:26.382+0000 7f982388c700 1 -- 172.21.15.178:0/3445671304 wait complete. 2024-02-11T17:06:26.388 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:26.383+0000 7f982388c700 1 Processor -- start 2024-02-11T17:06:26.388 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:26.383+0000 7f982388c700 1 -- start start 2024-02-11T17:06:26.389 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:26.383+0000 7f982388c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f981c0f5410 0x7f981c109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:06:26.389 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:26.383+0000 7f982388c700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f981c060240 con 0x7f981c0f5410 2024-02-11T17:06:26.389 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:26.383+0000 7f9821628700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f981c0f5410 0x7f981c109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:06:26.389 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:26.383+0000 7f9821628700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f981c0f5410 0x7f981c109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:56204/0 (socket says 172.21.15.178:56204) 2024-02-11T17:06:26.389 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:26.383+0000 7f9821628700 1 -- 172.21.15.178:0/1701360476 learned_addr learned my addr 172.21.15.178:0/1701360476 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:06:26.389 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:26.383+0000 7f9821628700 1 -- 172.21.15.178:0/1701360476 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f980c005040 con 0x7f981c0f5410 2024-02-11T17:06:26.389 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:26.384+0000 7f9821628700 1 --2- 172.21.15.178:0/1701360476 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f981c0f5410 0x7f981c109430 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f980c013510 tx=0x7f980c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:06:26.389 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:26.384+0000 7f9811ffb700 1 -- 172.21.15.178:0/1701360476 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f980c009320 con 0x7f981c0f5410 2024-02-11T17:06:26.389 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:26.384+0000 7f982388c700 1 -- 172.21.15.178:0/1701360476 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f981c060440 con 0x7f981c0f5410 2024-02-11T17:06:26.389 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:26.384+0000 7f9811ffb700 1 -- 172.21.15.178:0/1701360476 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f980c013f00 con 0x7f981c0f5410 2024-02-11T17:06:26.389 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:26.384+0000 7f9811ffb700 1 -- 172.21.15.178:0/1701360476 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f980c014d90 con 0x7f981c0f5410 2024-02-11T17:06:26.389 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:26.384+0000 7f982388c700 1 -- 172.21.15.178:0/1701360476 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f981c109ae0 con 0x7f981c0f5410 2024-02-11T17:06:26.389 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:26.385+0000 7f9811ffb700 1 -- 172.21.15.178:0/1701360476 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f980c0263a0 con 0x7f981c0f5410 2024-02-11T17:06:26.390 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:26.385+0000 7f9811ffb700 1 --2- 172.21.15.178:0/1701360476 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f980802ff60 0x7f9808032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:06:26.390 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:26.385+0000 7f9811ffb700 1 -- 172.21.15.178:0/1701360476 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f980c0410b0 con 0x7f981c0f5410 2024-02-11T17:06:26.390 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:26.385+0000 7f982388c700 1 -- 172.21.15.178:0/1701360476 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f981c045ad0 con 0x7f981c0f5410 2024-02-11T17:06:26.390 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:26.385+0000 7f9820e27700 1 --2- 172.21.15.178:0/1701360476 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f980802ff60 0x7f9808032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:06:26.390 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:26.386+0000 7f9820e27700 1 --2- 172.21.15.178:0/1701360476 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f980802ff60 0x7f9808032420 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f98180096a0 tx=0x7f9818006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:06:26.391 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:26.390+0000 7f9811ffb700 1 -- 172.21.15.178:0/1701360476 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f980c012920 con 0x7f981c0f5410 2024-02-11T17:06:26.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:06:26 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:06:25.187478+0000 mgr.smithi037.eofnaq (mgr.14162) 197 : cluster [DBG] pgmap v185: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:06:26.595 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:26.593+0000 7f982388c700 1 -- 172.21.15.178:0/1701360476 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f981c10a1f0 con 0x7f981c0f5410 2024-02-11T17:06:26.596 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:26.594+0000 7f9811ffb700 1 -- 172.21.15.178:0/1701360476 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f980c02b020 con 0x7f981c0f5410 2024-02-11T17:06:26.596 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:06:26.596 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:06:26.597 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:26.596+0000 7f982388c700 1 -- 172.21.15.178:0/1701360476 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f980802ff60 msgr2=0x7f9808032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:06:26.597 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:26.596+0000 7f982388c700 1 --2- 172.21.15.178:0/1701360476 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f980802ff60 0x7f9808032420 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f98180096a0 tx=0x7f9818006b40).stop 2024-02-11T17:06:26.598 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:26.596+0000 7f982388c700 1 -- 172.21.15.178:0/1701360476 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f981c0f5410 msgr2=0x7f981c109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:06:26.598 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:26.596+0000 7f982388c700 1 --2- 172.21.15.178:0/1701360476 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f981c0f5410 0x7f981c109430 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f980c013510 tx=0x7f980c00b040).stop 2024-02-11T17:06:26.598 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:26.596+0000 7f982388c700 1 -- 172.21.15.178:0/1701360476 shutdown_connections 2024-02-11T17:06:26.598 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:26.596+0000 7f982388c700 1 --2- 172.21.15.178:0/1701360476 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f980802ff60 0x7f9808032420 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:06:26.598 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:26.596+0000 7f982388c700 1 --2- 172.21.15.178:0/1701360476 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f981c0f5410 0x7f981c109430 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:06:26.598 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:26.596+0000 7f982388c700 1 -- 172.21.15.178:0/1701360476 >> 172.21.15.178:0/1701360476 conn(0x7f981c0f0b70 msgr2=0x7f981c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:06:26.598 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:26.596+0000 7f982388c700 1 -- 172.21.15.178:0/1701360476 shutdown_connections 2024-02-11T17:06:26.598 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:26.596+0000 7f982388c700 1 -- 172.21.15.178:0/1701360476 wait complete. 2024-02-11T17:06:26.598 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:06:27.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:06:27 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:06:26.595336+0000 mon.smithi037 (mon.0) 230 : audit [DBG] from='client.? 172.21.15.178:0/1701360476' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:06:28.394 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:06:28.395 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:06:28.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:06:28 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:06:27.187862+0000 mgr.smithi037.eofnaq (mgr.14162) 198 : cluster [DBG] pgmap v186: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:06:30.301 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:30.296+0000 7f63a6baa700 1 -- 172.21.15.178:0/140042649 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f63a00f5410 msgr2=0x7f63a00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:06:30.302 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:30.296+0000 7f63a6baa700 1 --2- 172.21.15.178:0/140042649 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f63a00f5410 0x7f63a00f5830 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f6394009fb0 tx=0x7f6394009b30).stop 2024-02-11T17:06:30.302 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:30.296+0000 7f63a6baa700 1 -- 172.21.15.178:0/140042649 shutdown_connections 2024-02-11T17:06:30.302 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:30.296+0000 7f63a6baa700 1 --2- 172.21.15.178:0/140042649 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f63a00f5410 0x7f63a00f5830 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:06:30.302 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:30.296+0000 7f63a6baa700 1 -- 172.21.15.178:0/140042649 >> 172.21.15.178:0/140042649 conn(0x7f63a00f0b70 msgr2=0x7f63a00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:06:30.302 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:30.296+0000 7f63a6baa700 1 -- 172.21.15.178:0/140042649 shutdown_connections 2024-02-11T17:06:30.302 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:30.296+0000 7f63a6baa700 1 -- 172.21.15.178:0/140042649 wait complete. 2024-02-11T17:06:30.302 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:30.296+0000 7f63a6baa700 1 Processor -- start 2024-02-11T17:06:30.303 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:30.296+0000 7f63a6baa700 1 -- start start 2024-02-11T17:06:30.303 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:30.297+0000 7f63a6baa700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f63a00f5410 0x7f63a0109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:06:30.303 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:30.297+0000 7f63a6baa700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63a0060240 con 0x7f63a00f5410 2024-02-11T17:06:30.303 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:30.297+0000 7f63a4946700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f63a00f5410 0x7f63a0109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:06:30.303 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:30.297+0000 7f63a4946700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f63a00f5410 0x7f63a0109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:36416/0 (socket says 172.21.15.178:36416) 2024-02-11T17:06:30.303 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:30.297+0000 7f63a4946700 1 -- 172.21.15.178:0/2353259319 learned_addr learned my addr 172.21.15.178:0/2353259319 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:06:30.303 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:30.297+0000 7f63a4946700 1 -- 172.21.15.178:0/2353259319 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6394005040 con 0x7f63a00f5410 2024-02-11T17:06:30.304 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:30.297+0000 7f63a4946700 1 --2- 172.21.15.178:0/2353259319 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f63a00f5410 0x7f63a0109430 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f6394013510 tx=0x7f639400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:06:30.304 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:30.297+0000 7f639d7fa700 1 -- 172.21.15.178:0/2353259319 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6394009320 con 0x7f63a00f5410 2024-02-11T17:06:30.304 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:30.297+0000 7f63a6baa700 1 -- 172.21.15.178:0/2353259319 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63a0060440 con 0x7f63a00f5410 2024-02-11T17:06:30.304 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:30.297+0000 7f639d7fa700 1 -- 172.21.15.178:0/2353259319 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6394013f00 con 0x7f63a00f5410 2024-02-11T17:06:30.304 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:30.298+0000 7f639d7fa700 1 -- 172.21.15.178:0/2353259319 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6394014d90 con 0x7f63a00f5410 2024-02-11T17:06:30.304 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:30.298+0000 7f63a6baa700 1 -- 172.21.15.178:0/2353259319 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63a0109ae0 con 0x7f63a00f5410 2024-02-11T17:06:30.305 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:30.298+0000 7f639d7fa700 1 -- 172.21.15.178:0/2353259319 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f63940263a0 con 0x7f63a00f5410 2024-02-11T17:06:30.305 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:30.298+0000 7f639d7fa700 1 --2- 172.21.15.178:0/2353259319 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f639002ff60 0x7f6390032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:06:30.305 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:30.299+0000 7f639d7fa700 1 -- 172.21.15.178:0/2353259319 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6394041100 con 0x7f63a00f5410 2024-02-11T17:06:30.305 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:30.299+0000 7f63a6baa700 1 -- 172.21.15.178:0/2353259319 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63a0045ad0 con 0x7f63a00f5410 2024-02-11T17:06:30.305 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:30.299+0000 7f639ffff700 1 --2- 172.21.15.178:0/2353259319 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f639002ff60 0x7f6390032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:06:30.305 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:30.299+0000 7f639ffff700 1 --2- 172.21.15.178:0/2353259319 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f639002ff60 0x7f6390032420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f638c0096a0 tx=0x7f638c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:06:30.306 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:30.304+0000 7f639d7fa700 1 -- 172.21.15.178:0/2353259319 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6394012920 con 0x7f63a00f5410 2024-02-11T17:06:30.510 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:30.509+0000 7f63a6baa700 1 -- 172.21.15.178:0/2353259319 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f63a010a1f0 con 0x7f63a00f5410 2024-02-11T17:06:30.511 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:30.509+0000 7f639d7fa700 1 -- 172.21.15.178:0/2353259319 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f639402b020 con 0x7f63a00f5410 2024-02-11T17:06:30.511 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:06:30.511 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:06:30.513 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:30.511+0000 7f63a6baa700 1 -- 172.21.15.178:0/2353259319 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f639002ff60 msgr2=0x7f6390032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:06:30.513 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:30.511+0000 7f63a6baa700 1 --2- 172.21.15.178:0/2353259319 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f639002ff60 0x7f6390032420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f638c0096a0 tx=0x7f638c006b40).stop 2024-02-11T17:06:30.513 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:30.511+0000 7f63a6baa700 1 -- 172.21.15.178:0/2353259319 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f63a00f5410 msgr2=0x7f63a0109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:06:30.513 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:30.511+0000 7f63a6baa700 1 --2- 172.21.15.178:0/2353259319 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f63a00f5410 0x7f63a0109430 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f6394013510 tx=0x7f639400b040).stop 2024-02-11T17:06:30.513 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:30.511+0000 7f63a6baa700 1 -- 172.21.15.178:0/2353259319 shutdown_connections 2024-02-11T17:06:30.514 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:30.511+0000 7f63a6baa700 1 --2- 172.21.15.178:0/2353259319 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f639002ff60 0x7f6390032420 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:06:30.514 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:30.511+0000 7f63a6baa700 1 --2- 172.21.15.178:0/2353259319 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f63a00f5410 0x7f63a0109430 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:06:30.514 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:30.511+0000 7f63a6baa700 1 -- 172.21.15.178:0/2353259319 >> 172.21.15.178:0/2353259319 conn(0x7f63a00f0b70 msgr2=0x7f63a00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:06:30.514 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:30.511+0000 7f63a6baa700 1 -- 172.21.15.178:0/2353259319 shutdown_connections 2024-02-11T17:06:30.514 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:30.511+0000 7f63a6baa700 1 -- 172.21.15.178:0/2353259319 wait complete. 2024-02-11T17:06:30.515 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:06:30.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:06:30 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:06:29.188203+0000 mgr.smithi037.eofnaq (mgr.14162) 199 : cluster [DBG] pgmap v187: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:06:31.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:06:31 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:06:30.510658+0000 mon.smithi037 (mon.0) 231 : audit [DBG] from='client.? 172.21.15.178:0/2353259319' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:06:32.234 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:06:32.234 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:06:32.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:06:32 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:06:31.188575+0000 mgr.smithi037.eofnaq (mgr.14162) 200 : cluster [DBG] pgmap v188: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:06:34.129 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:34.124+0000 7fb03f17a700 1 -- 172.21.15.178:0/4090084883 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb0380f6510 msgr2=0x7fb0380f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:06:34.129 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:34.124+0000 7fb03f17a700 1 --2- 172.21.15.178:0/4090084883 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb0380f6510 0x7fb0380f6930 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7fb02c009fb0 tx=0x7fb02c009b30).stop 2024-02-11T17:06:34.129 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:34.124+0000 7fb03f17a700 1 -- 172.21.15.178:0/4090084883 shutdown_connections 2024-02-11T17:06:34.129 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:34.124+0000 7fb03f17a700 1 --2- 172.21.15.178:0/4090084883 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb0380f6510 0x7fb0380f6930 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:06:34.130 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:34.124+0000 7fb03f17a700 1 -- 172.21.15.178:0/4090084883 >> 172.21.15.178:0/4090084883 conn(0x7fb0380f1c70 msgr2=0x7fb0380f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:06:34.130 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:34.124+0000 7fb03f17a700 1 -- 172.21.15.178:0/4090084883 shutdown_connections 2024-02-11T17:06:34.130 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:34.124+0000 7fb03f17a700 1 -- 172.21.15.178:0/4090084883 wait complete. 2024-02-11T17:06:34.130 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:34.125+0000 7fb03f17a700 1 Processor -- start 2024-02-11T17:06:34.130 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:34.125+0000 7fb03f17a700 1 -- start start 2024-02-11T17:06:34.130 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:34.125+0000 7fb03f17a700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb0380f6510 0x7fb03806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:06:34.130 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:34.125+0000 7fb03f17a700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb038069e20 con 0x7fb0380f6510 2024-02-11T17:06:34.130 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:34.125+0000 7fb03cf16700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb0380f6510 0x7fb03806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:06:34.130 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:34.125+0000 7fb03cf16700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb0380f6510 0x7fb03806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:36438/0 (socket says 172.21.15.178:36438) 2024-02-11T17:06:34.130 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:34.125+0000 7fb03cf16700 1 -- 172.21.15.178:0/3248264950 learned_addr learned my addr 172.21.15.178:0/3248264950 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:06:34.130 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:34.125+0000 7fb03cf16700 1 -- 172.21.15.178:0/3248264950 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb02c005040 con 0x7fb0380f6510 2024-02-11T17:06:34.130 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:34.126+0000 7fb03cf16700 1 --2- 172.21.15.178:0/3248264950 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb0380f6510 0x7fb03806b9a0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7fb02c013510 tx=0x7fb02c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:06:34.130 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:34.126+0000 7fb0357fa700 1 -- 172.21.15.178:0/3248264950 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb02c009320 con 0x7fb0380f6510 2024-02-11T17:06:34.130 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:34.126+0000 7fb03f17a700 1 -- 172.21.15.178:0/3248264950 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb038069fc0 con 0x7fb0380f6510 2024-02-11T17:06:34.131 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:34.126+0000 7fb0357fa700 1 -- 172.21.15.178:0/3248264950 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb02c013f00 con 0x7fb0380f6510 2024-02-11T17:06:34.131 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:34.126+0000 7fb0357fa700 1 -- 172.21.15.178:0/3248264950 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb02c014d90 con 0x7fb0380f6510 2024-02-11T17:06:34.131 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:34.126+0000 7fb03f17a700 1 -- 172.21.15.178:0/3248264950 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb03806a450 con 0x7fb0380f6510 2024-02-11T17:06:34.131 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:34.127+0000 7fb0357fa700 1 -- 172.21.15.178:0/3248264950 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb02c0263a0 con 0x7fb0380f6510 2024-02-11T17:06:34.131 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:34.127+0000 7fb03f17a700 1 -- 172.21.15.178:0/3248264950 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb038059100 con 0x7fb0380f6510 2024-02-11T17:06:34.131 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:34.127+0000 7fb0357fa700 1 --2- 172.21.15.178:0/3248264950 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb02802ff60 0x7fb028032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:06:34.131 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:34.127+0000 7fb0357fa700 1 -- 172.21.15.178:0/3248264950 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb02c0413b0 con 0x7fb0380f6510 2024-02-11T17:06:34.131 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:34.127+0000 7fb037fff700 1 --2- 172.21.15.178:0/3248264950 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb02802ff60 0x7fb028032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:06:34.131 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:34.128+0000 7fb037fff700 1 --2- 172.21.15.178:0/3248264950 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb02802ff60 0x7fb028032420 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fb0240096a0 tx=0x7fb024006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:06:34.133 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:34.132+0000 7fb0357fa700 1 -- 172.21.15.178:0/3248264950 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb02c012510 con 0x7fb0380f6510 2024-02-11T17:06:34.337 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:34.336+0000 7fb03f17a700 1 -- 172.21.15.178:0/3248264950 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb038045ad0 con 0x7fb0380f6510 2024-02-11T17:06:34.338 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:34.336+0000 7fb0357fa700 1 -- 172.21.15.178:0/3248264950 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb02c00f340 con 0x7fb0380f6510 2024-02-11T17:06:34.338 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:06:34.338 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:06:34.340 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:34.338+0000 7fb03f17a700 1 -- 172.21.15.178:0/3248264950 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb02802ff60 msgr2=0x7fb028032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:06:34.340 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:34.338+0000 7fb03f17a700 1 --2- 172.21.15.178:0/3248264950 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb02802ff60 0x7fb028032420 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fb0240096a0 tx=0x7fb024006b40).stop 2024-02-11T17:06:34.340 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:34.338+0000 7fb03f17a700 1 -- 172.21.15.178:0/3248264950 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb0380f6510 msgr2=0x7fb03806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:06:34.340 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:34.338+0000 7fb03f17a700 1 --2- 172.21.15.178:0/3248264950 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb0380f6510 0x7fb03806b9a0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7fb02c013510 tx=0x7fb02c00b040).stop 2024-02-11T17:06:34.340 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:34.338+0000 7fb03f17a700 1 -- 172.21.15.178:0/3248264950 shutdown_connections 2024-02-11T17:06:34.340 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:34.338+0000 7fb03f17a700 1 --2- 172.21.15.178:0/3248264950 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb02802ff60 0x7fb028032420 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:06:34.340 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:34.338+0000 7fb03f17a700 1 --2- 172.21.15.178:0/3248264950 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb0380f6510 0x7fb03806b9a0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:06:34.340 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:34.338+0000 7fb03f17a700 1 -- 172.21.15.178:0/3248264950 >> 172.21.15.178:0/3248264950 conn(0x7fb0380f1c70 msgr2=0x7fb0380f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:06:34.340 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:34.338+0000 7fb03f17a700 1 -- 172.21.15.178:0/3248264950 shutdown_connections 2024-02-11T17:06:34.340 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:34.338+0000 7fb03f17a700 1 -- 172.21.15.178:0/3248264950 wait complete. 2024-02-11T17:06:34.341 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:06:34.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:06:34 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:06:33.188967+0000 mgr.smithi037.eofnaq (mgr.14162) 201 : cluster [DBG] pgmap v189: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:06:35.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:06:35 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:06:34.337830+0000 mon.smithi037 (mon.0) 232 : audit [DBG] from='client.? 172.21.15.178:0/3248264950' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:06:36.074 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:06:36.074 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:06:36.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:06:36 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:06:35.189363+0000 mgr.smithi037.eofnaq (mgr.14162) 202 : cluster [DBG] pgmap v190: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:06:37.971 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:37.968+0000 7f9ffffff700 1 -- 172.21.15.178:0/1313102858 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa0000f51a0 msgr2=0x7fa0000f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:06:37.971 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:37.968+0000 7f9ffffff700 1 --2- 172.21.15.178:0/1313102858 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa0000f51a0 0x7fa0000f55c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7f9ff0009fb0 tx=0x7f9ff0009b30).stop 2024-02-11T17:06:37.971 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:37.968+0000 7f9ffffff700 1 -- 172.21.15.178:0/1313102858 shutdown_connections 2024-02-11T17:06:37.971 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:37.968+0000 7f9ffffff700 1 --2- 172.21.15.178:0/1313102858 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa0000f51a0 0x7fa0000f55c0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:06:37.971 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:37.968+0000 7f9ffffff700 1 -- 172.21.15.178:0/1313102858 >> 172.21.15.178:0/1313102858 conn(0x7fa0000f09a0 msgr2=0x7fa0000f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:06:37.971 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:37.968+0000 7f9ffffff700 1 -- 172.21.15.178:0/1313102858 shutdown_connections 2024-02-11T17:06:37.971 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:37.968+0000 7f9ffffff700 1 -- 172.21.15.178:0/1313102858 wait complete. 2024-02-11T17:06:37.971 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:37.968+0000 7f9ffffff700 1 Processor -- start 2024-02-11T17:06:37.971 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:37.968+0000 7f9ffffff700 1 -- start start 2024-02-11T17:06:37.971 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:37.969+0000 7f9ffffff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa0000f51a0 0x7fa0001053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:06:37.971 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:37.969+0000 7f9ffffff700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0001058a0 con 0x7fa0000f51a0 2024-02-11T17:06:37.972 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:37.969+0000 7f9ffeffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa0000f51a0 0x7fa0001053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:06:37.972 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:37.969+0000 7f9ffeffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa0000f51a0 0x7fa0001053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:36450/0 (socket says 172.21.15.178:36450) 2024-02-11T17:06:37.972 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:37.969+0000 7f9ffeffd700 1 -- 172.21.15.178:0/1647144783 learned_addr learned my addr 172.21.15.178:0/1647144783 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:06:37.972 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:37.969+0000 7f9ffeffd700 1 -- 172.21.15.178:0/1647144783 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ff0005040 con 0x7fa0000f51a0 2024-02-11T17:06:37.972 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:37.969+0000 7f9ffeffd700 1 --2- 172.21.15.178:0/1647144783 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa0000f51a0 0x7fa0001053b0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f9ff0009830 tx=0x7f9ff000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:06:37.973 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:37.969+0000 7f9fe77fe700 1 -- 172.21.15.178:0/1647144783 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ff0009320 con 0x7fa0000f51a0 2024-02-11T17:06:37.973 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:37.969+0000 7f9fe77fe700 1 -- 172.21.15.178:0/1647144783 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9ff00136c0 con 0x7fa0000f51a0 2024-02-11T17:06:37.973 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:37.969+0000 7f9ffffff700 1 -- 172.21.15.178:0/1647144783 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa000105aa0 con 0x7fa0000f51a0 2024-02-11T17:06:37.973 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:37.969+0000 7f9fe77fe700 1 -- 172.21.15.178:0/1647144783 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ff0020e90 con 0x7fa0000f51a0 2024-02-11T17:06:37.973 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:37.969+0000 7f9ffffff700 1 -- 172.21.15.178:0/1647144783 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa000105f30 con 0x7fa0000f51a0 2024-02-11T17:06:37.973 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:37.970+0000 7f9fe77fe700 1 -- 172.21.15.178:0/1647144783 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9ff0014440 con 0x7fa0000f51a0 2024-02-11T17:06:37.974 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:37.970+0000 7f9ffffff700 1 -- 172.21.15.178:0/1647144783 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa000045ad0 con 0x7fa0000f51a0 2024-02-11T17:06:37.974 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:37.970+0000 7f9fe77fe700 1 --2- 172.21.15.178:0/1647144783 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f9fe802ff10 0x7f9fe80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:06:37.974 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:37.970+0000 7f9fe77fe700 1 -- 172.21.15.178:0/1647144783 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9ff0043100 con 0x7fa0000f51a0 2024-02-11T17:06:37.974 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:37.971+0000 7f9ffe7fc700 1 --2- 172.21.15.178:0/1647144783 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f9fe802ff10 0x7f9fe80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:06:37.974 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:37.971+0000 7f9ffe7fc700 1 --2- 172.21.15.178:0/1647144783 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f9fe802ff10 0x7f9fe80323d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f9ff40096a0 tx=0x7f9ff4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:06:37.977 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:37.975+0000 7f9fe77fe700 1 -- 172.21.15.178:0/1647144783 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9ff001ed60 con 0x7fa0000f51a0 2024-02-11T17:06:38.182 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:38.180+0000 7f9ffffff700 1 -- 172.21.15.178:0/1647144783 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa000106de0 con 0x7fa0000f51a0 2024-02-11T17:06:38.183 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:38.181+0000 7f9fe77fe700 1 -- 172.21.15.178:0/1647144783 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9ff002e2d0 con 0x7fa0000f51a0 2024-02-11T17:06:38.183 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:06:38.183 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:06:38.185 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:38.183+0000 7f9ffffff700 1 -- 172.21.15.178:0/1647144783 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f9fe802ff10 msgr2=0x7f9fe80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:06:38.185 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:38.183+0000 7f9ffffff700 1 --2- 172.21.15.178:0/1647144783 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f9fe802ff10 0x7f9fe80323d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f9ff40096a0 tx=0x7f9ff4006b40).stop 2024-02-11T17:06:38.185 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:38.183+0000 7f9ffffff700 1 -- 172.21.15.178:0/1647144783 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa0000f51a0 msgr2=0x7fa0001053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:06:38.185 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:38.183+0000 7f9ffffff700 1 --2- 172.21.15.178:0/1647144783 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa0000f51a0 0x7fa0001053b0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f9ff0009830 tx=0x7f9ff000b040).stop 2024-02-11T17:06:38.185 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:38.183+0000 7f9ffffff700 1 -- 172.21.15.178:0/1647144783 shutdown_connections 2024-02-11T17:06:38.185 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:38.183+0000 7f9ffffff700 1 --2- 172.21.15.178:0/1647144783 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f9fe802ff10 0x7f9fe80323d0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:06:38.185 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:38.183+0000 7f9ffffff700 1 --2- 172.21.15.178:0/1647144783 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa0000f51a0 0x7fa0001053b0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:06:38.185 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:38.183+0000 7f9ffffff700 1 -- 172.21.15.178:0/1647144783 >> 172.21.15.178:0/1647144783 conn(0x7fa0000f09a0 msgr2=0x7fa0000f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:06:38.185 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:38.183+0000 7f9ffffff700 1 -- 172.21.15.178:0/1647144783 shutdown_connections 2024-02-11T17:06:38.185 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:38.183+0000 7f9ffffff700 1 -- 172.21.15.178:0/1647144783 wait complete. 2024-02-11T17:06:38.186 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:06:38.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:06:38 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:06:37.189700+0000 mgr.smithi037.eofnaq (mgr.14162) 203 : cluster [DBG] pgmap v191: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:06:38.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:06:38 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:06:38.182648+0000 mon.smithi037 (mon.0) 233 : audit [DBG] from='client.? 172.21.15.178:0/1647144783' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:06:39.960 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:06:39.960 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:06:40.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:06:40 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:06:39.189978+0000 mgr.smithi037.eofnaq (mgr.14162) 204 : cluster [DBG] pgmap v192: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:06:41.811 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:41.808+0000 7fe75c937700 1 -- 172.21.15.178:0/3646013381 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7580f5410 msgr2=0x7fe7580f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:06:41.812 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:41.808+0000 7fe75c937700 1 --2- 172.21.15.178:0/3646013381 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7580f5410 0x7fe7580f5830 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7fe740009fb0 tx=0x7fe740009b30).stop 2024-02-11T17:06:41.812 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:41.808+0000 7fe75c937700 1 -- 172.21.15.178:0/3646013381 shutdown_connections 2024-02-11T17:06:41.812 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:41.808+0000 7fe75c937700 1 --2- 172.21.15.178:0/3646013381 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7580f5410 0x7fe7580f5830 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:06:41.812 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:41.808+0000 7fe75c937700 1 -- 172.21.15.178:0/3646013381 >> 172.21.15.178:0/3646013381 conn(0x7fe7580f0b70 msgr2=0x7fe7580f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:06:41.812 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:41.808+0000 7fe75c937700 1 -- 172.21.15.178:0/3646013381 shutdown_connections 2024-02-11T17:06:41.812 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:41.808+0000 7fe75c937700 1 -- 172.21.15.178:0/3646013381 wait complete. 2024-02-11T17:06:41.812 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:41.809+0000 7fe75c937700 1 Processor -- start 2024-02-11T17:06:41.812 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:41.809+0000 7fe75c937700 1 -- start start 2024-02-11T17:06:41.812 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:41.809+0000 7fe75c937700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7580f5410 0x7fe758105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:06:41.812 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:41.809+0000 7fe75c937700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe758105af0 con 0x7fe7580f5410 2024-02-11T17:06:41.813 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:41.809+0000 7fe75659c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7580f5410 0x7fe758105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:06:41.813 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:41.809+0000 7fe75659c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7580f5410 0x7fe758105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:59578/0 (socket says 172.21.15.178:59578) 2024-02-11T17:06:41.813 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:41.809+0000 7fe75659c700 1 -- 172.21.15.178:0/308867440 learned_addr learned my addr 172.21.15.178:0/308867440 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:06:41.813 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:41.809+0000 7fe75659c700 1 -- 172.21.15.178:0/308867440 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe740005040 con 0x7fe7580f5410 2024-02-11T17:06:41.813 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:41.810+0000 7fe75659c700 1 --2- 172.21.15.178:0/308867440 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7580f5410 0x7fe758105600 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7fe740013510 tx=0x7fe74000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:06:41.813 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:41.810+0000 7fe74effd700 1 -- 172.21.15.178:0/308867440 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe740009320 con 0x7fe7580f5410 2024-02-11T17:06:41.813 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:41.810+0000 7fe75c937700 1 -- 172.21.15.178:0/308867440 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe758105cf0 con 0x7fe7580f5410 2024-02-11T17:06:41.813 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:41.810+0000 7fe74effd700 1 -- 172.21.15.178:0/308867440 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe740013f00 con 0x7fe7580f5410 2024-02-11T17:06:41.813 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:41.810+0000 7fe74effd700 1 -- 172.21.15.178:0/308867440 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe740014d90 con 0x7fe7580f5410 2024-02-11T17:06:41.813 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:41.810+0000 7fe75c937700 1 -- 172.21.15.178:0/308867440 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe758106180 con 0x7fe7580f5410 2024-02-11T17:06:41.814 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:41.811+0000 7fe74effd700 1 -- 172.21.15.178:0/308867440 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe7400095f0 con 0x7fe7580f5410 2024-02-11T17:06:41.814 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:41.811+0000 7fe75c937700 1 -- 172.21.15.178:0/308867440 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe758045ad0 con 0x7fe7580f5410 2024-02-11T17:06:41.814 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:41.811+0000 7fe74effd700 1 --2- 172.21.15.178:0/308867440 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fe74402ffb0 0x7fe744032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:06:41.814 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:41.811+0000 7fe74effd700 1 -- 172.21.15.178:0/308867440 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe740042e00 con 0x7fe7580f5410 2024-02-11T17:06:41.814 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:41.811+0000 7fe755d9b700 1 --2- 172.21.15.178:0/308867440 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fe74402ffb0 0x7fe744032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:06:41.814 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:41.811+0000 7fe755d9b700 1 --2- 172.21.15.178:0/308867440 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fe74402ffb0 0x7fe744032470 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fe7480096a0 tx=0x7fe748006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:06:41.817 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:41.816+0000 7fe74effd700 1 -- 172.21.15.178:0/308867440 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe740014420 con 0x7fe7580f5410 2024-02-11T17:06:42.021 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:42.019+0000 7fe75c937700 1 -- 172.21.15.178:0/308867440 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe7580fae60 con 0x7fe7580f5410 2024-02-11T17:06:42.021 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:42.020+0000 7fe74effd700 1 -- 172.21.15.178:0/308867440 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe740011290 con 0x7fe7580f5410 2024-02-11T17:06:42.022 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:06:42.022 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:06:42.023 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:42.021+0000 7fe75c937700 1 -- 172.21.15.178:0/308867440 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fe74402ffb0 msgr2=0x7fe744032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:06:42.023 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:42.021+0000 7fe75c937700 1 --2- 172.21.15.178:0/308867440 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fe74402ffb0 0x7fe744032470 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fe7480096a0 tx=0x7fe748006b40).stop 2024-02-11T17:06:42.023 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:42.021+0000 7fe75c937700 1 -- 172.21.15.178:0/308867440 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7580f5410 msgr2=0x7fe758105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:06:42.024 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:42.021+0000 7fe75c937700 1 --2- 172.21.15.178:0/308867440 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7580f5410 0x7fe758105600 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7fe740013510 tx=0x7fe74000b040).stop 2024-02-11T17:06:42.024 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:42.022+0000 7fe75c937700 1 -- 172.21.15.178:0/308867440 shutdown_connections 2024-02-11T17:06:42.024 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:42.022+0000 7fe75c937700 1 --2- 172.21.15.178:0/308867440 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fe74402ffb0 0x7fe744032470 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:06:42.024 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:42.022+0000 7fe75c937700 1 --2- 172.21.15.178:0/308867440 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7580f5410 0x7fe758105600 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:06:42.024 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:42.022+0000 7fe75c937700 1 -- 172.21.15.178:0/308867440 >> 172.21.15.178:0/308867440 conn(0x7fe7580f0b70 msgr2=0x7fe7580f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:06:42.024 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:42.022+0000 7fe75c937700 1 -- 172.21.15.178:0/308867440 shutdown_connections 2024-02-11T17:06:42.024 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:42.022+0000 7fe75c937700 1 -- 172.21.15.178:0/308867440 wait complete. 2024-02-11T17:06:42.024 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:06:42.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:06:42 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:06:41.190326+0000 mgr.smithi037.eofnaq (mgr.14162) 205 : cluster [DBG] pgmap v193: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:06:42.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:06:42 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:06:42.021393+0000 mon.smithi037 (mon.0) 234 : audit [DBG] from='client.? 172.21.15.178:0/308867440' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:06:43.706 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:06:43.707 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:06:44.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:06:44 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:06:43.190638+0000 mgr.smithi037.eofnaq (mgr.14162) 206 : cluster [DBG] pgmap v194: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:06:45.818 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:45.816+0000 7f260d5f1700 1 -- 172.21.15.178:0/3533090319 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2608060280 msgr2=0x7f26080f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:06:45.818 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:45.816+0000 7f260d5f1700 1 --2- 172.21.15.178:0/3533090319 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2608060280 0x7f26080f90f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f25f8009fb0 tx=0x7f25f8009b30).stop 2024-02-11T17:06:45.818 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:45.816+0000 7f260d5f1700 1 -- 172.21.15.178:0/3533090319 shutdown_connections 2024-02-11T17:06:45.818 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:45.816+0000 7f260d5f1700 1 --2- 172.21.15.178:0/3533090319 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2608060280 0x7f26080f90f0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:06:45.818 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:45.816+0000 7f260d5f1700 1 -- 172.21.15.178:0/3533090319 >> 172.21.15.178:0/3533090319 conn(0x7f26080f0b70 msgr2=0x7f26080f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:06:45.818 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:45.816+0000 7f260d5f1700 1 -- 172.21.15.178:0/3533090319 shutdown_connections 2024-02-11T17:06:45.818 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:45.816+0000 7f260d5f1700 1 -- 172.21.15.178:0/3533090319 wait complete. 2024-02-11T17:06:45.821 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:45.816+0000 7f260d5f1700 1 Processor -- start 2024-02-11T17:06:45.821 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:45.817+0000 7f260d5f1700 1 -- start start 2024-02-11T17:06:45.821 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:45.817+0000 7f260d5f1700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2608060280 0x7f26081066c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:06:45.821 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:45.817+0000 7f260d5f1700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2608106bb0 con 0x7f2608060280 2024-02-11T17:06:45.821 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:45.817+0000 7f2606ffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2608060280 0x7f26081066c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:06:45.821 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:45.817+0000 7f2606ffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2608060280 0x7f26081066c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:59596/0 (socket says 172.21.15.178:59596) 2024-02-11T17:06:45.821 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:45.817+0000 7f2606ffd700 1 -- 172.21.15.178:0/2788541430 learned_addr learned my addr 172.21.15.178:0/2788541430 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:06:45.821 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:45.817+0000 7f2606ffd700 1 -- 172.21.15.178:0/2788541430 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25f8005040 con 0x7f2608060280 2024-02-11T17:06:45.821 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:45.817+0000 7f2606ffd700 1 --2- 172.21.15.178:0/2788541430 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2608060280 0x7f26081066c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f25f8013510 tx=0x7f25f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:06:45.822 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:45.818+0000 7f25ef7fe700 1 -- 172.21.15.178:0/2788541430 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f25f8009320 con 0x7f2608060280 2024-02-11T17:06:45.822 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:45.818+0000 7f260d5f1700 1 -- 172.21.15.178:0/2788541430 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2608106db0 con 0x7f2608060280 2024-02-11T17:06:45.822 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:45.818+0000 7f25ef7fe700 1 -- 172.21.15.178:0/2788541430 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f25f8013f00 con 0x7f2608060280 2024-02-11T17:06:45.822 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:45.818+0000 7f25ef7fe700 1 -- 172.21.15.178:0/2788541430 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f25f8014dc0 con 0x7f2608060280 2024-02-11T17:06:45.822 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:45.818+0000 7f260d5f1700 1 -- 172.21.15.178:0/2788541430 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2608107240 con 0x7f2608060280 2024-02-11T17:06:45.822 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:45.818+0000 7f25ef7fe700 1 -- 172.21.15.178:0/2788541430 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f25f801d3d0 con 0x7f2608060280 2024-02-11T17:06:45.822 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:45.818+0000 7f260d5f1700 1 -- 172.21.15.178:0/2788541430 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2608045ad0 con 0x7f2608060280 2024-02-11T17:06:45.822 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:45.818+0000 7f25ef7fe700 1 --2- 172.21.15.178:0/2788541430 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f25f002ff60 0x7f25f0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:06:45.822 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:45.819+0000 7f25ef7fe700 1 -- 172.21.15.178:0/2788541430 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f25f80413e0 con 0x7f2608060280 2024-02-11T17:06:45.822 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:45.819+0000 7f26067fc700 1 --2- 172.21.15.178:0/2788541430 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f25f002ff60 0x7f25f0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:06:45.822 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:45.819+0000 7f26067fc700 1 --2- 172.21.15.178:0/2788541430 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f25f002ff60 0x7f25f0032420 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f25fc0096a0 tx=0x7f25fc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:06:45.825 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:45.824+0000 7f25ef7fe700 1 -- 172.21.15.178:0/2788541430 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f25f8012510 con 0x7f2608060280 2024-02-11T17:06:46.032 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:46.031+0000 7f260d5f1700 1 -- 172.21.15.178:0/2788541430 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f26081080f0 con 0x7f2608060280 2024-02-11T17:06:46.033 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:46.031+0000 7f25ef7fe700 1 -- 172.21.15.178:0/2788541430 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f25f8012330 con 0x7f2608060280 2024-02-11T17:06:46.033 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:06:46.033 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:06:46.035 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:46.033+0000 7f260d5f1700 1 -- 172.21.15.178:0/2788541430 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f25f002ff60 msgr2=0x7f25f0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:06:46.035 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:46.033+0000 7f260d5f1700 1 --2- 172.21.15.178:0/2788541430 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f25f002ff60 0x7f25f0032420 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f25fc0096a0 tx=0x7f25fc006b40).stop 2024-02-11T17:06:46.035 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:46.033+0000 7f260d5f1700 1 -- 172.21.15.178:0/2788541430 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2608060280 msgr2=0x7f26081066c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:06:46.035 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:46.033+0000 7f260d5f1700 1 --2- 172.21.15.178:0/2788541430 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2608060280 0x7f26081066c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f25f8013510 tx=0x7f25f800b040).stop 2024-02-11T17:06:46.035 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:46.033+0000 7f260d5f1700 1 -- 172.21.15.178:0/2788541430 shutdown_connections 2024-02-11T17:06:46.035 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:46.033+0000 7f260d5f1700 1 --2- 172.21.15.178:0/2788541430 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f25f002ff60 0x7f25f0032420 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:06:46.035 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:46.033+0000 7f260d5f1700 1 --2- 172.21.15.178:0/2788541430 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2608060280 0x7f26081066c0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:06:46.035 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:46.033+0000 7f260d5f1700 1 -- 172.21.15.178:0/2788541430 >> 172.21.15.178:0/2788541430 conn(0x7f26080f0b70 msgr2=0x7f26080f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:06:46.035 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:46.033+0000 7f260d5f1700 1 -- 172.21.15.178:0/2788541430 shutdown_connections 2024-02-11T17:06:46.035 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:46.033+0000 7f260d5f1700 1 -- 172.21.15.178:0/2788541430 wait complete. 2024-02-11T17:06:46.036 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:06:46.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:06:46 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:06:45.190904+0000 mgr.smithi037.eofnaq (mgr.14162) 207 : cluster [DBG] pgmap v195: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:06:46.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:06:46 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:06:46.032774+0000 mon.smithi037 (mon.0) 235 : audit [DBG] from='client.? 172.21.15.178:0/2788541430' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:06:47.782 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:06:47.782 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:06:48.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:06:48 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:06:47.191264+0000 mgr.smithi037.eofnaq (mgr.14162) 208 : cluster [DBG] pgmap v196: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:06:49.753 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:49.739+0000 7f1060225700 1 -- 172.21.15.178:0/3230262397 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f10580f5410 msgr2=0x7f10580f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:06:49.753 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:49.739+0000 7f1060225700 1 --2- 172.21.15.178:0/3230262397 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f10580f5410 0x7f10580f5830 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7f1048009fb0 tx=0x7f1048009b30).stop 2024-02-11T17:06:49.753 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:49.739+0000 7f1060225700 1 -- 172.21.15.178:0/3230262397 shutdown_connections 2024-02-11T17:06:49.753 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:49.739+0000 7f1060225700 1 --2- 172.21.15.178:0/3230262397 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f10580f5410 0x7f10580f5830 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:06:49.753 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:49.739+0000 7f1060225700 1 -- 172.21.15.178:0/3230262397 >> 172.21.15.178:0/3230262397 conn(0x7f10580f0b70 msgr2=0x7f10580f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:06:49.754 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:49.739+0000 7f1060225700 1 -- 172.21.15.178:0/3230262397 shutdown_connections 2024-02-11T17:06:49.754 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:49.739+0000 7f1060225700 1 -- 172.21.15.178:0/3230262397 wait complete. 2024-02-11T17:06:49.754 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:49.740+0000 7f1060225700 1 Processor -- start 2024-02-11T17:06:49.754 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:49.740+0000 7f1060225700 1 -- start start 2024-02-11T17:06:49.754 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:49.740+0000 7f1060225700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f10580f5410 0x7f1058109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:06:49.754 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:49.740+0000 7f1060225700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1058060240 con 0x7f10580f5410 2024-02-11T17:06:49.754 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:49.740+0000 7f105dfc1700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f10580f5410 0x7f1058109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:06:49.754 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:49.740+0000 7f105dfc1700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f10580f5410 0x7f1058109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:56248/0 (socket says 172.21.15.178:56248) 2024-02-11T17:06:49.754 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:49.740+0000 7f105dfc1700 1 -- 172.21.15.178:0/4279654385 learned_addr learned my addr 172.21.15.178:0/4279654385 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:06:49.754 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:49.741+0000 7f105dfc1700 1 -- 172.21.15.178:0/4279654385 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1048005040 con 0x7f10580f5410 2024-02-11T17:06:49.754 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:49.741+0000 7f105dfc1700 1 --2- 172.21.15.178:0/4279654385 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f10580f5410 0x7f1058109430 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f1048014000 tx=0x7f104800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:06:49.754 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:49.741+0000 7f104e7fc700 1 -- 172.21.15.178:0/4279654385 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f10480095f0 con 0x7f10580f5410 2024-02-11T17:06:49.754 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:49.741+0000 7f1060225700 1 -- 172.21.15.178:0/4279654385 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1058060440 con 0x7f10580f5410 2024-02-11T17:06:49.754 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:49.741+0000 7f104e7fc700 1 -- 172.21.15.178:0/4279654385 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1048007e70 con 0x7f10580f5410 2024-02-11T17:06:49.755 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:49.741+0000 7f104e7fc700 1 -- 172.21.15.178:0/4279654385 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1048014ea0 con 0x7f10580f5410 2024-02-11T17:06:49.755 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:49.741+0000 7f1060225700 1 -- 172.21.15.178:0/4279654385 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1058109ae0 con 0x7f10580f5410 2024-02-11T17:06:49.755 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:49.742+0000 7f1060225700 1 -- 172.21.15.178:0/4279654385 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1058045ad0 con 0x7f10580f5410 2024-02-11T17:06:49.755 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:49.742+0000 7f104e7fc700 1 -- 172.21.15.178:0/4279654385 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1048025070 con 0x7f10580f5410 2024-02-11T17:06:49.755 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:49.742+0000 7f104e7fc700 1 --2- 172.21.15.178:0/4279654385 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f104402ff60 0x7f1044032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:06:49.755 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:49.742+0000 7f104e7fc700 1 -- 172.21.15.178:0/4279654385 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1048040940 con 0x7f10580f5410 2024-02-11T17:06:49.755 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:49.743+0000 7f105d7c0700 1 --2- 172.21.15.178:0/4279654385 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f104402ff60 0x7f1044032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:06:49.755 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:49.743+0000 7f105d7c0700 1 --2- 172.21.15.178:0/4279654385 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f104402ff60 0x7f1044032420 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f10540096a0 tx=0x7f1054006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:06:49.755 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:49.747+0000 7f104e7fc700 1 -- 172.21.15.178:0/4279654385 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f10480079c0 con 0x7f10580f5410 2024-02-11T17:06:49.954 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:49.953+0000 7f1060225700 1 -- 172.21.15.178:0/4279654385 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f10580fae60 con 0x7f10580f5410 2024-02-11T17:06:49.954 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:49.954+0000 7f104e7fc700 1 -- 172.21.15.178:0/4279654385 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f104800f300 con 0x7f10580f5410 2024-02-11T17:06:49.955 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:06:49.955 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:06:49.957 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:49.955+0000 7f1060225700 1 -- 172.21.15.178:0/4279654385 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f104402ff60 msgr2=0x7f1044032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:06:49.957 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:49.955+0000 7f1060225700 1 --2- 172.21.15.178:0/4279654385 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f104402ff60 0x7f1044032420 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f10540096a0 tx=0x7f1054006b40).stop 2024-02-11T17:06:49.957 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:49.955+0000 7f1060225700 1 -- 172.21.15.178:0/4279654385 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f10580f5410 msgr2=0x7f1058109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:06:49.957 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:49.955+0000 7f1060225700 1 --2- 172.21.15.178:0/4279654385 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f10580f5410 0x7f1058109430 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f1048014000 tx=0x7f104800b040).stop 2024-02-11T17:06:49.957 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:49.955+0000 7f1060225700 1 -- 172.21.15.178:0/4279654385 shutdown_connections 2024-02-11T17:06:49.957 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:49.955+0000 7f1060225700 1 --2- 172.21.15.178:0/4279654385 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f104402ff60 0x7f1044032420 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:06:49.957 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:49.956+0000 7f1060225700 1 --2- 172.21.15.178:0/4279654385 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f10580f5410 0x7f1058109430 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:06:49.957 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:49.956+0000 7f1060225700 1 -- 172.21.15.178:0/4279654385 >> 172.21.15.178:0/4279654385 conn(0x7f10580f0b70 msgr2=0x7f10580f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:06:49.957 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:49.956+0000 7f1060225700 1 -- 172.21.15.178:0/4279654385 shutdown_connections 2024-02-11T17:06:49.957 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:49.956+0000 7f1060225700 1 -- 172.21.15.178:0/4279654385 wait complete. 2024-02-11T17:06:49.957 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:06:50.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:06:50 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:06:49.191583+0000 mgr.smithi037.eofnaq (mgr.14162) 209 : cluster [DBG] pgmap v197: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:06:50.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:06:50 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:06:49.954219+0000 mon.smithi037 (mon.0) 236 : audit [DBG] from='client.? 172.21.15.178:0/4279654385' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:06:51.721 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:06:51.722 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:06:52.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:06:52 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:06:51.191846+0000 mgr.smithi037.eofnaq (mgr.14162) 210 : cluster [DBG] pgmap v198: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:06:53.645 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:53.643+0000 7f416eb0c700 1 -- 172.21.15.178:0/293253287 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41680f6510 msgr2=0x7f41680f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:06:53.645 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:53.643+0000 7f416eb0c700 1 --2- 172.21.15.178:0/293253287 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41680f6510 0x7f41680f6930 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f415c009fb0 tx=0x7f415c009b30).stop 2024-02-11T17:06:53.645 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:53.643+0000 7f416eb0c700 1 -- 172.21.15.178:0/293253287 shutdown_connections 2024-02-11T17:06:53.645 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:53.643+0000 7f416eb0c700 1 --2- 172.21.15.178:0/293253287 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41680f6510 0x7f41680f6930 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:06:53.645 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:53.643+0000 7f416eb0c700 1 -- 172.21.15.178:0/293253287 >> 172.21.15.178:0/293253287 conn(0x7f41680f1c70 msgr2=0x7f41680f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:06:53.645 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:53.643+0000 7f416eb0c700 1 -- 172.21.15.178:0/293253287 shutdown_connections 2024-02-11T17:06:53.646 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:53.644+0000 7f416eb0c700 1 -- 172.21.15.178:0/293253287 wait complete. 2024-02-11T17:06:53.646 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:53.644+0000 7f416eb0c700 1 Processor -- start 2024-02-11T17:06:53.646 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:53.644+0000 7f416eb0c700 1 -- start start 2024-02-11T17:06:53.646 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:53.644+0000 7f416eb0c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41680f6510 0x7f416806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:06:53.646 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:53.644+0000 7f416eb0c700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4168069e20 con 0x7f41680f6510 2024-02-11T17:06:53.648 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:53.645+0000 7f416c8a8700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41680f6510 0x7f416806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:06:53.648 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:53.645+0000 7f416c8a8700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41680f6510 0x7f416806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:56268/0 (socket says 172.21.15.178:56268) 2024-02-11T17:06:53.648 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:53.645+0000 7f416c8a8700 1 -- 172.21.15.178:0/1782224680 learned_addr learned my addr 172.21.15.178:0/1782224680 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:06:53.648 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:53.645+0000 7f416c8a8700 1 -- 172.21.15.178:0/1782224680 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f415c005040 con 0x7f41680f6510 2024-02-11T17:06:53.648 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:53.645+0000 7f416c8a8700 1 --2- 172.21.15.178:0/1782224680 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41680f6510 0x7f416806b9a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f415c013510 tx=0x7f415c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:06:53.648 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:53.645+0000 7f41657fa700 1 -- 172.21.15.178:0/1782224680 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f415c009320 con 0x7f41680f6510 2024-02-11T17:06:53.649 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:53.645+0000 7f41657fa700 1 -- 172.21.15.178:0/1782224680 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f415c013f00 con 0x7f41680f6510 2024-02-11T17:06:53.649 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:53.645+0000 7f416eb0c700 1 -- 172.21.15.178:0/1782224680 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4168069fc0 con 0x7f41680f6510 2024-02-11T17:06:53.649 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:53.645+0000 7f41657fa700 1 -- 172.21.15.178:0/1782224680 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f415c014d90 con 0x7f41680f6510 2024-02-11T17:06:53.649 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:53.645+0000 7f416eb0c700 1 -- 172.21.15.178:0/1782224680 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f416806a450 con 0x7f41680f6510 2024-02-11T17:06:53.649 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:53.646+0000 7f41657fa700 1 -- 172.21.15.178:0/1782224680 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f415c0095f0 con 0x7f41680f6510 2024-02-11T17:06:53.649 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:53.646+0000 7f416eb0c700 1 -- 172.21.15.178:0/1782224680 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4168059100 con 0x7f41680f6510 2024-02-11T17:06:53.649 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:53.646+0000 7f41657fa700 1 --2- 172.21.15.178:0/1782224680 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f415802ff60 0x7f4158032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:06:53.649 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:53.646+0000 7f41657fa700 1 -- 172.21.15.178:0/1782224680 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f415c040fe0 con 0x7f41680f6510 2024-02-11T17:06:53.649 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:53.647+0000 7f4167fff700 1 --2- 172.21.15.178:0/1782224680 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f415802ff60 0x7f4158032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:06:53.649 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:53.647+0000 7f4167fff700 1 --2- 172.21.15.178:0/1782224680 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f415802ff60 0x7f4158032420 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f41540096a0 tx=0x7f4154006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:06:53.651 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:53.651+0000 7f41657fa700 1 -- 172.21.15.178:0/1782224680 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f415c01d7f0 con 0x7f41680f6510 2024-02-11T17:06:53.859 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:53.858+0000 7f416eb0c700 1 -- 172.21.15.178:0/1782224680 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f41680fbf60 con 0x7f41680f6510 2024-02-11T17:06:53.859 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:53.859+0000 7f41657fa700 1 -- 172.21.15.178:0/1782224680 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f415c010230 con 0x7f41680f6510 2024-02-11T17:06:53.860 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:06:53.860 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:06:53.862 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:53.860+0000 7f416eb0c700 1 -- 172.21.15.178:0/1782224680 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f415802ff60 msgr2=0x7f4158032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:06:53.862 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:53.860+0000 7f416eb0c700 1 --2- 172.21.15.178:0/1782224680 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f415802ff60 0x7f4158032420 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f41540096a0 tx=0x7f4154006b40).stop 2024-02-11T17:06:53.862 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:53.861+0000 7f416eb0c700 1 -- 172.21.15.178:0/1782224680 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41680f6510 msgr2=0x7f416806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:06:53.862 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:53.861+0000 7f416eb0c700 1 --2- 172.21.15.178:0/1782224680 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41680f6510 0x7f416806b9a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f415c013510 tx=0x7f415c00b040).stop 2024-02-11T17:06:53.862 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:53.861+0000 7f416eb0c700 1 -- 172.21.15.178:0/1782224680 shutdown_connections 2024-02-11T17:06:53.862 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:53.861+0000 7f416eb0c700 1 --2- 172.21.15.178:0/1782224680 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f415802ff60 0x7f4158032420 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:06:53.862 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:53.861+0000 7f416eb0c700 1 --2- 172.21.15.178:0/1782224680 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41680f6510 0x7f416806b9a0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:06:53.862 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:53.861+0000 7f416eb0c700 1 -- 172.21.15.178:0/1782224680 >> 172.21.15.178:0/1782224680 conn(0x7f41680f1c70 msgr2=0x7f41680f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:06:53.862 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:53.861+0000 7f416eb0c700 1 -- 172.21.15.178:0/1782224680 shutdown_connections 2024-02-11T17:06:53.862 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:53.861+0000 7f416eb0c700 1 -- 172.21.15.178:0/1782224680 wait complete. 2024-02-11T17:06:53.862 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:06:54.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:06:54 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:06:53.192083+0000 mgr.smithi037.eofnaq (mgr.14162) 211 : cluster [DBG] pgmap v199: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:06:54.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:06:54 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:06:53.859101+0000 mon.smithi037 (mon.0) 237 : audit [DBG] from='client.? 172.21.15.178:0/1782224680' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:06:55.621 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:06:55.621 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:06:56.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:06:56 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:06:55.192431+0000 mgr.smithi037.eofnaq (mgr.14162) 212 : cluster [DBG] pgmap v200: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:06:57.485 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:57.482+0000 7fb8ef15c700 1 -- 172.21.15.178:0/1618021252 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb8e80f6510 msgr2=0x7fb8e80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:06:57.485 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:57.482+0000 7fb8ef15c700 1 --2- 172.21.15.178:0/1618021252 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb8e80f6510 0x7fb8e80f6930 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7fb8dc009fb0 tx=0x7fb8dc009b30).stop 2024-02-11T17:06:57.485 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:57.482+0000 7fb8ef15c700 1 -- 172.21.15.178:0/1618021252 shutdown_connections 2024-02-11T17:06:57.485 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:57.482+0000 7fb8ef15c700 1 --2- 172.21.15.178:0/1618021252 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb8e80f6510 0x7fb8e80f6930 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:06:57.485 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:57.482+0000 7fb8ef15c700 1 -- 172.21.15.178:0/1618021252 >> 172.21.15.178:0/1618021252 conn(0x7fb8e80f1c70 msgr2=0x7fb8e80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:06:57.485 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:57.482+0000 7fb8ef15c700 1 -- 172.21.15.178:0/1618021252 shutdown_connections 2024-02-11T17:06:57.485 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:57.482+0000 7fb8ef15c700 1 -- 172.21.15.178:0/1618021252 wait complete. 2024-02-11T17:06:57.485 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:57.483+0000 7fb8ef15c700 1 Processor -- start 2024-02-11T17:06:57.485 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:57.483+0000 7fb8ef15c700 1 -- start start 2024-02-11T17:06:57.486 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:57.483+0000 7fb8ef15c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb8e80f6510 0x7fb8e8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:06:57.486 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:57.483+0000 7fb8ef15c700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e8106c20 con 0x7fb8e80f6510 2024-02-11T17:06:57.487 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:57.483+0000 7fb8ecef8700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb8e80f6510 0x7fb8e8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:06:57.487 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:57.483+0000 7fb8ecef8700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb8e80f6510 0x7fb8e8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:56294/0 (socket says 172.21.15.178:56294) 2024-02-11T17:06:57.488 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:57.483+0000 7fb8ecef8700 1 -- 172.21.15.178:0/363043042 learned_addr learned my addr 172.21.15.178:0/363043042 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:06:57.488 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:57.484+0000 7fb8ecef8700 1 -- 172.21.15.178:0/363043042 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8dc005040 con 0x7fb8e80f6510 2024-02-11T17:06:57.488 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:57.484+0000 7fb8ecef8700 1 --2- 172.21.15.178:0/363043042 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb8e80f6510 0x7fb8e8106730 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7fb8dc013510 tx=0x7fb8dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:06:57.488 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:57.484+0000 7fb8e57fa700 1 -- 172.21.15.178:0/363043042 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8dc009320 con 0x7fb8e80f6510 2024-02-11T17:06:57.488 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:57.484+0000 7fb8ef15c700 1 -- 172.21.15.178:0/363043042 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8e8106e20 con 0x7fb8e80f6510 2024-02-11T17:06:57.488 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:57.484+0000 7fb8e57fa700 1 -- 172.21.15.178:0/363043042 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb8dc013f00 con 0x7fb8e80f6510 2024-02-11T17:06:57.488 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:57.484+0000 7fb8e57fa700 1 -- 172.21.15.178:0/363043042 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8dc014d90 con 0x7fb8e80f6510 2024-02-11T17:06:57.488 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:57.484+0000 7fb8ef15c700 1 -- 172.21.15.178:0/363043042 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8e81072b0 con 0x7fb8e80f6510 2024-02-11T17:06:57.488 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:57.485+0000 7fb8e57fa700 1 -- 172.21.15.178:0/363043042 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb8dc0095f0 con 0x7fb8e80f6510 2024-02-11T17:06:57.488 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:57.485+0000 7fb8ef15c700 1 -- 172.21.15.178:0/363043042 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8e8045ad0 con 0x7fb8e80f6510 2024-02-11T17:06:57.488 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:57.485+0000 7fb8e57fa700 1 --2- 172.21.15.178:0/363043042 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb8d802ff60 0x7fb8d8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:06:57.488 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:57.485+0000 7fb8e57fa700 1 -- 172.21.15.178:0/363043042 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb8dc040fc0 con 0x7fb8e80f6510 2024-02-11T17:06:57.488 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:57.485+0000 7fb8e7fff700 1 --2- 172.21.15.178:0/363043042 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb8d802ff60 0x7fb8d8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:06:57.489 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:57.486+0000 7fb8e7fff700 1 --2- 172.21.15.178:0/363043042 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb8d802ff60 0x7fb8d8032420 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fb8d40096a0 tx=0x7fb8d4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:06:57.491 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:57.490+0000 7fb8e57fa700 1 -- 172.21.15.178:0/363043042 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb8dc014420 con 0x7fb8e80f6510 2024-02-11T17:06:57.698 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:57.696+0000 7fb8ef15c700 1 -- 172.21.15.178:0/363043042 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb8e80fbf60 con 0x7fb8e80f6510 2024-02-11T17:06:57.698 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:57.697+0000 7fb8e57fa700 1 -- 172.21.15.178:0/363043042 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb8dc010230 con 0x7fb8e80f6510 2024-02-11T17:06:57.699 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:06:57.699 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:06:57.700 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:57.698+0000 7fb8ef15c700 1 -- 172.21.15.178:0/363043042 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb8d802ff60 msgr2=0x7fb8d8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:06:57.700 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:57.698+0000 7fb8ef15c700 1 --2- 172.21.15.178:0/363043042 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb8d802ff60 0x7fb8d8032420 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fb8d40096a0 tx=0x7fb8d4006b40).stop 2024-02-11T17:06:57.701 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:57.698+0000 7fb8ef15c700 1 -- 172.21.15.178:0/363043042 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb8e80f6510 msgr2=0x7fb8e8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:06:57.701 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:57.698+0000 7fb8ef15c700 1 --2- 172.21.15.178:0/363043042 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb8e80f6510 0x7fb8e8106730 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7fb8dc013510 tx=0x7fb8dc00b040).stop 2024-02-11T17:06:57.701 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:57.698+0000 7fb8ef15c700 1 -- 172.21.15.178:0/363043042 shutdown_connections 2024-02-11T17:06:57.701 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:57.698+0000 7fb8ef15c700 1 --2- 172.21.15.178:0/363043042 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb8d802ff60 0x7fb8d8032420 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:06:57.701 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:57.698+0000 7fb8ef15c700 1 --2- 172.21.15.178:0/363043042 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb8e80f6510 0x7fb8e8106730 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:06:57.701 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:57.698+0000 7fb8ef15c700 1 -- 172.21.15.178:0/363043042 >> 172.21.15.178:0/363043042 conn(0x7fb8e80f1c70 msgr2=0x7fb8e80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:06:57.701 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:57.699+0000 7fb8ef15c700 1 -- 172.21.15.178:0/363043042 shutdown_connections 2024-02-11T17:06:57.701 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:06:57.699+0000 7fb8ef15c700 1 -- 172.21.15.178:0/363043042 wait complete. 2024-02-11T17:06:57.701 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:06:58.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:06:58 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:06:57.192762+0000 mgr.smithi037.eofnaq (mgr.14162) 213 : cluster [DBG] pgmap v201: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:06:58.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:06:58 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:06:57.698137+0000 mon.smithi037 (mon.0) 238 : audit [DBG] from='client.? 172.21.15.178:0/363043042' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:06:59.535 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:06:59.535 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:07:00.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:00 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:06:59.193243+0000 mgr.smithi037.eofnaq (mgr.14162) 214 : cluster [DBG] pgmap v202: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:07:01.665 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:01.662+0000 7f6ed22a9700 1 -- 172.21.15.178:0/3118858110 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6ecc0f4800 msgr2=0x7f6ecc0f4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:07:01.665 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:01.662+0000 7f6ed22a9700 1 --2- 172.21.15.178:0/3118858110 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6ecc0f4800 0x7f6ecc0f4c20 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f6ec0009fb0 tx=0x7f6ec0009b30).stop 2024-02-11T17:07:01.665 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:01.663+0000 7f6ed22a9700 1 -- 172.21.15.178:0/3118858110 shutdown_connections 2024-02-11T17:07:01.666 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:01.663+0000 7f6ed22a9700 1 --2- 172.21.15.178:0/3118858110 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6ecc0f4800 0x7f6ecc0f4c20 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:07:01.666 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:01.663+0000 7f6ed22a9700 1 -- 172.21.15.178:0/3118858110 >> 172.21.15.178:0/3118858110 conn(0x7f6ecc0f0260 msgr2=0x7f6ecc0f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:07:01.666 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:01.663+0000 7f6ed22a9700 1 -- 172.21.15.178:0/3118858110 shutdown_connections 2024-02-11T17:07:01.666 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:01.663+0000 7f6ed22a9700 1 -- 172.21.15.178:0/3118858110 wait complete. 2024-02-11T17:07:01.666 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:01.663+0000 7f6ed22a9700 1 Processor -- start 2024-02-11T17:07:01.666 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:01.663+0000 7f6ed22a9700 1 -- start start 2024-02-11T17:07:01.666 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:01.664+0000 7f6ed22a9700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6ecc0f4800 0x7f6ecc1066b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:07:01.666 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:01.664+0000 7f6ed22a9700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ecc106ba0 con 0x7f6ecc0f4800 2024-02-11T17:07:01.668 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:01.664+0000 7f6ecbfff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6ecc0f4800 0x7f6ecc1066b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:07:01.668 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:01.664+0000 7f6ecbfff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6ecc0f4800 0x7f6ecc1066b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:50332/0 (socket says 172.21.15.178:50332) 2024-02-11T17:07:01.668 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:01.664+0000 7f6ecbfff700 1 -- 172.21.15.178:0/1174539720 learned_addr learned my addr 172.21.15.178:0/1174539720 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:07:01.668 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:01.664+0000 7f6ecbfff700 1 -- 172.21.15.178:0/1174539720 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ec0005040 con 0x7f6ecc0f4800 2024-02-11T17:07:01.669 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:01.665+0000 7f6ecbfff700 1 --2- 172.21.15.178:0/1174539720 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6ecc0f4800 0x7f6ecc1066b0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f6ec0013510 tx=0x7f6ec000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:07:01.669 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:01.665+0000 7f6ec8ff9700 1 -- 172.21.15.178:0/1174539720 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ec0009320 con 0x7f6ecc0f4800 2024-02-11T17:07:01.669 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:01.665+0000 7f6ec8ff9700 1 -- 172.21.15.178:0/1174539720 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6ec0013f00 con 0x7f6ecc0f4800 2024-02-11T17:07:01.669 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:01.665+0000 7f6ed22a9700 1 -- 172.21.15.178:0/1174539720 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ecc106da0 con 0x7f6ecc0f4800 2024-02-11T17:07:01.669 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:01.665+0000 7f6ec8ff9700 1 -- 172.21.15.178:0/1174539720 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ec0014dc0 con 0x7f6ecc0f4800 2024-02-11T17:07:01.669 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:01.665+0000 7f6ed22a9700 1 -- 172.21.15.178:0/1174539720 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ecc107230 con 0x7f6ecc0f4800 2024-02-11T17:07:01.669 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:01.666+0000 7f6ec8ff9700 1 -- 172.21.15.178:0/1174539720 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6ec00095f0 con 0x7f6ecc0f4800 2024-02-11T17:07:01.669 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:01.666+0000 7f6ed22a9700 1 -- 172.21.15.178:0/1174539720 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ecc045ad0 con 0x7f6ecc0f4800 2024-02-11T17:07:01.669 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:01.666+0000 7f6ec8ff9700 1 --2- 172.21.15.178:0/1174539720 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f6eb402ff60 0x7f6eb4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:07:01.669 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:01.666+0000 7f6ec8ff9700 1 -- 172.21.15.178:0/1174539720 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6ec0041000 con 0x7f6ecc0f4800 2024-02-11T17:07:01.669 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:01.666+0000 7f6ecb7fe700 1 --2- 172.21.15.178:0/1174539720 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f6eb402ff60 0x7f6eb4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:07:01.669 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:01.666+0000 7f6ecb7fe700 1 --2- 172.21.15.178:0/1174539720 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f6eb402ff60 0x7f6eb4032420 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f6ebc0096a0 tx=0x7f6ebc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:07:01.672 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:01.671+0000 7f6ec8ff9700 1 -- 172.21.15.178:0/1174539720 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6ec001c790 con 0x7f6ecc0f4800 2024-02-11T17:07:01.879 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:01.877+0000 7f6ed22a9700 1 -- 172.21.15.178:0/1174539720 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6ecc1080e0 con 0x7f6ecc0f4800 2024-02-11T17:07:01.880 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:01.878+0000 7f6ec8ff9700 1 -- 172.21.15.178:0/1174539720 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6ec0010230 con 0x7f6ecc0f4800 2024-02-11T17:07:01.880 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:07:01.880 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:07:01.882 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:01.880+0000 7f6ed22a9700 1 -- 172.21.15.178:0/1174539720 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f6eb402ff60 msgr2=0x7f6eb4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:07:01.882 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:01.880+0000 7f6ed22a9700 1 --2- 172.21.15.178:0/1174539720 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f6eb402ff60 0x7f6eb4032420 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f6ebc0096a0 tx=0x7f6ebc006b40).stop 2024-02-11T17:07:01.882 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:01.880+0000 7f6ed22a9700 1 -- 172.21.15.178:0/1174539720 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6ecc0f4800 msgr2=0x7f6ecc1066b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:07:01.882 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:01.880+0000 7f6ed22a9700 1 --2- 172.21.15.178:0/1174539720 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6ecc0f4800 0x7f6ecc1066b0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f6ec0013510 tx=0x7f6ec000b040).stop 2024-02-11T17:07:01.882 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:01.880+0000 7f6ed22a9700 1 -- 172.21.15.178:0/1174539720 shutdown_connections 2024-02-11T17:07:01.882 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:01.880+0000 7f6ed22a9700 1 --2- 172.21.15.178:0/1174539720 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f6eb402ff60 0x7f6eb4032420 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:07:01.882 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:01.880+0000 7f6ed22a9700 1 --2- 172.21.15.178:0/1174539720 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6ecc0f4800 0x7f6ecc1066b0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:07:01.882 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:01.880+0000 7f6ed22a9700 1 -- 172.21.15.178:0/1174539720 >> 172.21.15.178:0/1174539720 conn(0x7f6ecc0f0260 msgr2=0x7f6ecc0f0da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:07:01.882 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:01.880+0000 7f6ed22a9700 1 -- 172.21.15.178:0/1174539720 shutdown_connections 2024-02-11T17:07:01.882 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:01.880+0000 7f6ed22a9700 1 -- 172.21.15.178:0/1174539720 wait complete. 2024-02-11T17:07:01.883 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:07:02.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:02 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:07:01.193593+0000 mgr.smithi037.eofnaq (mgr.14162) 215 : cluster [DBG] pgmap v203: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:07:02.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:02 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:07:01.879590+0000 mon.smithi037 (mon.0) 239 : audit [DBG] from='client.? 172.21.15.178:0/1174539720' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:07:03.580 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:07:03.580 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:07:04.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:04 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:07:03.193932+0000 mgr.smithi037.eofnaq (mgr.14162) 216 : cluster [DBG] pgmap v204: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:07:04.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:04 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:07:03.277258+0000 mon.smithi037 (mon.0) 240 : audit [INF] from='mgr.14162 172.21.15.37:0/2745465012' entity='mgr.smithi037.eofnaq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi037.eofnaq/mirror_snapshot_schedule"}]: dispatch 2024-02-11T17:07:04.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:04 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:07:03.280174+0000 mon.smithi037 (mon.0) 241 : audit [INF] from='mgr.14162 172.21.15.37:0/2745465012' entity='mgr.smithi037.eofnaq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi037.eofnaq/trash_purge_schedule"}]: dispatch 2024-02-11T17:07:05.475 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:05.472+0000 7f7f31335700 1 -- 172.21.15.178:0/1806767704 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f2c0f5410 msgr2=0x7f7f2c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:07:05.475 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:05.472+0000 7f7f31335700 1 --2- 172.21.15.178:0/1806767704 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f2c0f5410 0x7f7f2c0f5830 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f7f20009fb0 tx=0x7f7f20009b30).stop 2024-02-11T17:07:05.476 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:05.472+0000 7f7f31335700 1 -- 172.21.15.178:0/1806767704 shutdown_connections 2024-02-11T17:07:05.476 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:05.472+0000 7f7f31335700 1 --2- 172.21.15.178:0/1806767704 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f2c0f5410 0x7f7f2c0f5830 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:07:05.476 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:05.472+0000 7f7f31335700 1 -- 172.21.15.178:0/1806767704 >> 172.21.15.178:0/1806767704 conn(0x7f7f2c0f0b70 msgr2=0x7f7f2c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:07:05.476 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:05.472+0000 7f7f31335700 1 -- 172.21.15.178:0/1806767704 shutdown_connections 2024-02-11T17:07:05.476 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:05.472+0000 7f7f31335700 1 -- 172.21.15.178:0/1806767704 wait complete. 2024-02-11T17:07:05.476 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:05.473+0000 7f7f31335700 1 Processor -- start 2024-02-11T17:07:05.476 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:05.473+0000 7f7f31335700 1 -- start start 2024-02-11T17:07:05.476 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:05.473+0000 7f7f31335700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f2c0f5410 0x7f7f2c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:07:05.476 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:05.473+0000 7f7f31335700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2c106c20 con 0x7f7f2c0f5410 2024-02-11T17:07:05.476 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:05.473+0000 7f7f2affd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f2c0f5410 0x7f7f2c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:07:05.476 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:05.473+0000 7f7f2affd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f2c0f5410 0x7f7f2c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:50356/0 (socket says 172.21.15.178:50356) 2024-02-11T17:07:05.476 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:05.473+0000 7f7f2affd700 1 -- 172.21.15.178:0/1146811922 learned_addr learned my addr 172.21.15.178:0/1146811922 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:07:05.477 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:05.473+0000 7f7f2affd700 1 -- 172.21.15.178:0/1146811922 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f20005040 con 0x7f7f2c0f5410 2024-02-11T17:07:05.477 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:05.473+0000 7f7f2affd700 1 --2- 172.21.15.178:0/1146811922 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f2c0f5410 0x7f7f2c106730 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f7f20013510 tx=0x7f7f2000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:07:05.477 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:05.474+0000 7f7f137fe700 1 -- 172.21.15.178:0/1146811922 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f20009320 con 0x7f7f2c0f5410 2024-02-11T17:07:05.477 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:05.474+0000 7f7f137fe700 1 -- 172.21.15.178:0/1146811922 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7f20013f00 con 0x7f7f2c0f5410 2024-02-11T17:07:05.477 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:05.474+0000 7f7f31335700 1 -- 172.21.15.178:0/1146811922 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f2c106e20 con 0x7f7f2c0f5410 2024-02-11T17:07:05.477 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:05.474+0000 7f7f137fe700 1 -- 172.21.15.178:0/1146811922 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f20014d90 con 0x7f7f2c0f5410 2024-02-11T17:07:05.477 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:05.474+0000 7f7f31335700 1 -- 172.21.15.178:0/1146811922 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f2c1072b0 con 0x7f7f2c0f5410 2024-02-11T17:07:05.478 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:05.474+0000 7f7f137fe700 1 -- 172.21.15.178:0/1146811922 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7f200095f0 con 0x7f7f2c0f5410 2024-02-11T17:07:05.478 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:05.474+0000 7f7f31335700 1 -- 172.21.15.178:0/1146811922 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f2c045ad0 con 0x7f7f2c0f5410 2024-02-11T17:07:05.478 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:05.475+0000 7f7f137fe700 1 --2- 172.21.15.178:0/1146811922 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f7f1402ff60 0x7f7f14032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:07:05.478 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:05.475+0000 7f7f137fe700 1 -- 172.21.15.178:0/1146811922 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7f20041000 con 0x7f7f2c0f5410 2024-02-11T17:07:05.478 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:05.475+0000 7f7f2a7fc700 1 --2- 172.21.15.178:0/1146811922 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f7f1402ff60 0x7f7f14032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:07:05.478 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:05.475+0000 7f7f2a7fc700 1 --2- 172.21.15.178:0/1146811922 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f7f1402ff60 0x7f7f14032420 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f7f1c0096a0 tx=0x7f7f1c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:07:05.481 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:05.479+0000 7f7f137fe700 1 -- 172.21.15.178:0/1146811922 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7f20014420 con 0x7f7f2c0f5410 2024-02-11T17:07:05.685 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:05.683+0000 7f7f31335700 1 -- 172.21.15.178:0/1146811922 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7f2c0fae60 con 0x7f7f2c0f5410 2024-02-11T17:07:05.686 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:05.684+0000 7f7f137fe700 1 -- 172.21.15.178:0/1146811922 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7f2001d7f0 con 0x7f7f2c0f5410 2024-02-11T17:07:05.686 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:07:05.686 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:07:05.687 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:05.686+0000 7f7f31335700 1 -- 172.21.15.178:0/1146811922 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f7f1402ff60 msgr2=0x7f7f14032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:07:05.688 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:05.686+0000 7f7f31335700 1 --2- 172.21.15.178:0/1146811922 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f7f1402ff60 0x7f7f14032420 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f7f1c0096a0 tx=0x7f7f1c006b40).stop 2024-02-11T17:07:05.688 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:05.686+0000 7f7f31335700 1 -- 172.21.15.178:0/1146811922 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f2c0f5410 msgr2=0x7f7f2c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:07:05.688 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:05.686+0000 7f7f31335700 1 --2- 172.21.15.178:0/1146811922 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f2c0f5410 0x7f7f2c106730 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f7f20013510 tx=0x7f7f2000b040).stop 2024-02-11T17:07:05.688 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:05.686+0000 7f7f31335700 1 -- 172.21.15.178:0/1146811922 shutdown_connections 2024-02-11T17:07:05.688 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:05.686+0000 7f7f31335700 1 --2- 172.21.15.178:0/1146811922 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f7f1402ff60 0x7f7f14032420 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:07:05.688 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:05.686+0000 7f7f31335700 1 --2- 172.21.15.178:0/1146811922 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f2c0f5410 0x7f7f2c106730 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:07:05.688 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:05.686+0000 7f7f31335700 1 -- 172.21.15.178:0/1146811922 >> 172.21.15.178:0/1146811922 conn(0x7f7f2c0f0b70 msgr2=0x7f7f2c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:07:05.688 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:05.686+0000 7f7f31335700 1 -- 172.21.15.178:0/1146811922 shutdown_connections 2024-02-11T17:07:05.688 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:05.686+0000 7f7f31335700 1 -- 172.21.15.178:0/1146811922 wait complete. 2024-02-11T17:07:05.688 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:07:06.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:06 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:07:05.194220+0000 mgr.smithi037.eofnaq (mgr.14162) 217 : cluster [DBG] pgmap v205: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:07:06.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:06 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:07:05.685513+0000 mon.smithi037 (mon.0) 242 : audit [DBG] from='client.? 172.21.15.178:0/1146811922' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:07:07.445 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:07:07.446 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:07:08.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:08 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:07:07.194586+0000 mgr.smithi037.eofnaq (mgr.14162) 218 : cluster [DBG] pgmap v206: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:07:09.414 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:09.411+0000 7fa0d5bd0700 1 -- 172.21.15.178:0/20054118 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa0d00f5410 msgr2=0x7fa0d00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:07:09.414 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:09.411+0000 7fa0d5bd0700 1 --2- 172.21.15.178:0/20054118 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa0d00f5410 0x7fa0d00f5830 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7fa0c0009fb0 tx=0x7fa0c0009b30).stop 2024-02-11T17:07:09.414 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:09.411+0000 7fa0d5bd0700 1 -- 172.21.15.178:0/20054118 shutdown_connections 2024-02-11T17:07:09.414 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:09.411+0000 7fa0d5bd0700 1 --2- 172.21.15.178:0/20054118 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa0d00f5410 0x7fa0d00f5830 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:07:09.414 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:09.411+0000 7fa0d5bd0700 1 -- 172.21.15.178:0/20054118 >> 172.21.15.178:0/20054118 conn(0x7fa0d00f0b70 msgr2=0x7fa0d00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:07:09.414 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:09.411+0000 7fa0d5bd0700 1 -- 172.21.15.178:0/20054118 shutdown_connections 2024-02-11T17:07:09.414 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:09.411+0000 7fa0d5bd0700 1 -- 172.21.15.178:0/20054118 wait complete. 2024-02-11T17:07:09.414 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:09.411+0000 7fa0d5bd0700 1 Processor -- start 2024-02-11T17:07:09.414 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:09.411+0000 7fa0d5bd0700 1 -- start start 2024-02-11T17:07:09.414 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:09.411+0000 7fa0d5bd0700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa0d00f5410 0x7fa0d0106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:07:09.415 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:09.411+0000 7fa0d5bd0700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0d0106c10 con 0x7fa0d00f5410 2024-02-11T17:07:09.415 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:09.412+0000 7fa0cf7fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa0d00f5410 0x7fa0d0106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:07:09.415 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:09.412+0000 7fa0cf7fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa0d00f5410 0x7fa0d0106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:59198/0 (socket says 172.21.15.178:59198) 2024-02-11T17:07:09.415 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:09.412+0000 7fa0cf7fe700 1 -- 172.21.15.178:0/1581995365 learned_addr learned my addr 172.21.15.178:0/1581995365 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:07:09.415 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:09.412+0000 7fa0cf7fe700 1 -- 172.21.15.178:0/1581995365 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0c0005040 con 0x7fa0d00f5410 2024-02-11T17:07:09.415 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:09.412+0000 7fa0cf7fe700 1 --2- 172.21.15.178:0/1581995365 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa0d00f5410 0x7fa0d0106720 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7fa0c0013510 tx=0x7fa0c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:07:09.415 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:09.412+0000 7fa0ccff9700 1 -- 172.21.15.178:0/1581995365 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa0c0009320 con 0x7fa0d00f5410 2024-02-11T17:07:09.415 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:09.412+0000 7fa0ccff9700 1 -- 172.21.15.178:0/1581995365 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa0c0013f00 con 0x7fa0d00f5410 2024-02-11T17:07:09.415 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:09.412+0000 7fa0d5bd0700 1 -- 172.21.15.178:0/1581995365 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0d0106e10 con 0x7fa0d00f5410 2024-02-11T17:07:09.415 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:09.412+0000 7fa0ccff9700 1 -- 172.21.15.178:0/1581995365 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa0c0014d90 con 0x7fa0d00f5410 2024-02-11T17:07:09.416 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:09.412+0000 7fa0d5bd0700 1 -- 172.21.15.178:0/1581995365 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0d01072a0 con 0x7fa0d00f5410 2024-02-11T17:07:09.416 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:09.413+0000 7fa0ccff9700 1 -- 172.21.15.178:0/1581995365 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa0c0014420 con 0x7fa0d00f5410 2024-02-11T17:07:09.417 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:09.413+0000 7fa0d5bd0700 1 -- 172.21.15.178:0/1581995365 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0d0045ad0 con 0x7fa0d00f5410 2024-02-11T17:07:09.417 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:09.413+0000 7fa0ccff9700 1 --2- 172.21.15.178:0/1581995365 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fa0bc02ff60 0x7fa0bc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:07:09.417 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:09.413+0000 7fa0ccff9700 1 -- 172.21.15.178:0/1581995365 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa0c00410c0 con 0x7fa0d00f5410 2024-02-11T17:07:09.417 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:09.414+0000 7fa0c7fff700 1 --2- 172.21.15.178:0/1581995365 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fa0bc02ff60 0x7fa0bc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:07:09.417 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:09.414+0000 7fa0c7fff700 1 --2- 172.21.15.178:0/1581995365 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fa0bc02ff60 0x7fa0bc032420 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fa0b80096a0 tx=0x7fa0b8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:07:09.419 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:09.418+0000 7fa0ccff9700 1 -- 172.21.15.178:0/1581995365 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa0c00095f0 con 0x7fa0d00f5410 2024-02-11T17:07:09.623 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:09.622+0000 7fa0d5bd0700 1 -- 172.21.15.178:0/1581995365 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa0d0108150 con 0x7fa0d00f5410 2024-02-11T17:07:09.624 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:09.623+0000 7fa0ccff9700 1 -- 172.21.15.178:0/1581995365 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa0c001db70 con 0x7fa0d00f5410 2024-02-11T17:07:09.624 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:07:09.625 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:07:09.626 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:09.624+0000 7fa0d5bd0700 1 -- 172.21.15.178:0/1581995365 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fa0bc02ff60 msgr2=0x7fa0bc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:07:09.626 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:09.624+0000 7fa0d5bd0700 1 --2- 172.21.15.178:0/1581995365 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fa0bc02ff60 0x7fa0bc032420 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fa0b80096a0 tx=0x7fa0b8006b40).stop 2024-02-11T17:07:09.626 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:09.624+0000 7fa0d5bd0700 1 -- 172.21.15.178:0/1581995365 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa0d00f5410 msgr2=0x7fa0d0106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:07:09.626 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:09.624+0000 7fa0d5bd0700 1 --2- 172.21.15.178:0/1581995365 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa0d00f5410 0x7fa0d0106720 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7fa0c0013510 tx=0x7fa0c000b040).stop 2024-02-11T17:07:09.626 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:09.624+0000 7fa0d5bd0700 1 -- 172.21.15.178:0/1581995365 shutdown_connections 2024-02-11T17:07:09.626 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:09.624+0000 7fa0d5bd0700 1 --2- 172.21.15.178:0/1581995365 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fa0bc02ff60 0x7fa0bc032420 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:07:09.626 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:09.624+0000 7fa0d5bd0700 1 --2- 172.21.15.178:0/1581995365 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa0d00f5410 0x7fa0d0106720 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:07:09.626 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:09.624+0000 7fa0d5bd0700 1 -- 172.21.15.178:0/1581995365 >> 172.21.15.178:0/1581995365 conn(0x7fa0d00f0b70 msgr2=0x7fa0d005fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:07:09.627 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:09.624+0000 7fa0d5bd0700 1 -- 172.21.15.178:0/1581995365 shutdown_connections 2024-02-11T17:07:09.627 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:09.624+0000 7fa0d5bd0700 1 -- 172.21.15.178:0/1581995365 wait complete. 2024-02-11T17:07:09.627 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:07:10.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:10 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:07:09.194957+0000 mgr.smithi037.eofnaq (mgr.14162) 219 : cluster [DBG] pgmap v207: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:07:10.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:10 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:07:09.623950+0000 mon.smithi037 (mon.0) 243 : audit [DBG] from='client.? 172.21.15.178:0/1581995365' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:07:11.225 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:07:11.225 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:07:12.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:12 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:07:11.195243+0000 mgr.smithi037.eofnaq (mgr.14162) 220 : cluster [DBG] pgmap v208: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:07:13.238 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:13.236+0000 7ffa91083700 1 -- 172.21.15.178:0/3486553952 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffa8c0f5410 msgr2=0x7ffa8c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:07:13.238 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:13.236+0000 7ffa91083700 1 --2- 172.21.15.178:0/3486553952 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffa8c0f5410 0x7ffa8c0f5830 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7ffa80009fb0 tx=0x7ffa80009b30).stop 2024-02-11T17:07:13.238 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:13.236+0000 7ffa91083700 1 -- 172.21.15.178:0/3486553952 shutdown_connections 2024-02-11T17:07:13.238 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:13.236+0000 7ffa91083700 1 --2- 172.21.15.178:0/3486553952 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffa8c0f5410 0x7ffa8c0f5830 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:07:13.238 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:13.236+0000 7ffa91083700 1 -- 172.21.15.178:0/3486553952 >> 172.21.15.178:0/3486553952 conn(0x7ffa8c0f0b70 msgr2=0x7ffa8c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:07:13.238 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:13.236+0000 7ffa91083700 1 -- 172.21.15.178:0/3486553952 shutdown_connections 2024-02-11T17:07:13.238 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:13.236+0000 7ffa91083700 1 -- 172.21.15.178:0/3486553952 wait complete. 2024-02-11T17:07:13.240 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:13.237+0000 7ffa91083700 1 Processor -- start 2024-02-11T17:07:13.240 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:13.237+0000 7ffa91083700 1 -- start start 2024-02-11T17:07:13.240 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:13.237+0000 7ffa91083700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffa8c0f5410 0x7ffa8c105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:07:13.240 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:13.237+0000 7ffa91083700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa8c105b20 con 0x7ffa8c0f5410 2024-02-11T17:07:13.240 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:13.237+0000 7ffa8ad9d700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffa8c0f5410 0x7ffa8c105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:07:13.240 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:13.237+0000 7ffa8ad9d700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffa8c0f5410 0x7ffa8c105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:59208/0 (socket says 172.21.15.178:59208) 2024-02-11T17:07:13.240 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:13.237+0000 7ffa8ad9d700 1 -- 172.21.15.178:0/4261703743 learned_addr learned my addr 172.21.15.178:0/4261703743 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:07:13.241 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:13.238+0000 7ffa8ad9d700 1 -- 172.21.15.178:0/4261703743 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa80005040 con 0x7ffa8c0f5410 2024-02-11T17:07:13.241 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:13.238+0000 7ffa8ad9d700 1 --2- 172.21.15.178:0/4261703743 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffa8c0f5410 0x7ffa8c105630 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7ffa80013510 tx=0x7ffa8000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:07:13.241 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:13.238+0000 7ffa737fe700 1 -- 172.21.15.178:0/4261703743 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa80009320 con 0x7ffa8c0f5410 2024-02-11T17:07:13.241 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:13.238+0000 7ffa91083700 1 -- 172.21.15.178:0/4261703743 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa8c105d20 con 0x7ffa8c0f5410 2024-02-11T17:07:13.241 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:13.238+0000 7ffa737fe700 1 -- 172.21.15.178:0/4261703743 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffa80013f00 con 0x7ffa8c0f5410 2024-02-11T17:07:13.241 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:13.238+0000 7ffa737fe700 1 -- 172.21.15.178:0/4261703743 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa80014d90 con 0x7ffa8c0f5410 2024-02-11T17:07:13.241 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:13.238+0000 7ffa91083700 1 -- 172.21.15.178:0/4261703743 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa8c1061b0 con 0x7ffa8c0f5410 2024-02-11T17:07:13.241 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:13.239+0000 7ffa737fe700 1 -- 172.21.15.178:0/4261703743 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ffa800095f0 con 0x7ffa8c0f5410 2024-02-11T17:07:13.242 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:13.239+0000 7ffa91083700 1 -- 172.21.15.178:0/4261703743 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa8c045ad0 con 0x7ffa8c0f5410 2024-02-11T17:07:13.242 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:13.239+0000 7ffa737fe700 1 --2- 172.21.15.178:0/4261703743 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7ffa7402ff60 0x7ffa74032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:07:13.242 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:13.239+0000 7ffa737fe700 1 -- 172.21.15.178:0/4261703743 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ffa80040fe0 con 0x7ffa8c0f5410 2024-02-11T17:07:13.242 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:13.239+0000 7ffa8a59c700 1 --2- 172.21.15.178:0/4261703743 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7ffa7402ff60 0x7ffa74032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:07:13.242 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:13.240+0000 7ffa8a59c700 1 --2- 172.21.15.178:0/4261703743 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7ffa7402ff60 0x7ffa74032420 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7ffa7c0096a0 tx=0x7ffa7c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:07:13.245 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:13.244+0000 7ffa737fe700 1 -- 172.21.15.178:0/4261703743 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ffa8001d9d0 con 0x7ffa8c0f5410 2024-02-11T17:07:13.453 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:13.451+0000 7ffa91083700 1 -- 172.21.15.178:0/4261703743 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ffa8c0fae60 con 0x7ffa8c0f5410 2024-02-11T17:07:13.453 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:13.452+0000 7ffa737fe700 1 -- 172.21.15.178:0/4261703743 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ffa80010230 con 0x7ffa8c0f5410 2024-02-11T17:07:13.454 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:07:13.454 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:07:13.455 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:13.453+0000 7ffa91083700 1 -- 172.21.15.178:0/4261703743 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7ffa7402ff60 msgr2=0x7ffa74032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:07:13.455 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:13.453+0000 7ffa91083700 1 --2- 172.21.15.178:0/4261703743 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7ffa7402ff60 0x7ffa74032420 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7ffa7c0096a0 tx=0x7ffa7c006b40).stop 2024-02-11T17:07:13.456 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:13.453+0000 7ffa91083700 1 -- 172.21.15.178:0/4261703743 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffa8c0f5410 msgr2=0x7ffa8c105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:07:13.456 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:13.453+0000 7ffa91083700 1 --2- 172.21.15.178:0/4261703743 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffa8c0f5410 0x7ffa8c105630 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7ffa80013510 tx=0x7ffa8000b040).stop 2024-02-11T17:07:13.456 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:13.454+0000 7ffa91083700 1 -- 172.21.15.178:0/4261703743 shutdown_connections 2024-02-11T17:07:13.456 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:13.454+0000 7ffa91083700 1 --2- 172.21.15.178:0/4261703743 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7ffa7402ff60 0x7ffa74032420 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:07:13.456 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:13.454+0000 7ffa91083700 1 --2- 172.21.15.178:0/4261703743 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffa8c0f5410 0x7ffa8c105630 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:07:13.456 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:13.454+0000 7ffa91083700 1 -- 172.21.15.178:0/4261703743 >> 172.21.15.178:0/4261703743 conn(0x7ffa8c0f0b70 msgr2=0x7ffa8c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:07:13.456 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:13.454+0000 7ffa91083700 1 -- 172.21.15.178:0/4261703743 shutdown_connections 2024-02-11T17:07:13.456 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:13.454+0000 7ffa91083700 1 -- 172.21.15.178:0/4261703743 wait complete. 2024-02-11T17:07:13.456 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:07:14.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:14 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:07:13.195579+0000 mgr.smithi037.eofnaq (mgr.14162) 221 : cluster [DBG] pgmap v209: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:07:14.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:14 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:07:13.453146+0000 mon.smithi037 (mon.0) 244 : audit [DBG] from='client.? 172.21.15.178:0/4261703743' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:07:15.126 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:07:15.126 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:07:16.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:16 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:07:15.195942+0000 mgr.smithi037.eofnaq (mgr.14162) 222 : cluster [DBG] pgmap v210: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:07:17.071 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:17.035+0000 7f931759a700 1 -- 172.21.15.178:0/1022586264 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f93100f57f0 msgr2=0x7f93100f5c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:07:17.072 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:17.035+0000 7f931759a700 1 --2- 172.21.15.178:0/1022586264 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f93100f57f0 0x7f93100f5c10 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f9300009fb0 tx=0x7f9300009b30).stop 2024-02-11T17:07:17.072 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:17.035+0000 7f931759a700 1 -- 172.21.15.178:0/1022586264 shutdown_connections 2024-02-11T17:07:17.072 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:17.035+0000 7f931759a700 1 --2- 172.21.15.178:0/1022586264 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f93100f57f0 0x7f93100f5c10 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:07:17.072 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:17.035+0000 7f931759a700 1 -- 172.21.15.178:0/1022586264 >> 172.21.15.178:0/1022586264 conn(0x7f93100f0120 msgr2=0x7f93100f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:07:17.072 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:17.035+0000 7f931759a700 1 -- 172.21.15.178:0/1022586264 shutdown_connections 2024-02-11T17:07:17.072 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:17.035+0000 7f931759a700 1 -- 172.21.15.178:0/1022586264 wait complete. 2024-02-11T17:07:17.072 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:17.035+0000 7f931759a700 1 Processor -- start 2024-02-11T17:07:17.072 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:17.035+0000 7f931759a700 1 -- start start 2024-02-11T17:07:17.072 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:17.035+0000 7f931759a700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f93100f57f0 0x7f93101086e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:07:17.072 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:17.036+0000 7f931759a700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9310108bd0 con 0x7f93100f57f0 2024-02-11T17:07:17.072 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:17.036+0000 7f9316598700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f93100f57f0 0x7f93101086e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:07:17.073 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:17.036+0000 7f9316598700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f93100f57f0 0x7f93101086e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:59228/0 (socket says 172.21.15.178:59228) 2024-02-11T17:07:17.073 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:17.036+0000 7f9316598700 1 -- 172.21.15.178:0/4232099002 learned_addr learned my addr 172.21.15.178:0/4232099002 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:07:17.073 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:17.036+0000 7f9316598700 1 -- 172.21.15.178:0/4232099002 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9300005040 con 0x7f93100f57f0 2024-02-11T17:07:17.073 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:17.036+0000 7f9316598700 1 --2- 172.21.15.178:0/4232099002 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f93100f57f0 0x7f93101086e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f9300013510 tx=0x7f930000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:07:17.073 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:17.036+0000 7f9306ffd700 1 -- 172.21.15.178:0/4232099002 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9300009320 con 0x7f93100f57f0 2024-02-11T17:07:17.073 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:17.036+0000 7f931759a700 1 -- 172.21.15.178:0/4232099002 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9310108dd0 con 0x7f93100f57f0 2024-02-11T17:07:17.073 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:17.036+0000 7f9306ffd700 1 -- 172.21.15.178:0/4232099002 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9300013f00 con 0x7f93100f57f0 2024-02-11T17:07:17.073 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:17.036+0000 7f9306ffd700 1 -- 172.21.15.178:0/4232099002 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9300014dc0 con 0x7f93100f57f0 2024-02-11T17:07:17.073 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:17.036+0000 7f931759a700 1 -- 172.21.15.178:0/4232099002 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9310109260 con 0x7f93100f57f0 2024-02-11T17:07:17.073 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:17.037+0000 7f931759a700 1 -- 172.21.15.178:0/4232099002 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9310045ad0 con 0x7f93100f57f0 2024-02-11T17:07:17.073 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:17.037+0000 7f9306ffd700 1 -- 172.21.15.178:0/4232099002 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f93000095f0 con 0x7f93100f57f0 2024-02-11T17:07:17.073 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:17.037+0000 7f9306ffd700 1 --2- 172.21.15.178:0/4232099002 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f92fc02ff60 0x7f92fc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:07:17.074 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:17.037+0000 7f9306ffd700 1 -- 172.21.15.178:0/4232099002 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9300041020 con 0x7f93100f57f0 2024-02-11T17:07:17.074 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:17.038+0000 7f9315d97700 1 --2- 172.21.15.178:0/4232099002 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f92fc02ff60 0x7f92fc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:07:17.074 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:17.038+0000 7f9315d97700 1 --2- 172.21.15.178:0/4232099002 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f92fc02ff60 0x7f92fc032420 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f930c0096a0 tx=0x7f930c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:07:17.074 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:17.042+0000 7f9306ffd700 1 -- 172.21.15.178:0/4232099002 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9300014420 con 0x7f93100f57f0 2024-02-11T17:07:17.250 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:17.248+0000 7f931759a700 1 -- 172.21.15.178:0/4232099002 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f931010a0b0 con 0x7f93100f57f0 2024-02-11T17:07:17.250 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:17.249+0000 7f9306ffd700 1 -- 172.21.15.178:0/4232099002 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f930001c920 con 0x7f93100f57f0 2024-02-11T17:07:17.251 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:07:17.251 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:07:17.252 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:17.250+0000 7f931759a700 1 -- 172.21.15.178:0/4232099002 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f92fc02ff60 msgr2=0x7f92fc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:07:17.253 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:17.250+0000 7f931759a700 1 --2- 172.21.15.178:0/4232099002 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f92fc02ff60 0x7f92fc032420 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f930c0096a0 tx=0x7f930c006b40).stop 2024-02-11T17:07:17.253 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:17.250+0000 7f931759a700 1 -- 172.21.15.178:0/4232099002 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f93100f57f0 msgr2=0x7f93101086e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:07:17.253 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:17.250+0000 7f931759a700 1 --2- 172.21.15.178:0/4232099002 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f93100f57f0 0x7f93101086e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f9300013510 tx=0x7f930000b040).stop 2024-02-11T17:07:17.253 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:17.250+0000 7f931759a700 1 -- 172.21.15.178:0/4232099002 shutdown_connections 2024-02-11T17:07:17.253 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:17.250+0000 7f931759a700 1 --2- 172.21.15.178:0/4232099002 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f92fc02ff60 0x7f92fc032420 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:07:17.253 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:17.250+0000 7f931759a700 1 --2- 172.21.15.178:0/4232099002 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f93100f57f0 0x7f93101086e0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:07:17.253 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:17.250+0000 7f931759a700 1 -- 172.21.15.178:0/4232099002 >> 172.21.15.178:0/4232099002 conn(0x7f93100f0120 msgr2=0x7f93100f4d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:07:17.253 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:17.251+0000 7f931759a700 1 -- 172.21.15.178:0/4232099002 shutdown_connections 2024-02-11T17:07:17.253 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:17.251+0000 7f931759a700 1 -- 172.21.15.178:0/4232099002 wait complete. 2024-02-11T17:07:17.253 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:07:18.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:18 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:07:17.196269+0000 mgr.smithi037.eofnaq (mgr.14162) 223 : cluster [DBG] pgmap v211: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:07:18.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:18 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:07:17.250264+0000 mon.smithi037 (mon.0) 245 : audit [DBG] from='client.? 172.21.15.178:0/4232099002' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:07:18.856 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:07:18.856 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:07:20.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:20 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:07:19.196650+0000 mgr.smithi037.eofnaq (mgr.14162) 224 : cluster [DBG] pgmap v212: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:07:20.806 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:20.803+0000 7f418cb16700 1 -- 172.21.15.178:0/3592667889 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41880f5410 msgr2=0x7f41880f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:07:20.806 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:20.803+0000 7f418cb16700 1 --2- 172.21.15.178:0/3592667889 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41880f5410 0x7f41880f5830 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f4170009fb0 tx=0x7f4170009b30).stop 2024-02-11T17:07:20.806 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:20.803+0000 7f418cb16700 1 -- 172.21.15.178:0/3592667889 shutdown_connections 2024-02-11T17:07:20.806 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:20.803+0000 7f418cb16700 1 --2- 172.21.15.178:0/3592667889 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41880f5410 0x7f41880f5830 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:07:20.807 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:20.803+0000 7f418cb16700 1 -- 172.21.15.178:0/3592667889 >> 172.21.15.178:0/3592667889 conn(0x7f41880f0b70 msgr2=0x7f41880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:07:20.807 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:20.804+0000 7f418cb16700 1 -- 172.21.15.178:0/3592667889 shutdown_connections 2024-02-11T17:07:20.807 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:20.804+0000 7f418cb16700 1 -- 172.21.15.178:0/3592667889 wait complete. 2024-02-11T17:07:20.807 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:20.804+0000 7f418cb16700 1 Processor -- start 2024-02-11T17:07:20.807 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:20.804+0000 7f418cb16700 1 -- start start 2024-02-11T17:07:20.807 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:20.804+0000 7f418cb16700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41880f5410 0x7f4188106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:07:20.807 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:20.804+0000 7f418cb16700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4188106bf0 con 0x7f41880f5410 2024-02-11T17:07:20.807 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:20.805+0000 7f418659c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41880f5410 0x7f4188106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:07:20.807 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:20.805+0000 7f418659c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41880f5410 0x7f4188106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:49652/0 (socket says 172.21.15.178:49652) 2024-02-11T17:07:20.807 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:20.805+0000 7f418659c700 1 -- 172.21.15.178:0/588330931 learned_addr learned my addr 172.21.15.178:0/588330931 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:07:20.808 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:20.805+0000 7f418659c700 1 -- 172.21.15.178:0/588330931 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4170005040 con 0x7f41880f5410 2024-02-11T17:07:20.809 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:20.805+0000 7f418659c700 1 --2- 172.21.15.178:0/588330931 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41880f5410 0x7f4188106700 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f4170013510 tx=0x7f417000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:07:20.809 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:20.805+0000 7f417effd700 1 -- 172.21.15.178:0/588330931 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4170009320 con 0x7f41880f5410 2024-02-11T17:07:20.809 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:20.805+0000 7f418cb16700 1 -- 172.21.15.178:0/588330931 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4188106df0 con 0x7f41880f5410 2024-02-11T17:07:20.809 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:20.805+0000 7f417effd700 1 -- 172.21.15.178:0/588330931 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4170013f00 con 0x7f41880f5410 2024-02-11T17:07:20.809 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:20.805+0000 7f418cb16700 1 -- 172.21.15.178:0/588330931 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4188107280 con 0x7f41880f5410 2024-02-11T17:07:20.809 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:20.805+0000 7f417effd700 1 -- 172.21.15.178:0/588330931 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4170014d90 con 0x7f41880f5410 2024-02-11T17:07:20.809 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:20.806+0000 7f418cb16700 1 -- 172.21.15.178:0/588330931 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4188045ad0 con 0x7f41880f5410 2024-02-11T17:07:20.809 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:20.806+0000 7f417effd700 1 -- 172.21.15.178:0/588330931 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f41700095f0 con 0x7f41880f5410 2024-02-11T17:07:20.809 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:20.806+0000 7f417effd700 1 --2- 172.21.15.178:0/588330931 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f417402ff60 0x7f4174032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:07:20.809 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:20.806+0000 7f417effd700 1 -- 172.21.15.178:0/588330931 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4170040ff0 con 0x7f41880f5410 2024-02-11T17:07:20.809 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:20.806+0000 7f4185d9b700 1 --2- 172.21.15.178:0/588330931 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f417402ff60 0x7f4174032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:07:20.809 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:20.807+0000 7f4185d9b700 1 --2- 172.21.15.178:0/588330931 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f417402ff60 0x7f4174032420 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f41780096a0 tx=0x7f4178006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:07:20.812 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:20.811+0000 7f417effd700 1 -- 172.21.15.178:0/588330931 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4170014420 con 0x7f41880f5410 2024-02-11T17:07:21.018 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:21.016+0000 7f418cb16700 1 -- 172.21.15.178:0/588330931 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f41880fae60 con 0x7f41880f5410 2024-02-11T17:07:21.018 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:21.017+0000 7f417effd700 1 -- 172.21.15.178:0/588330931 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4170010230 con 0x7f41880f5410 2024-02-11T17:07:21.019 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:07:21.019 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:07:21.020 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:21.018+0000 7f418cb16700 1 -- 172.21.15.178:0/588330931 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f417402ff60 msgr2=0x7f4174032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:07:21.020 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:21.018+0000 7f418cb16700 1 --2- 172.21.15.178:0/588330931 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f417402ff60 0x7f4174032420 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f41780096a0 tx=0x7f4178006b40).stop 2024-02-11T17:07:21.020 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:21.018+0000 7f418cb16700 1 -- 172.21.15.178:0/588330931 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41880f5410 msgr2=0x7f4188106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:07:21.021 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:21.018+0000 7f418cb16700 1 --2- 172.21.15.178:0/588330931 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41880f5410 0x7f4188106700 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f4170013510 tx=0x7f417000b040).stop 2024-02-11T17:07:21.021 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:21.018+0000 7f418cb16700 1 -- 172.21.15.178:0/588330931 shutdown_connections 2024-02-11T17:07:21.021 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:21.018+0000 7f418cb16700 1 --2- 172.21.15.178:0/588330931 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f417402ff60 0x7f4174032420 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:07:21.021 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:21.018+0000 7f418cb16700 1 --2- 172.21.15.178:0/588330931 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41880f5410 0x7f4188106700 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:07:21.021 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:21.018+0000 7f418cb16700 1 -- 172.21.15.178:0/588330931 >> 172.21.15.178:0/588330931 conn(0x7f41880f0b70 msgr2=0x7f41880f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:07:21.021 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:21.018+0000 7f418cb16700 1 -- 172.21.15.178:0/588330931 shutdown_connections 2024-02-11T17:07:21.021 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:21.019+0000 7f418cb16700 1 -- 172.21.15.178:0/588330931 wait complete. 2024-02-11T17:07:21.023 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:07:21.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:21 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:07:21.018062+0000 mon.smithi037 (mon.0) 246 : audit [DBG] from='client.? 172.21.15.178:0/588330931' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:07:22.568 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:07:22.629 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:07:22.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:22 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:07:21.196993+0000 mgr.smithi037.eofnaq (mgr.14162) 225 : cluster [DBG] pgmap v213: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:07:24.620 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:24.581+0000 7fe021f82700 1 -- 172.21.15.178:0/4190476028 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe01c0f5410 msgr2=0x7fe01c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:07:24.620 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:24.581+0000 7fe021f82700 1 --2- 172.21.15.178:0/4190476028 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe01c0f5410 0x7fe01c0f5830 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7fe00c009fb0 tx=0x7fe00c009b30).stop 2024-02-11T17:07:24.620 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:24.581+0000 7fe021f82700 1 -- 172.21.15.178:0/4190476028 shutdown_connections 2024-02-11T17:07:24.621 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:24.581+0000 7fe021f82700 1 --2- 172.21.15.178:0/4190476028 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe01c0f5410 0x7fe01c0f5830 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:07:24.621 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:24.581+0000 7fe021f82700 1 -- 172.21.15.178:0/4190476028 >> 172.21.15.178:0/4190476028 conn(0x7fe01c0f0b70 msgr2=0x7fe01c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:07:24.621 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:24.581+0000 7fe021f82700 1 -- 172.21.15.178:0/4190476028 shutdown_connections 2024-02-11T17:07:24.621 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:24.581+0000 7fe021f82700 1 -- 172.21.15.178:0/4190476028 wait complete. 2024-02-11T17:07:24.621 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:24.582+0000 7fe021f82700 1 Processor -- start 2024-02-11T17:07:24.621 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:24.582+0000 7fe021f82700 1 -- start start 2024-02-11T17:07:24.621 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:24.583+0000 7fe021f82700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe01c0f5410 0x7fe01c109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:07:24.621 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:24.583+0000 7fe021f82700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe01c060220 con 0x7fe01c0f5410 2024-02-11T17:07:24.621 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:24.583+0000 7fe01b7fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe01c0f5410 0x7fe01c109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:07:24.621 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:24.583+0000 7fe01b7fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe01c0f5410 0x7fe01c109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:49670/0 (socket says 172.21.15.178:49670) 2024-02-11T17:07:24.621 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:24.583+0000 7fe01b7fe700 1 -- 172.21.15.178:0/4243093843 learned_addr learned my addr 172.21.15.178:0/4243093843 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:07:24.621 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:24.583+0000 7fe01b7fe700 1 -- 172.21.15.178:0/4243093843 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe00c005040 con 0x7fe01c0f5410 2024-02-11T17:07:24.621 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:24.583+0000 7fe01b7fe700 1 --2- 172.21.15.178:0/4243093843 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe01c0f5410 0x7fe01c109450 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7fe00c013510 tx=0x7fe00c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:07:24.621 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:24.583+0000 7fe003fff700 1 -- 172.21.15.178:0/4243093843 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe00c009320 con 0x7fe01c0f5410 2024-02-11T17:07:24.622 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:24.583+0000 7fe021f82700 1 -- 172.21.15.178:0/4243093843 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe01c060420 con 0x7fe01c0f5410 2024-02-11T17:07:24.622 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:24.583+0000 7fe003fff700 1 -- 172.21.15.178:0/4243093843 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe00c013f00 con 0x7fe01c0f5410 2024-02-11T17:07:24.622 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:24.583+0000 7fe003fff700 1 -- 172.21.15.178:0/4243093843 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe00c014d90 con 0x7fe01c0f5410 2024-02-11T17:07:24.622 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:24.584+0000 7fe021f82700 1 -- 172.21.15.178:0/4243093843 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe01c109b00 con 0x7fe01c0f5410 2024-02-11T17:07:24.622 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:24.584+0000 7fe003fff700 1 -- 172.21.15.178:0/4243093843 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe00c0095f0 con 0x7fe01c0f5410 2024-02-11T17:07:24.622 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:24.584+0000 7fe021f82700 1 -- 172.21.15.178:0/4243093843 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe01c059100 con 0x7fe01c0f5410 2024-02-11T17:07:24.622 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:24.584+0000 7fe003fff700 1 --2- 172.21.15.178:0/4243093843 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fe00402ff60 0x7fe004032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:07:24.622 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:24.584+0000 7fe003fff700 1 -- 172.21.15.178:0/4243093843 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe00c041030 con 0x7fe01c0f5410 2024-02-11T17:07:24.622 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:24.585+0000 7fe01affd700 1 --2- 172.21.15.178:0/4243093843 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fe00402ff60 0x7fe004032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:07:24.622 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:24.585+0000 7fe01affd700 1 --2- 172.21.15.178:0/4243093843 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fe00402ff60 0x7fe004032420 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fe0100096a0 tx=0x7fe010006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:07:24.622 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:24.589+0000 7fe003fff700 1 -- 172.21.15.178:0/4243093843 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe00c014420 con 0x7fe01c0f5410 2024-02-11T17:07:24.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:24 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:07:23.197388+0000 mgr.smithi037.eofnaq (mgr.14162) 226 : cluster [DBG] pgmap v214: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:07:24.796 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:24.795+0000 7fe021f82700 1 -- 172.21.15.178:0/4243093843 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe01c045ad0 con 0x7fe01c0f5410 2024-02-11T17:07:24.797 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:24.796+0000 7fe003fff700 1 -- 172.21.15.178:0/4243093843 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe00c010230 con 0x7fe01c0f5410 2024-02-11T17:07:24.797 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:07:24.798 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:07:24.799 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:24.797+0000 7fe021f82700 1 -- 172.21.15.178:0/4243093843 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fe00402ff60 msgr2=0x7fe004032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:07:24.799 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:24.797+0000 7fe021f82700 1 --2- 172.21.15.178:0/4243093843 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fe00402ff60 0x7fe004032420 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fe0100096a0 tx=0x7fe010006b40).stop 2024-02-11T17:07:24.799 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:24.797+0000 7fe021f82700 1 -- 172.21.15.178:0/4243093843 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe01c0f5410 msgr2=0x7fe01c109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:07:24.799 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:24.797+0000 7fe021f82700 1 --2- 172.21.15.178:0/4243093843 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe01c0f5410 0x7fe01c109450 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7fe00c013510 tx=0x7fe00c00b040).stop 2024-02-11T17:07:24.799 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:24.797+0000 7fe021f82700 1 -- 172.21.15.178:0/4243093843 shutdown_connections 2024-02-11T17:07:24.799 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:24.797+0000 7fe021f82700 1 --2- 172.21.15.178:0/4243093843 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fe00402ff60 0x7fe004032420 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:07:24.799 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:24.797+0000 7fe021f82700 1 --2- 172.21.15.178:0/4243093843 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe01c0f5410 0x7fe01c109450 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:07:24.799 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:24.797+0000 7fe021f82700 1 -- 172.21.15.178:0/4243093843 >> 172.21.15.178:0/4243093843 conn(0x7fe01c0f0b70 msgr2=0x7fe01c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:07:24.800 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:24.797+0000 7fe021f82700 1 -- 172.21.15.178:0/4243093843 shutdown_connections 2024-02-11T17:07:24.800 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:24.797+0000 7fe021f82700 1 -- 172.21.15.178:0/4243093843 wait complete. 2024-02-11T17:07:24.800 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:07:25.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:25 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:07:24.796986+0000 mon.smithi037 (mon.0) 247 : audit [DBG] from='client.? 172.21.15.178:0/4243093843' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:07:26.413 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:07:26.413 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:07:26.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:26 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:07:25.197723+0000 mgr.smithi037.eofnaq (mgr.14162) 227 : cluster [DBG] pgmap v215: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:07:28.375 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:28.372+0000 7f7b6b59e700 1 -- 172.21.15.178:0/1929008589 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7b6c0f62a0 msgr2=0x7f7b6c0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:07:28.375 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:28.372+0000 7f7b6b59e700 1 --2- 172.21.15.178:0/1929008589 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7b6c0f62a0 0x7f7b6c0f66c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f7b5c009fb0 tx=0x7f7b5c009b30).stop 2024-02-11T17:07:28.375 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:28.372+0000 7f7b6b59e700 1 -- 172.21.15.178:0/1929008589 shutdown_connections 2024-02-11T17:07:28.375 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:28.372+0000 7f7b6b59e700 1 --2- 172.21.15.178:0/1929008589 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7b6c0f62a0 0x7f7b6c0f66c0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:07:28.375 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:28.372+0000 7f7b6b59e700 1 -- 172.21.15.178:0/1929008589 >> 172.21.15.178:0/1929008589 conn(0x7f7b6c0f1aa0 msgr2=0x7f7b6c0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:07:28.375 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:28.373+0000 7f7b6b59e700 1 -- 172.21.15.178:0/1929008589 shutdown_connections 2024-02-11T17:07:28.376 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:28.373+0000 7f7b6b59e700 1 -- 172.21.15.178:0/1929008589 wait complete. 2024-02-11T17:07:28.376 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:28.373+0000 7f7b6b59e700 1 Processor -- start 2024-02-11T17:07:28.376 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:28.373+0000 7f7b6b59e700 1 -- start start 2024-02-11T17:07:28.376 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:28.374+0000 7f7b6b59e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7b6c0f62a0 0x7f7b6c1064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:07:28.376 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:28.374+0000 7f7b6b59e700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b6c1069a0 con 0x7f7b6c0f62a0 2024-02-11T17:07:28.377 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:28.374+0000 7f7b6a59c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7b6c0f62a0 0x7f7b6c1064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:07:28.377 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:28.374+0000 7f7b6a59c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7b6c0f62a0 0x7f7b6c1064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:49702/0 (socket says 172.21.15.178:49702) 2024-02-11T17:07:28.378 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:28.374+0000 7f7b6a59c700 1 -- 172.21.15.178:0/207603448 learned_addr learned my addr 172.21.15.178:0/207603448 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:07:28.378 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:28.374+0000 7f7b6a59c700 1 -- 172.21.15.178:0/207603448 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b5c005040 con 0x7f7b6c0f62a0 2024-02-11T17:07:28.378 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:28.374+0000 7f7b6a59c700 1 --2- 172.21.15.178:0/207603448 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7b6c0f62a0 0x7f7b6c1064b0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f7b5c013510 tx=0x7f7b5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:07:28.378 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:28.375+0000 7f7b5affd700 1 -- 172.21.15.178:0/207603448 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b5c009320 con 0x7f7b6c0f62a0 2024-02-11T17:07:28.378 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:28.375+0000 7f7b6b59e700 1 -- 172.21.15.178:0/207603448 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b6c106ba0 con 0x7f7b6c0f62a0 2024-02-11T17:07:28.378 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:28.375+0000 7f7b5affd700 1 -- 172.21.15.178:0/207603448 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7b5c013f00 con 0x7f7b6c0f62a0 2024-02-11T17:07:28.378 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:28.375+0000 7f7b5affd700 1 -- 172.21.15.178:0/207603448 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b5c014dc0 con 0x7f7b6c0f62a0 2024-02-11T17:07:28.378 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:28.375+0000 7f7b6b59e700 1 -- 172.21.15.178:0/207603448 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b6c107030 con 0x7f7b6c0f62a0 2024-02-11T17:07:28.378 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:28.375+0000 7f7b5affd700 1 -- 172.21.15.178:0/207603448 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7b5c0095f0 con 0x7f7b6c0f62a0 2024-02-11T17:07:28.378 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:28.376+0000 7f7b6b59e700 1 -- 172.21.15.178:0/207603448 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b6c045ad0 con 0x7f7b6c0f62a0 2024-02-11T17:07:28.378 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:28.376+0000 7f7b5affd700 1 --2- 172.21.15.178:0/207603448 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f7b5402ffb0 0x7f7b54032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:07:28.378 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:28.376+0000 7f7b5affd700 1 -- 172.21.15.178:0/207603448 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7b5c040ff0 con 0x7f7b6c0f62a0 2024-02-11T17:07:28.378 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:28.376+0000 7f7b69d9b700 1 --2- 172.21.15.178:0/207603448 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f7b5402ffb0 0x7f7b54032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:07:28.379 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:28.376+0000 7f7b69d9b700 1 --2- 172.21.15.178:0/207603448 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f7b5402ffb0 0x7f7b54032470 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f7b600096a0 tx=0x7f7b60006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:07:28.382 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:28.381+0000 7f7b5affd700 1 -- 172.21.15.178:0/207603448 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7b5c06d2b0 con 0x7f7b6c0f62a0 2024-02-11T17:07:28.587 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:28.585+0000 7f7b6b59e700 1 -- 172.21.15.178:0/207603448 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7b6c107ee0 con 0x7f7b6c0f62a0 2024-02-11T17:07:28.588 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:28.586+0000 7f7b5affd700 1 -- 172.21.15.178:0/207603448 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7b5c014420 con 0x7f7b6c0f62a0 2024-02-11T17:07:28.588 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:07:28.588 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:07:28.589 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:28.587+0000 7f7b6b59e700 1 -- 172.21.15.178:0/207603448 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f7b5402ffb0 msgr2=0x7f7b54032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:07:28.589 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:28.587+0000 7f7b6b59e700 1 --2- 172.21.15.178:0/207603448 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f7b5402ffb0 0x7f7b54032470 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f7b600096a0 tx=0x7f7b60006b40).stop 2024-02-11T17:07:28.589 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:28.587+0000 7f7b6b59e700 1 -- 172.21.15.178:0/207603448 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7b6c0f62a0 msgr2=0x7f7b6c1064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:07:28.589 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:28.587+0000 7f7b6b59e700 1 --2- 172.21.15.178:0/207603448 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7b6c0f62a0 0x7f7b6c1064b0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f7b5c013510 tx=0x7f7b5c00b040).stop 2024-02-11T17:07:28.589 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:28.587+0000 7f7b6b59e700 1 -- 172.21.15.178:0/207603448 shutdown_connections 2024-02-11T17:07:28.590 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:28.588+0000 7f7b6b59e700 1 --2- 172.21.15.178:0/207603448 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f7b5402ffb0 0x7f7b54032470 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:07:28.590 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:28.588+0000 7f7b6b59e700 1 --2- 172.21.15.178:0/207603448 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7b6c0f62a0 0x7f7b6c1064b0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:07:28.590 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:28.588+0000 7f7b6b59e700 1 -- 172.21.15.178:0/207603448 >> 172.21.15.178:0/207603448 conn(0x7f7b6c0f1aa0 msgr2=0x7f7b6c0f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:07:28.590 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:28.588+0000 7f7b6b59e700 1 -- 172.21.15.178:0/207603448 shutdown_connections 2024-02-11T17:07:28.590 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:28.588+0000 7f7b6b59e700 1 -- 172.21.15.178:0/207603448 wait complete. 2024-02-11T17:07:28.590 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:07:28.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:28 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:07:27.198057+0000 mgr.smithi037.eofnaq (mgr.14162) 228 : cluster [DBG] pgmap v216: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:07:29.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:29 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:07:28.587316+0000 mon.smithi037 (mon.0) 248 : audit [DBG] from='client.? 172.21.15.178:0/207603448' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:07:30.216 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:07:30.217 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:07:30.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:30 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:07:29.198358+0000 mgr.smithi037.eofnaq (mgr.14162) 229 : cluster [DBG] pgmap v217: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:07:32.223 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:32.221+0000 7fb3df59e700 1 -- 172.21.15.178:0/4138994468 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3e00f51a0 msgr2=0x7fb3e00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:07:32.223 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:32.221+0000 7fb3df59e700 1 --2- 172.21.15.178:0/4138994468 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3e00f51a0 0x7fb3e00f55c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7fb3d4009fb0 tx=0x7fb3d4009b30).stop 2024-02-11T17:07:32.223 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:32.221+0000 7fb3df59e700 1 -- 172.21.15.178:0/4138994468 shutdown_connections 2024-02-11T17:07:32.223 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:32.221+0000 7fb3df59e700 1 --2- 172.21.15.178:0/4138994468 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3e00f51a0 0x7fb3e00f55c0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:07:32.223 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:32.221+0000 7fb3df59e700 1 -- 172.21.15.178:0/4138994468 >> 172.21.15.178:0/4138994468 conn(0x7fb3e00f09a0 msgr2=0x7fb3e00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:07:32.223 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:32.221+0000 7fb3df59e700 1 -- 172.21.15.178:0/4138994468 shutdown_connections 2024-02-11T17:07:32.223 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:32.221+0000 7fb3df59e700 1 -- 172.21.15.178:0/4138994468 wait complete. 2024-02-11T17:07:32.225 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:32.221+0000 7fb3df59e700 1 Processor -- start 2024-02-11T17:07:32.225 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:32.221+0000 7fb3df59e700 1 -- start start 2024-02-11T17:07:32.225 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:32.222+0000 7fb3df59e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3e00f51a0 0x7fb3e01053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:07:32.225 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:32.222+0000 7fb3df59e700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e01058a0 con 0x7fb3e00f51a0 2024-02-11T17:07:32.225 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:32.222+0000 7fb3de59c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3e00f51a0 0x7fb3e01053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:07:32.225 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:32.222+0000 7fb3de59c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3e00f51a0 0x7fb3e01053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:57454/0 (socket says 172.21.15.178:57454) 2024-02-11T17:07:32.225 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:32.222+0000 7fb3de59c700 1 -- 172.21.15.178:0/1002692314 learned_addr learned my addr 172.21.15.178:0/1002692314 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:07:32.225 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:32.222+0000 7fb3de59c700 1 -- 172.21.15.178:0/1002692314 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3d4005040 con 0x7fb3e00f51a0 2024-02-11T17:07:32.226 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:32.222+0000 7fb3de59c700 1 --2- 172.21.15.178:0/1002692314 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3e00f51a0 0x7fb3e01053b0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7fb3d4013510 tx=0x7fb3d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:07:32.226 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:32.222+0000 7fb3ceffd700 1 -- 172.21.15.178:0/1002692314 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3d4009320 con 0x7fb3e00f51a0 2024-02-11T17:07:32.226 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:32.222+0000 7fb3df59e700 1 -- 172.21.15.178:0/1002692314 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3e0105aa0 con 0x7fb3e00f51a0 2024-02-11T17:07:32.226 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:32.222+0000 7fb3ceffd700 1 -- 172.21.15.178:0/1002692314 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb3d4013f00 con 0x7fb3e00f51a0 2024-02-11T17:07:32.226 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:32.222+0000 7fb3ceffd700 1 -- 172.21.15.178:0/1002692314 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3d4014dc0 con 0x7fb3e00f51a0 2024-02-11T17:07:32.226 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:32.222+0000 7fb3df59e700 1 -- 172.21.15.178:0/1002692314 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3e0105f30 con 0x7fb3e00f51a0 2024-02-11T17:07:32.226 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:32.223+0000 7fb3df59e700 1 -- 172.21.15.178:0/1002692314 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3e0045ad0 con 0x7fb3e00f51a0 2024-02-11T17:07:32.226 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:32.223+0000 7fb3ceffd700 1 -- 172.21.15.178:0/1002692314 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb3d40095f0 con 0x7fb3e00f51a0 2024-02-11T17:07:32.226 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:32.223+0000 7fb3ceffd700 1 --2- 172.21.15.178:0/1002692314 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb3c802ff60 0x7fb3c8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:07:32.226 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:32.223+0000 7fb3ceffd700 1 -- 172.21.15.178:0/1002692314 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb3d4042e40 con 0x7fb3e00f51a0 2024-02-11T17:07:32.226 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:32.224+0000 7fb3ddd9b700 1 --2- 172.21.15.178:0/1002692314 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb3c802ff60 0x7fb3c8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:07:32.227 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:32.224+0000 7fb3ddd9b700 1 --2- 172.21.15.178:0/1002692314 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb3c802ff60 0x7fb3c8032420 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7fb3d00096a0 tx=0x7fb3d0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:07:32.230 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:32.228+0000 7fb3ceffd700 1 -- 172.21.15.178:0/1002692314 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb3d4014420 con 0x7fb3e00f51a0 2024-02-11T17:07:32.435 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:32.433+0000 7fb3df59e700 1 -- 172.21.15.178:0/1002692314 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb3e0106de0 con 0x7fb3e00f51a0 2024-02-11T17:07:32.435 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:32.434+0000 7fb3ceffd700 1 -- 172.21.15.178:0/1002692314 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb3d401f7f0 con 0x7fb3e00f51a0 2024-02-11T17:07:32.436 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:07:32.436 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:07:32.437 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:32.436+0000 7fb3df59e700 1 -- 172.21.15.178:0/1002692314 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb3c802ff60 msgr2=0x7fb3c8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:07:32.438 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:32.436+0000 7fb3df59e700 1 --2- 172.21.15.178:0/1002692314 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb3c802ff60 0x7fb3c8032420 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7fb3d00096a0 tx=0x7fb3d0006b40).stop 2024-02-11T17:07:32.438 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:32.436+0000 7fb3df59e700 1 -- 172.21.15.178:0/1002692314 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3e00f51a0 msgr2=0x7fb3e01053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:07:32.438 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:32.436+0000 7fb3df59e700 1 --2- 172.21.15.178:0/1002692314 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3e00f51a0 0x7fb3e01053b0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7fb3d4013510 tx=0x7fb3d400b040).stop 2024-02-11T17:07:32.438 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:32.436+0000 7fb3df59e700 1 -- 172.21.15.178:0/1002692314 shutdown_connections 2024-02-11T17:07:32.438 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:32.436+0000 7fb3df59e700 1 --2- 172.21.15.178:0/1002692314 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb3c802ff60 0x7fb3c8032420 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:07:32.438 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:32.436+0000 7fb3df59e700 1 --2- 172.21.15.178:0/1002692314 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3e00f51a0 0x7fb3e01053b0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:07:32.438 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:32.436+0000 7fb3df59e700 1 -- 172.21.15.178:0/1002692314 >> 172.21.15.178:0/1002692314 conn(0x7fb3e00f09a0 msgr2=0x7fb3e00f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:07:32.438 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:32.436+0000 7fb3df59e700 1 -- 172.21.15.178:0/1002692314 shutdown_connections 2024-02-11T17:07:32.438 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:32.436+0000 7fb3df59e700 1 -- 172.21.15.178:0/1002692314 wait complete. 2024-02-11T17:07:32.439 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:07:32.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:32 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:07:31.198721+0000 mgr.smithi037.eofnaq (mgr.14162) 230 : cluster [DBG] pgmap v218: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:07:33.684 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:33 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:07:32.435321+0000 mon.smithi037 (mon.0) 249 : audit [DBG] from='client.? 172.21.15.178:0/1002692314' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:07:34.084 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:07:34.084 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:07:34.684 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:34 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cephadm 2024-02-11T17:07:32.972814+0000 mgr.smithi037.eofnaq (mgr.14162) 231 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi037 ... 2024-02-11T17:07:34.684 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:34 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: Verifying port 9100 ... 2024-02-11T17:07:34.684 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:34 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: Non-zero exit code 1 from systemctl start ceph-c65cd888-c8fe-11ee-95b9-87774f69a715@node-exporter.smithi037 2024-02-11T17:07:34.684 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:34 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: systemctl: stderr Job for ceph-c65cd888-c8fe-11ee-95b9-87774f69a715@node-exporter.smithi037.service failed because a timeout was exceeded. 2024-02-11T17:07:34.684 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:34 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: systemctl: stderr See "systemctl status ceph-c65cd888-c8fe-11ee-95b9-87774f69a715@node-exporter.smithi037.service" and "journalctl -xe" for details. 2024-02-11T17:07:34.684 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:34 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: Traceback (most recent call last): 2024-02-11T17:07:34.684 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:34 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: File "/var/lib/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-11T17:07:34.684 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:34 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: main() 2024-02-11T17:07:34.684 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:34 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: File "/var/lib/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-11T17:07:34.684 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:34 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: r = ctx.func(ctx) 2024-02-11T17:07:34.685 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:34 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: File "/var/lib/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-11T17:07:34.685 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:34 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: return func(ctx) 2024-02-11T17:07:34.685 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:34 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: File "/var/lib/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-02-11T17:07:34.685 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:34 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: ports=daemon_ports) 2024-02-11T17:07:34.685 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:34 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: File "/var/lib/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2024-02-11T17:07:34.685 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:34 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: c, osd_fsid=osd_fsid, ports=ports) 2024-02-11T17:07:34.685 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:34 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: File "/var/lib/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2024-02-11T17:07:34.685 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:34 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-02-11T17:07:34.685 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:34 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: File "/var/lib/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-02-11T17:07:34.685 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:34 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-02-11T17:07:34.685 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:34 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: RuntimeError: Failed command: systemctl start ceph-c65cd888-c8fe-11ee-95b9-87774f69a715@node-exporter.smithi037 2024-02-11T17:07:34.685 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:34 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: Traceback (most recent call last): 2024-02-11T17:07:34.685 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:34 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-02-11T17:07:34.685 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:34 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: yield (conn, connr) 2024-02-11T17:07:34.685 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:34 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-02-11T17:07:34.686 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:34 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: code, '\n'.join(err))) 2024-02-11T17:07:34.686 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:34 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi037 ... 2024-02-11T17:07:34.686 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:34 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: Verifying port 9100 ... 2024-02-11T17:07:34.686 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:34 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: Non-zero exit code 1 from systemctl start ceph-c65cd888-c8fe-11ee-95b9-87774f69a715@node-exporter.smithi037 2024-02-11T17:07:34.686 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:34 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: systemctl: stderr Job for ceph-c65cd888-c8fe-11ee-95b9-87774f69a715@node-exporter.smithi037.service failed because a timeout was exceeded. 2024-02-11T17:07:34.686 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:34 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: systemctl: stderr See "systemctl status ceph-c65cd888-c8fe-11ee-95b9-87774f69a715@node-exporter.smithi037.service" and "journalctl -xe" for details. 2024-02-11T17:07:34.686 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:34 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: Traceback (most recent call last): 2024-02-11T17:07:34.686 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:34 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: File "/var/lib/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-11T17:07:34.686 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:34 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: main() 2024-02-11T17:07:34.686 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:34 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: File "/var/lib/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-11T17:07:34.686 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:34 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: r = ctx.func(ctx) 2024-02-11T17:07:34.686 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:34 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: File "/var/lib/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-11T17:07:34.686 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:34 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: return func(ctx) 2024-02-11T17:07:34.686 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:34 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: File "/var/lib/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-02-11T17:07:34.686 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:34 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: ports=daemon_ports) 2024-02-11T17:07:34.687 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:34 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: File "/var/lib/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2024-02-11T17:07:34.687 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:34 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: c, osd_fsid=osd_fsid, ports=ports) 2024-02-11T17:07:34.687 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:34 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: File "/var/lib/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2024-02-11T17:07:34.687 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:34 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-02-11T17:07:34.687 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:34 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: File "/var/lib/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-02-11T17:07:34.687 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:34 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-02-11T17:07:34.687 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:34 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: RuntimeError: Failed command: systemctl start ceph-c65cd888-c8fe-11ee-95b9-87774f69a715@node-exporter.smithi037 2024-02-11T17:07:34.687 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:34 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cephadm 2024-02-11T17:07:32.978486+0000 mgr.smithi037.eofnaq (mgr.14162) 232 : cephadm [INF] Deploying daemon prometheus.smithi037 on smithi037 2024-02-11T17:07:34.687 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:34 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:07:33.198980+0000 mgr.smithi037.eofnaq (mgr.14162) 233 : cluster [DBG] pgmap v219: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:07:36.138 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:36.121+0000 7fa897fff700 1 -- 172.21.15.178:0/2653801821 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa8980601e0 msgr2=0x7fa8980f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:07:36.138 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:36.121+0000 7fa897fff700 1 --2- 172.21.15.178:0/2653801821 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa8980601e0 0x7fa8980f8f20 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7fa888009fb0 tx=0x7fa888009b30).stop 2024-02-11T17:07:36.138 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:36.121+0000 7fa897fff700 1 -- 172.21.15.178:0/2653801821 shutdown_connections 2024-02-11T17:07:36.138 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:36.121+0000 7fa897fff700 1 --2- 172.21.15.178:0/2653801821 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa8980601e0 0x7fa8980f8f20 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:07:36.139 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:36.121+0000 7fa897fff700 1 -- 172.21.15.178:0/2653801821 >> 172.21.15.178:0/2653801821 conn(0x7fa8980f09a0 msgr2=0x7fa8980f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:07:36.139 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:36.122+0000 7fa897fff700 1 -- 172.21.15.178:0/2653801821 shutdown_connections 2024-02-11T17:07:36.139 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:36.122+0000 7fa897fff700 1 -- 172.21.15.178:0/2653801821 wait complete. 2024-02-11T17:07:36.139 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:36.122+0000 7fa897fff700 1 Processor -- start 2024-02-11T17:07:36.139 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:36.122+0000 7fa897fff700 1 -- start start 2024-02-11T17:07:36.139 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:36.122+0000 7fa897fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa8980f6960 0x7fa8980f4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:07:36.139 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:36.122+0000 7fa897fff700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8980f6d80 con 0x7fa8980f6960 2024-02-11T17:07:36.139 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:36.123+0000 7fa896ffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa8980f6960 0x7fa8980f4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:07:36.139 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:36.123+0000 7fa896ffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa8980f6960 0x7fa8980f4e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:57472/0 (socket says 172.21.15.178:57472) 2024-02-11T17:07:36.139 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:36.123+0000 7fa896ffd700 1 -- 172.21.15.178:0/1571217009 learned_addr learned my addr 172.21.15.178:0/1571217009 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:07:36.139 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:36.123+0000 7fa896ffd700 1 -- 172.21.15.178:0/1571217009 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa888005040 con 0x7fa8980f6960 2024-02-11T17:07:36.139 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:36.123+0000 7fa896ffd700 1 --2- 172.21.15.178:0/1571217009 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa8980f6960 0x7fa8980f4e20 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7fa888009f80 tx=0x7fa888009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:07:36.139 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:36.123+0000 7fa87f7fe700 1 -- 172.21.15.178:0/1571217009 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8880136c0 con 0x7fa8980f6960 2024-02-11T17:07:36.140 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:36.123+0000 7fa897fff700 1 -- 172.21.15.178:0/1571217009 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8980f5310 con 0x7fa8980f6960 2024-02-11T17:07:36.140 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:36.123+0000 7fa87f7fe700 1 -- 172.21.15.178:0/1571217009 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa888013e70 con 0x7fa8980f6960 2024-02-11T17:07:36.140 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:36.123+0000 7fa897fff700 1 -- 172.21.15.178:0/1571217009 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8980f5800 con 0x7fa8980f6960 2024-02-11T17:07:36.140 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:36.123+0000 7fa87f7fe700 1 -- 172.21.15.178:0/1571217009 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa888014420 con 0x7fa8980f6960 2024-02-11T17:07:36.140 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:36.124+0000 7fa87f7fe700 1 -- 172.21.15.178:0/1571217009 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa888025060 con 0x7fa8980f6960 2024-02-11T17:07:36.140 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:36.124+0000 7fa897fff700 1 -- 172.21.15.178:0/1571217009 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa898045ad0 con 0x7fa8980f6960 2024-02-11T17:07:36.140 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:36.124+0000 7fa87f7fe700 1 --2- 172.21.15.178:0/1571217009 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fa88002ff60 0x7fa880032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:07:36.140 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:36.124+0000 7fa87f7fe700 1 -- 172.21.15.178:0/1571217009 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa888040f10 con 0x7fa8980f6960 2024-02-11T17:07:36.140 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:36.124+0000 7fa8967fc700 1 --2- 172.21.15.178:0/1571217009 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fa88002ff60 0x7fa880032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:07:36.140 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:36.125+0000 7fa8967fc700 1 --2- 172.21.15.178:0/1571217009 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fa88002ff60 0x7fa880032420 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fa88c0096a0 tx=0x7fa88c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:07:36.140 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:36.129+0000 7fa87f7fe700 1 -- 172.21.15.178:0/1571217009 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa88801cd50 con 0x7fa8980f6960 2024-02-11T17:07:36.337 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:36.335+0000 7fa897fff700 1 -- 172.21.15.178:0/1571217009 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa89810c7f0 con 0x7fa8980f6960 2024-02-11T17:07:36.338 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:36.336+0000 7fa87f7fe700 1 -- 172.21.15.178:0/1571217009 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa888004490 con 0x7fa8980f6960 2024-02-11T17:07:36.338 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:07:36.338 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:07:36.339 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:36.338+0000 7fa897fff700 1 -- 172.21.15.178:0/1571217009 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fa88002ff60 msgr2=0x7fa880032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:07:36.339 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:36.338+0000 7fa897fff700 1 --2- 172.21.15.178:0/1571217009 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fa88002ff60 0x7fa880032420 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fa88c0096a0 tx=0x7fa88c006b40).stop 2024-02-11T17:07:36.340 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:36.338+0000 7fa897fff700 1 -- 172.21.15.178:0/1571217009 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa8980f6960 msgr2=0x7fa8980f4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:07:36.340 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:36.338+0000 7fa897fff700 1 --2- 172.21.15.178:0/1571217009 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa8980f6960 0x7fa8980f4e20 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7fa888009f80 tx=0x7fa888009b30).stop 2024-02-11T17:07:36.340 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:36.338+0000 7fa897fff700 1 -- 172.21.15.178:0/1571217009 shutdown_connections 2024-02-11T17:07:36.340 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:36.338+0000 7fa897fff700 1 --2- 172.21.15.178:0/1571217009 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fa88002ff60 0x7fa880032420 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:07:36.340 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:36.338+0000 7fa897fff700 1 --2- 172.21.15.178:0/1571217009 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa8980f6960 0x7fa8980f4e20 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:07:36.340 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:36.338+0000 7fa897fff700 1 -- 172.21.15.178:0/1571217009 >> 172.21.15.178:0/1571217009 conn(0x7fa8980f09a0 msgr2=0x7fa8980f14a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:07:36.340 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:36.338+0000 7fa897fff700 1 -- 172.21.15.178:0/1571217009 shutdown_connections 2024-02-11T17:07:36.340 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:36.338+0000 7fa897fff700 1 -- 172.21.15.178:0/1571217009 wait complete. 2024-02-11T17:07:36.340 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:07:36.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:36 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:07:35.199589+0000 mgr.smithi037.eofnaq (mgr.14162) 234 : cluster [DBG] pgmap v220: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:07:37.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:37 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:07:36.337483+0000 mon.smithi037 (mon.0) 250 : audit [DBG] from='client.? 172.21.15.178:0/1571217009' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:07:38.032 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:07:38.033 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:07:38.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:38 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:07:37.199946+0000 mgr.smithi037.eofnaq (mgr.14162) 235 : cluster [DBG] pgmap v221: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:07:39.989 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:39.987+0000 7effafac2700 1 -- 172.21.15.178:0/652221934 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7effa80f4740 msgr2=0x7effa80f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:07:39.990 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:39.987+0000 7effafac2700 1 --2- 172.21.15.178:0/652221934 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7effa80f4740 0x7effa80f4b60 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7eff9c009fb0 tx=0x7eff9c009b30).stop 2024-02-11T17:07:39.990 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:39.987+0000 7effafac2700 1 -- 172.21.15.178:0/652221934 shutdown_connections 2024-02-11T17:07:39.990 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:39.987+0000 7effafac2700 1 --2- 172.21.15.178:0/652221934 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7effa80f4740 0x7effa80f4b60 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:07:39.990 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:39.987+0000 7effafac2700 1 -- 172.21.15.178:0/652221934 >> 172.21.15.178:0/652221934 conn(0x7effa80f0260 msgr2=0x7effa80f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:07:39.990 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:39.987+0000 7effafac2700 1 -- 172.21.15.178:0/652221934 shutdown_connections 2024-02-11T17:07:39.990 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:39.987+0000 7effafac2700 1 -- 172.21.15.178:0/652221934 wait complete. 2024-02-11T17:07:39.990 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:39.988+0000 7effafac2700 1 Processor -- start 2024-02-11T17:07:39.990 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:39.988+0000 7effafac2700 1 -- start start 2024-02-11T17:07:39.990 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:39.988+0000 7effafac2700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7effa80f4740 0x7effa81066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:07:39.990 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:39.988+0000 7effafac2700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa8106bc0 con 0x7effa80f4740 2024-02-11T17:07:39.992 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:39.988+0000 7effad85e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7effa80f4740 0x7effa81066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:07:39.992 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:39.988+0000 7effad85e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7effa80f4740 0x7effa81066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:44076/0 (socket says 172.21.15.178:44076) 2024-02-11T17:07:39.992 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:39.988+0000 7effad85e700 1 -- 172.21.15.178:0/105453834 learned_addr learned my addr 172.21.15.178:0/105453834 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:07:39.992 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:39.989+0000 7effad85e700 1 -- 172.21.15.178:0/105453834 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff9c005040 con 0x7effa80f4740 2024-02-11T17:07:39.992 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:39.989+0000 7effad85e700 1 --2- 172.21.15.178:0/105453834 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7effa80f4740 0x7effa81066d0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7eff9c013510 tx=0x7eff9c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:07:39.992 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:39.989+0000 7eff9a7fc700 1 -- 172.21.15.178:0/105453834 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff9c009320 con 0x7effa80f4740 2024-02-11T17:07:39.992 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:39.989+0000 7effafac2700 1 -- 172.21.15.178:0/105453834 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effa8106dc0 con 0x7effa80f4740 2024-02-11T17:07:39.992 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:39.989+0000 7eff9a7fc700 1 -- 172.21.15.178:0/105453834 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7eff9c013f00 con 0x7effa80f4740 2024-02-11T17:07:39.992 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:39.989+0000 7eff9a7fc700 1 -- 172.21.15.178:0/105453834 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff9c014d90 con 0x7effa80f4740 2024-02-11T17:07:39.992 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:39.989+0000 7effafac2700 1 -- 172.21.15.178:0/105453834 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effa8107250 con 0x7effa80f4740 2024-02-11T17:07:39.993 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:39.990+0000 7eff9a7fc700 1 -- 172.21.15.178:0/105453834 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7eff9c0095f0 con 0x7effa80f4740 2024-02-11T17:07:39.993 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:39.990+0000 7effafac2700 1 -- 172.21.15.178:0/105453834 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effa8045ad0 con 0x7effa80f4740 2024-02-11T17:07:39.993 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:39.990+0000 7eff9a7fc700 1 --2- 172.21.15.178:0/105453834 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7eff9402ff60 0x7eff94032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:07:39.993 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:39.990+0000 7eff9a7fc700 1 -- 172.21.15.178:0/105453834 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7eff9c040fe0 con 0x7effa80f4740 2024-02-11T17:07:39.993 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:39.990+0000 7effad05d700 1 --2- 172.21.15.178:0/105453834 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7eff9402ff60 0x7eff94032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:07:39.993 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:39.991+0000 7effad05d700 1 --2- 172.21.15.178:0/105453834 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7eff9402ff60 0x7eff94032420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7effa40096a0 tx=0x7effa4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:07:39.996 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:39.995+0000 7eff9a7fc700 1 -- 172.21.15.178:0/105453834 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7eff9c01d7f0 con 0x7effa80f4740 2024-02-11T17:07:40.199 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:40.197+0000 7effafac2700 1 -- 172.21.15.178:0/105453834 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7effa8108100 con 0x7effa80f4740 2024-02-11T17:07:40.200 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:40.198+0000 7eff9a7fc700 1 -- 172.21.15.178:0/105453834 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7eff9c010230 con 0x7effa80f4740 2024-02-11T17:07:40.200 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:07:40.200 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:07:40.201 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:40.200+0000 7effafac2700 1 -- 172.21.15.178:0/105453834 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7eff9402ff60 msgr2=0x7eff94032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:07:40.201 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:40.200+0000 7effafac2700 1 --2- 172.21.15.178:0/105453834 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7eff9402ff60 0x7eff94032420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7effa40096a0 tx=0x7effa4006b40).stop 2024-02-11T17:07:40.201 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:40.200+0000 7effafac2700 1 -- 172.21.15.178:0/105453834 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7effa80f4740 msgr2=0x7effa81066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:07:40.202 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:40.200+0000 7effafac2700 1 --2- 172.21.15.178:0/105453834 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7effa80f4740 0x7effa81066d0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7eff9c013510 tx=0x7eff9c00b040).stop 2024-02-11T17:07:40.202 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:40.200+0000 7effafac2700 1 -- 172.21.15.178:0/105453834 shutdown_connections 2024-02-11T17:07:40.202 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:40.200+0000 7effafac2700 1 --2- 172.21.15.178:0/105453834 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7eff9402ff60 0x7eff94032420 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:07:40.202 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:40.200+0000 7effafac2700 1 --2- 172.21.15.178:0/105453834 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7effa80f4740 0x7effa81066d0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:07:40.202 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:40.200+0000 7effafac2700 1 -- 172.21.15.178:0/105453834 >> 172.21.15.178:0/105453834 conn(0x7effa80f0260 msgr2=0x7effa80f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:07:40.202 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:40.200+0000 7effafac2700 1 -- 172.21.15.178:0/105453834 shutdown_connections 2024-02-11T17:07:40.202 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:40.200+0000 7effafac2700 1 -- 172.21.15.178:0/105453834 wait complete. 2024-02-11T17:07:40.202 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:07:40.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:40 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:07:39.200326+0000 mgr.smithi037.eofnaq (mgr.14162) 236 : cluster [DBG] pgmap v222: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:07:40.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:40 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:07:40.199323+0000 mon.smithi037 (mon.0) 251 : audit [DBG] from='client.? 172.21.15.178:0/105453834' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:07:41.890 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:07:41.891 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:07:42.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:42 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:07:41.200709+0000 mgr.smithi037.eofnaq (mgr.14162) 237 : cluster [DBG] pgmap v223: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:07:43.857 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:43.855+0000 7f00c93c6700 1 -- 172.21.15.178:0/2960563838 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00c40f6510 msgr2=0x7f00c40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:07:43.857 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:43.855+0000 7f00c93c6700 1 --2- 172.21.15.178:0/2960563838 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00c40f6510 0x7f00c40f6930 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f00b8009fb0 tx=0x7f00b8009b30).stop 2024-02-11T17:07:43.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:43.855+0000 7f00c93c6700 1 -- 172.21.15.178:0/2960563838 shutdown_connections 2024-02-11T17:07:43.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:43.855+0000 7f00c93c6700 1 --2- 172.21.15.178:0/2960563838 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00c40f6510 0x7f00c40f6930 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:07:43.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:43.855+0000 7f00c93c6700 1 -- 172.21.15.178:0/2960563838 >> 172.21.15.178:0/2960563838 conn(0x7f00c40f1c70 msgr2=0x7f00c40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:07:43.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:43.855+0000 7f00c93c6700 1 -- 172.21.15.178:0/2960563838 shutdown_connections 2024-02-11T17:07:43.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:43.855+0000 7f00c93c6700 1 -- 172.21.15.178:0/2960563838 wait complete. 2024-02-11T17:07:43.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:43.856+0000 7f00c93c6700 1 Processor -- start 2024-02-11T17:07:43.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:43.856+0000 7f00c93c6700 1 -- start start 2024-02-11T17:07:43.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:43.856+0000 7f00c93c6700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00c40f6510 0x7f00c4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:07:43.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:43.856+0000 7f00c93c6700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00c4106c20 con 0x7f00c40f6510 2024-02-11T17:07:43.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:43.856+0000 7f00c2ffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00c40f6510 0x7f00c4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:07:43.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:43.856+0000 7f00c2ffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00c40f6510 0x7f00c4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:44096/0 (socket says 172.21.15.178:44096) 2024-02-11T17:07:43.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:43.856+0000 7f00c2ffd700 1 -- 172.21.15.178:0/3058313488 learned_addr learned my addr 172.21.15.178:0/3058313488 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:07:43.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:43.856+0000 7f00c2ffd700 1 -- 172.21.15.178:0/3058313488 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00b8005040 con 0x7f00c40f6510 2024-02-11T17:07:43.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:43.856+0000 7f00c2ffd700 1 --2- 172.21.15.178:0/3058313488 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00c40f6510 0x7f00c4106730 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f00b8013510 tx=0x7f00b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:07:43.859 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:43.857+0000 7f00ab7fe700 1 -- 172.21.15.178:0/3058313488 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f00b8009320 con 0x7f00c40f6510 2024-02-11T17:07:43.859 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:43.857+0000 7f00c93c6700 1 -- 172.21.15.178:0/3058313488 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00c4106e20 con 0x7f00c40f6510 2024-02-11T17:07:43.859 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:43.857+0000 7f00ab7fe700 1 -- 172.21.15.178:0/3058313488 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f00b8013f00 con 0x7f00c40f6510 2024-02-11T17:07:43.859 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:43.857+0000 7f00ab7fe700 1 -- 172.21.15.178:0/3058313488 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f00b8014d90 con 0x7f00c40f6510 2024-02-11T17:07:43.859 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:43.857+0000 7f00c93c6700 1 -- 172.21.15.178:0/3058313488 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00c41072b0 con 0x7f00c40f6510 2024-02-11T17:07:43.860 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:43.858+0000 7f00ab7fe700 1 -- 172.21.15.178:0/3058313488 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f00b80095f0 con 0x7f00c40f6510 2024-02-11T17:07:43.860 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:43.858+0000 7f00c93c6700 1 -- 172.21.15.178:0/3058313488 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00c4045ad0 con 0x7f00c40f6510 2024-02-11T17:07:43.860 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:43.858+0000 7f00ab7fe700 1 --2- 172.21.15.178:0/3058313488 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f00ac02ff60 0x7f00ac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:07:43.860 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:43.858+0000 7f00ab7fe700 1 -- 172.21.15.178:0/3058313488 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f00b8041090 con 0x7f00c40f6510 2024-02-11T17:07:43.860 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:43.858+0000 7f00c27fc700 1 --2- 172.21.15.178:0/3058313488 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f00ac02ff60 0x7f00ac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:07:43.860 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:43.858+0000 7f00c27fc700 1 --2- 172.21.15.178:0/3058313488 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f00ac02ff60 0x7f00ac032420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f00b40096a0 tx=0x7f00b4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:07:43.863 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:43.863+0000 7f00ab7fe700 1 -- 172.21.15.178:0/3058313488 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f00b8014420 con 0x7f00c40f6510 2024-02-11T17:07:44.066 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:44.066+0000 7f00c93c6700 1 -- 172.21.15.178:0/3058313488 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f00c40fbf60 con 0x7f00c40f6510 2024-02-11T17:07:44.067 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:44.066+0000 7f00ab7fe700 1 -- 172.21.15.178:0/3058313488 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f00b801d7f0 con 0x7f00c40f6510 2024-02-11T17:07:44.067 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:07:44.067 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:07:44.069 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:44.068+0000 7f00c93c6700 1 -- 172.21.15.178:0/3058313488 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f00ac02ff60 msgr2=0x7f00ac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:07:44.069 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:44.068+0000 7f00c93c6700 1 --2- 172.21.15.178:0/3058313488 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f00ac02ff60 0x7f00ac032420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f00b40096a0 tx=0x7f00b4006b40).stop 2024-02-11T17:07:44.069 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:44.068+0000 7f00c93c6700 1 -- 172.21.15.178:0/3058313488 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00c40f6510 msgr2=0x7f00c4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:07:44.070 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:44.068+0000 7f00c93c6700 1 --2- 172.21.15.178:0/3058313488 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00c40f6510 0x7f00c4106730 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f00b8013510 tx=0x7f00b800b040).stop 2024-02-11T17:07:44.070 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:44.068+0000 7f00c93c6700 1 -- 172.21.15.178:0/3058313488 shutdown_connections 2024-02-11T17:07:44.070 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:44.068+0000 7f00c93c6700 1 --2- 172.21.15.178:0/3058313488 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f00ac02ff60 0x7f00ac032420 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:07:44.070 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:44.068+0000 7f00c93c6700 1 --2- 172.21.15.178:0/3058313488 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f00c40f6510 0x7f00c4106730 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:07:44.070 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:44.068+0000 7f00c93c6700 1 -- 172.21.15.178:0/3058313488 >> 172.21.15.178:0/3058313488 conn(0x7f00c40f1c70 msgr2=0x7f00c40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:07:44.070 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:44.068+0000 7f00c93c6700 1 -- 172.21.15.178:0/3058313488 shutdown_connections 2024-02-11T17:07:44.070 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:44.068+0000 7f00c93c6700 1 -- 172.21.15.178:0/3058313488 wait complete. 2024-02-11T17:07:44.070 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:07:44.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:44 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:07:43.200964+0000 mgr.smithi037.eofnaq (mgr.14162) 238 : cluster [DBG] pgmap v224: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:07:44.684 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:44 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:07:44.066874+0000 mon.smithi037 (mon.0) 252 : audit [DBG] from='client.? 172.21.15.178:0/3058313488' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:07:45.771 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:07:45.772 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:07:46.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:46 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:07:45.201322+0000 mgr.smithi037.eofnaq (mgr.14162) 239 : cluster [DBG] pgmap v225: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:07:47.710 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:47.707+0000 7fb91d28f700 1 -- 172.21.15.178:0/1948203782 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb9180f51a0 msgr2=0x7fb9180f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:07:47.710 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:47.707+0000 7fb91d28f700 1 --2- 172.21.15.178:0/1948203782 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb9180f51a0 0x7fb9180f55c0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7fb90c009fb0 tx=0x7fb90c009b30).stop 2024-02-11T17:07:47.710 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:47.708+0000 7fb91d28f700 1 -- 172.21.15.178:0/1948203782 shutdown_connections 2024-02-11T17:07:47.710 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:47.708+0000 7fb91d28f700 1 --2- 172.21.15.178:0/1948203782 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb9180f51a0 0x7fb9180f55c0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:07:47.710 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:47.708+0000 7fb91d28f700 1 -- 172.21.15.178:0/1948203782 >> 172.21.15.178:0/1948203782 conn(0x7fb9180f09a0 msgr2=0x7fb9180f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:07:47.710 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:47.708+0000 7fb91d28f700 1 -- 172.21.15.178:0/1948203782 shutdown_connections 2024-02-11T17:07:47.710 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:47.708+0000 7fb91d28f700 1 -- 172.21.15.178:0/1948203782 wait complete. 2024-02-11T17:07:47.711 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:47.708+0000 7fb91d28f700 1 Processor -- start 2024-02-11T17:07:47.711 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:47.708+0000 7fb91d28f700 1 -- start start 2024-02-11T17:07:47.711 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:47.708+0000 7fb91d28f700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb9180f51a0 0x7fb9181064c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:07:47.711 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:47.708+0000 7fb91d28f700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9181069b0 con 0x7fb9180f51a0 2024-02-11T17:07:47.711 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:47.709+0000 7fb917fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb9180f51a0 0x7fb9181064c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:07:47.711 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:47.709+0000 7fb917fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb9180f51a0 0x7fb9181064c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:44110/0 (socket says 172.21.15.178:44110) 2024-02-11T17:07:47.711 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:47.709+0000 7fb917fff700 1 -- 172.21.15.178:0/3273640390 learned_addr learned my addr 172.21.15.178:0/3273640390 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:07:47.711 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:47.709+0000 7fb917fff700 1 -- 172.21.15.178:0/3273640390 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb90c005040 con 0x7fb9180f51a0 2024-02-11T17:07:47.711 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:47.709+0000 7fb917fff700 1 --2- 172.21.15.178:0/3273640390 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb9180f51a0 0x7fb9181064c0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7fb90c013510 tx=0x7fb90c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:07:47.711 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:47.709+0000 7fb914ff9700 1 -- 172.21.15.178:0/3273640390 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb90c009320 con 0x7fb9180f51a0 2024-02-11T17:07:47.711 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:47.709+0000 7fb91d28f700 1 -- 172.21.15.178:0/3273640390 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb918106bb0 con 0x7fb9180f51a0 2024-02-11T17:07:47.711 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:47.709+0000 7fb91d28f700 1 -- 172.21.15.178:0/3273640390 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb918107040 con 0x7fb9180f51a0 2024-02-11T17:07:47.711 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:47.709+0000 7fb914ff9700 1 -- 172.21.15.178:0/3273640390 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb90c013f00 con 0x7fb9180f51a0 2024-02-11T17:07:47.711 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:47.709+0000 7fb914ff9700 1 -- 172.21.15.178:0/3273640390 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb90c014d90 con 0x7fb9180f51a0 2024-02-11T17:07:47.712 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:47.710+0000 7fb91d28f700 1 -- 172.21.15.178:0/3273640390 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb918045ad0 con 0x7fb9180f51a0 2024-02-11T17:07:47.712 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:47.710+0000 7fb914ff9700 1 -- 172.21.15.178:0/3273640390 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb90c0095f0 con 0x7fb9180f51a0 2024-02-11T17:07:47.713 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:47.710+0000 7fb914ff9700 1 --2- 172.21.15.178:0/3273640390 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb90002ff60 0x7fb900032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:07:47.713 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:47.710+0000 7fb914ff9700 1 -- 172.21.15.178:0/3273640390 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb90c0410c0 con 0x7fb9180f51a0 2024-02-11T17:07:47.713 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:47.711+0000 7fb9177fe700 1 --2- 172.21.15.178:0/3273640390 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb90002ff60 0x7fb900032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:07:47.713 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:47.711+0000 7fb9177fe700 1 --2- 172.21.15.178:0/3273640390 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb90002ff60 0x7fb900032420 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fb9080096a0 tx=0x7fb908006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:07:47.715 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:47.715+0000 7fb914ff9700 1 -- 172.21.15.178:0/3273640390 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb90c014420 con 0x7fb9180f51a0 2024-02-11T17:07:47.918 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:47.917+0000 7fb91d28f700 1 -- 172.21.15.178:0/3273640390 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb918107ef0 con 0x7fb9180f51a0 2024-02-11T17:07:47.918 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:47.918+0000 7fb914ff9700 1 -- 172.21.15.178:0/3273640390 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb90c012980 con 0x7fb9180f51a0 2024-02-11T17:07:47.918 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:07:47.919 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:07:47.920 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:47.919+0000 7fb91d28f700 1 -- 172.21.15.178:0/3273640390 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb90002ff60 msgr2=0x7fb900032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:07:47.920 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:47.919+0000 7fb91d28f700 1 --2- 172.21.15.178:0/3273640390 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb90002ff60 0x7fb900032420 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fb9080096a0 tx=0x7fb908006b40).stop 2024-02-11T17:07:47.920 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:47.919+0000 7fb91d28f700 1 -- 172.21.15.178:0/3273640390 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb9180f51a0 msgr2=0x7fb9181064c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:07:47.920 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:47.919+0000 7fb91d28f700 1 --2- 172.21.15.178:0/3273640390 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb9180f51a0 0x7fb9181064c0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7fb90c013510 tx=0x7fb90c00b040).stop 2024-02-11T17:07:47.920 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:47.919+0000 7fb91d28f700 1 -- 172.21.15.178:0/3273640390 shutdown_connections 2024-02-11T17:07:47.920 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:47.919+0000 7fb91d28f700 1 --2- 172.21.15.178:0/3273640390 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb90002ff60 0x7fb900032420 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:07:47.921 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:47.919+0000 7fb91d28f700 1 --2- 172.21.15.178:0/3273640390 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb9180f51a0 0x7fb9181064c0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:07:47.921 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:47.919+0000 7fb91d28f700 1 -- 172.21.15.178:0/3273640390 >> 172.21.15.178:0/3273640390 conn(0x7fb9180f09a0 msgr2=0x7fb9180f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:07:47.921 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:47.919+0000 7fb91d28f700 1 -- 172.21.15.178:0/3273640390 shutdown_connections 2024-02-11T17:07:47.921 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:47.919+0000 7fb91d28f700 1 -- 172.21.15.178:0/3273640390 wait complete. 2024-02-11T17:07:47.921 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:07:48.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:48 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:07:47.201664+0000 mgr.smithi037.eofnaq (mgr.14162) 240 : cluster [DBG] pgmap v226: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:07:48.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:48 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:07:47.918178+0000 mon.smithi037 (mon.0) 253 : audit [DBG] from='client.? 172.21.15.178:0/3273640390' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:07:49.514 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:07:49.515 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:07:50.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:50 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:07:49.201996+0000 mgr.smithi037.eofnaq (mgr.14162) 241 : cluster [DBG] pgmap v227: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:07:51.371 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:51.367+0000 7fccc77f7700 1 -- 172.21.15.178:0/3166927912 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fccc00f6510 msgr2=0x7fccc00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:07:51.371 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:51.367+0000 7fccc77f7700 1 --2- 172.21.15.178:0/3166927912 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fccc00f6510 0x7fccc00f6930 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7fccb0009fb0 tx=0x7fccb0009b30).stop 2024-02-11T17:07:51.371 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:51.367+0000 7fccc77f7700 1 -- 172.21.15.178:0/3166927912 shutdown_connections 2024-02-11T17:07:51.371 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:51.367+0000 7fccc77f7700 1 --2- 172.21.15.178:0/3166927912 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fccc00f6510 0x7fccc00f6930 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:07:51.371 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:51.367+0000 7fccc77f7700 1 -- 172.21.15.178:0/3166927912 >> 172.21.15.178:0/3166927912 conn(0x7fccc00f1c70 msgr2=0x7fccc00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:07:51.371 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:51.367+0000 7fccc77f7700 1 -- 172.21.15.178:0/3166927912 shutdown_connections 2024-02-11T17:07:51.371 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:51.367+0000 7fccc77f7700 1 -- 172.21.15.178:0/3166927912 wait complete. 2024-02-11T17:07:51.372 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:51.368+0000 7fccc77f7700 1 Processor -- start 2024-02-11T17:07:51.372 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:51.368+0000 7fccc77f7700 1 -- start start 2024-02-11T17:07:51.372 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:51.368+0000 7fccc77f7700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fccc00f6510 0x7fccc0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:07:51.372 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:51.368+0000 7fccc77f7700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccc0106c20 con 0x7fccc00f6510 2024-02-11T17:07:51.372 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:51.368+0000 7fccc5593700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fccc00f6510 0x7fccc0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:07:51.372 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:51.368+0000 7fccc5593700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fccc00f6510 0x7fccc0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:47530/0 (socket says 172.21.15.178:47530) 2024-02-11T17:07:51.372 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:51.368+0000 7fccc5593700 1 -- 172.21.15.178:0/1209303725 learned_addr learned my addr 172.21.15.178:0/1209303725 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:07:51.372 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:51.369+0000 7fccc5593700 1 -- 172.21.15.178:0/1209303725 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccb0005040 con 0x7fccc00f6510 2024-02-11T17:07:51.372 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:51.369+0000 7fccc5593700 1 --2- 172.21.15.178:0/1209303725 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fccc00f6510 0x7fccc0106730 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7fccb0013510 tx=0x7fccb000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:07:51.372 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:51.369+0000 7fccb5ffb700 1 -- 172.21.15.178:0/1209303725 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fccb0009320 con 0x7fccc00f6510 2024-02-11T17:07:51.372 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:51.369+0000 7fccc77f7700 1 -- 172.21.15.178:0/1209303725 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccc0106e20 con 0x7fccc00f6510 2024-02-11T17:07:51.372 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:51.369+0000 7fccb5ffb700 1 -- 172.21.15.178:0/1209303725 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fccb0013f00 con 0x7fccc00f6510 2024-02-11T17:07:51.372 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:51.369+0000 7fccb5ffb700 1 -- 172.21.15.178:0/1209303725 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fccb0014d90 con 0x7fccc00f6510 2024-02-11T17:07:51.373 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:51.369+0000 7fccc77f7700 1 -- 172.21.15.178:0/1209303725 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccc01072b0 con 0x7fccc00f6510 2024-02-11T17:07:51.373 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:51.370+0000 7fccb5ffb700 1 -- 172.21.15.178:0/1209303725 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fccb00263a0 con 0x7fccc00f6510 2024-02-11T17:07:51.373 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:51.370+0000 7fccb5ffb700 1 --2- 172.21.15.178:0/1209303725 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fccac02ff60 0x7fccac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:07:51.374 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:51.370+0000 7fccc77f7700 1 -- 172.21.15.178:0/1209303725 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccc0045ad0 con 0x7fccc00f6510 2024-02-11T17:07:51.374 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:51.370+0000 7fccb5ffb700 1 -- 172.21.15.178:0/1209303725 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fccb00412f0 con 0x7fccc00f6510 2024-02-11T17:07:51.374 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:51.370+0000 7fccc4d92700 1 --2- 172.21.15.178:0/1209303725 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fccac02ff60 0x7fccac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:07:51.374 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:51.371+0000 7fccc4d92700 1 --2- 172.21.15.178:0/1209303725 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fccac02ff60 0x7fccac032420 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7fccbc0096a0 tx=0x7fccbc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:07:51.377 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:51.375+0000 7fccb5ffb700 1 -- 172.21.15.178:0/1209303725 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fccb0012920 con 0x7fccc00f6510 2024-02-11T17:07:51.579 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:51.577+0000 7fccc77f7700 1 -- 172.21.15.178:0/1209303725 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fccc00fc050 con 0x7fccc00f6510 2024-02-11T17:07:51.579 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:51.578+0000 7fccb5ffb700 1 -- 172.21.15.178:0/1209303725 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fccb000f340 con 0x7fccc00f6510 2024-02-11T17:07:51.580 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:07:51.580 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:07:51.581 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:51.579+0000 7fccc77f7700 1 -- 172.21.15.178:0/1209303725 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fccac02ff60 msgr2=0x7fccac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:07:51.581 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:51.579+0000 7fccc77f7700 1 --2- 172.21.15.178:0/1209303725 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fccac02ff60 0x7fccac032420 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7fccbc0096a0 tx=0x7fccbc006b40).stop 2024-02-11T17:07:51.581 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:51.579+0000 7fccc77f7700 1 -- 172.21.15.178:0/1209303725 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fccc00f6510 msgr2=0x7fccc0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:07:51.582 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:51.579+0000 7fccc77f7700 1 --2- 172.21.15.178:0/1209303725 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fccc00f6510 0x7fccc0106730 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7fccb0013510 tx=0x7fccb000b040).stop 2024-02-11T17:07:51.582 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:51.579+0000 7fccc77f7700 1 -- 172.21.15.178:0/1209303725 shutdown_connections 2024-02-11T17:07:51.582 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:51.579+0000 7fccc77f7700 1 --2- 172.21.15.178:0/1209303725 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fccac02ff60 0x7fccac032420 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:07:51.582 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:51.579+0000 7fccc77f7700 1 --2- 172.21.15.178:0/1209303725 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fccc00f6510 0x7fccc0106730 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:07:51.582 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:51.579+0000 7fccc77f7700 1 -- 172.21.15.178:0/1209303725 >> 172.21.15.178:0/1209303725 conn(0x7fccc00f1c70 msgr2=0x7fccc00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:07:51.582 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:51.580+0000 7fccc77f7700 1 -- 172.21.15.178:0/1209303725 shutdown_connections 2024-02-11T17:07:51.582 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:51.580+0000 7fccc77f7700 1 -- 172.21.15.178:0/1209303725 wait complete. 2024-02-11T17:07:51.582 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:07:52.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:52 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:07:51.202322+0000 mgr.smithi037.eofnaq (mgr.14162) 242 : cluster [DBG] pgmap v228: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:07:52.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:52 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:07:51.579234+0000 mon.smithi037 (mon.0) 254 : audit [DBG] from='client.? 172.21.15.178:0/1209303725' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:07:53.337 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:07:53.338 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:07:54.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:54 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:07:53.202625+0000 mgr.smithi037.eofnaq (mgr.14162) 243 : cluster [DBG] pgmap v229: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:07:55.190 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:55.187+0000 7f9d2a6f0700 1 -- 172.21.15.178:0/3663730927 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9d240f6510 msgr2=0x7f9d240f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:07:55.190 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:55.187+0000 7f9d2a6f0700 1 --2- 172.21.15.178:0/3663730927 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9d240f6510 0x7f9d240f6930 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7f9d18009fb0 tx=0x7f9d18009b30).stop 2024-02-11T17:07:55.190 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:55.187+0000 7f9d2a6f0700 1 -- 172.21.15.178:0/3663730927 shutdown_connections 2024-02-11T17:07:55.190 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:55.187+0000 7f9d2a6f0700 1 --2- 172.21.15.178:0/3663730927 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9d240f6510 0x7f9d240f6930 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:07:55.191 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:55.187+0000 7f9d2a6f0700 1 -- 172.21.15.178:0/3663730927 >> 172.21.15.178:0/3663730927 conn(0x7f9d240f1c70 msgr2=0x7f9d240f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:07:55.191 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:55.187+0000 7f9d2a6f0700 1 -- 172.21.15.178:0/3663730927 shutdown_connections 2024-02-11T17:07:55.191 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:55.187+0000 7f9d2a6f0700 1 -- 172.21.15.178:0/3663730927 wait complete. 2024-02-11T17:07:55.191 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:55.187+0000 7f9d2a6f0700 1 Processor -- start 2024-02-11T17:07:55.191 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:55.188+0000 7f9d2a6f0700 1 -- start start 2024-02-11T17:07:55.191 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:55.188+0000 7f9d2a6f0700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9d240f6510 0x7f9d2406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:07:55.191 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:55.188+0000 7f9d2a6f0700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d24069e20 con 0x7f9d240f6510 2024-02-11T17:07:55.191 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:55.188+0000 7f9d23fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9d240f6510 0x7f9d2406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:07:55.191 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:55.188+0000 7f9d23fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9d240f6510 0x7f9d2406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:47546/0 (socket says 172.21.15.178:47546) 2024-02-11T17:07:55.191 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:55.188+0000 7f9d23fff700 1 -- 172.21.15.178:0/4290910329 learned_addr learned my addr 172.21.15.178:0/4290910329 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:07:55.191 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:55.188+0000 7f9d23fff700 1 -- 172.21.15.178:0/4290910329 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d18005040 con 0x7f9d240f6510 2024-02-11T17:07:55.191 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:55.188+0000 7f9d23fff700 1 --2- 172.21.15.178:0/4290910329 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9d240f6510 0x7f9d2406b9a0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f9d18014000 tx=0x7f9d1800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:07:55.192 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:55.189+0000 7f9d20ff9700 1 -- 172.21.15.178:0/4290910329 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d180095f0 con 0x7f9d240f6510 2024-02-11T17:07:55.192 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:55.189+0000 7f9d2a6f0700 1 -- 172.21.15.178:0/4290910329 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d24069fc0 con 0x7f9d240f6510 2024-02-11T17:07:55.192 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:55.189+0000 7f9d20ff9700 1 -- 172.21.15.178:0/4290910329 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9d18007e70 con 0x7f9d240f6510 2024-02-11T17:07:55.192 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:55.189+0000 7f9d20ff9700 1 -- 172.21.15.178:0/4290910329 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d18014ea0 con 0x7f9d240f6510 2024-02-11T17:07:55.192 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:55.189+0000 7f9d2a6f0700 1 -- 172.21.15.178:0/4290910329 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d2406a450 con 0x7f9d240f6510 2024-02-11T17:07:55.193 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:55.189+0000 7f9d20ff9700 1 -- 172.21.15.178:0/4290910329 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9d18025070 con 0x7f9d240f6510 2024-02-11T17:07:55.193 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:55.189+0000 7f9d2a6f0700 1 -- 172.21.15.178:0/4290910329 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d24059100 con 0x7f9d240f6510 2024-02-11T17:07:55.193 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:55.190+0000 7f9d20ff9700 1 --2- 172.21.15.178:0/4290910329 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f9d0c02ff60 0x7f9d0c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:07:55.193 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:55.190+0000 7f9d20ff9700 1 -- 172.21.15.178:0/4290910329 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9d18040940 con 0x7f9d240f6510 2024-02-11T17:07:55.193 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:55.190+0000 7f9d237fe700 1 --2- 172.21.15.178:0/4290910329 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f9d0c02ff60 0x7f9d0c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:07:55.193 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:55.190+0000 7f9d237fe700 1 --2- 172.21.15.178:0/4290910329 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f9d0c02ff60 0x7f9d0c032420 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f9d140096a0 tx=0x7f9d14006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:07:55.196 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:55.194+0000 7f9d20ff9700 1 -- 172.21.15.178:0/4290910329 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9d180079c0 con 0x7f9d240f6510 2024-02-11T17:07:55.402 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:55.400+0000 7f9d2a6f0700 1 -- 172.21.15.178:0/4290910329 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9d24045ad0 con 0x7f9d240f6510 2024-02-11T17:07:55.403 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:55.401+0000 7f9d20ff9700 1 -- 172.21.15.178:0/4290910329 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9d1800f300 con 0x7f9d240f6510 2024-02-11T17:07:55.403 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:07:55.403 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:07:55.404 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:55.402+0000 7f9d2a6f0700 1 -- 172.21.15.178:0/4290910329 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f9d0c02ff60 msgr2=0x7f9d0c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:07:55.404 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:55.402+0000 7f9d2a6f0700 1 --2- 172.21.15.178:0/4290910329 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f9d0c02ff60 0x7f9d0c032420 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f9d140096a0 tx=0x7f9d14006b40).stop 2024-02-11T17:07:55.404 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:55.402+0000 7f9d2a6f0700 1 -- 172.21.15.178:0/4290910329 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9d240f6510 msgr2=0x7f9d2406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:07:55.404 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:55.402+0000 7f9d2a6f0700 1 --2- 172.21.15.178:0/4290910329 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9d240f6510 0x7f9d2406b9a0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f9d18014000 tx=0x7f9d1800b040).stop 2024-02-11T17:07:55.405 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:55.402+0000 7f9d2a6f0700 1 -- 172.21.15.178:0/4290910329 shutdown_connections 2024-02-11T17:07:55.405 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:55.402+0000 7f9d2a6f0700 1 --2- 172.21.15.178:0/4290910329 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f9d0c02ff60 0x7f9d0c032420 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:07:55.405 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:55.402+0000 7f9d2a6f0700 1 --2- 172.21.15.178:0/4290910329 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9d240f6510 0x7f9d2406b9a0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:07:55.405 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:55.402+0000 7f9d2a6f0700 1 -- 172.21.15.178:0/4290910329 >> 172.21.15.178:0/4290910329 conn(0x7f9d240f1c70 msgr2=0x7f9d240f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:07:55.405 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:55.403+0000 7f9d2a6f0700 1 -- 172.21.15.178:0/4290910329 shutdown_connections 2024-02-11T17:07:55.405 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:55.403+0000 7f9d2a6f0700 1 -- 172.21.15.178:0/4290910329 wait complete. 2024-02-11T17:07:55.405 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:07:56.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:56 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:07:55.202861+0000 mgr.smithi037.eofnaq (mgr.14162) 244 : cluster [DBG] pgmap v230: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:07:56.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:56 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:07:55.402261+0000 mon.smithi037 (mon.0) 255 : audit [DBG] from='client.? 172.21.15.178:0/4290910329' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:07:57.053 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:07:57.053 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:07:58.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:58 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:07:57.203180+0000 mgr.smithi037.eofnaq (mgr.14162) 245 : cluster [DBG] pgmap v231: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:07:58.951 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:58.949+0000 7fdb95229700 1 -- 172.21.15.178:0/1437704921 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdb900f6510 msgr2=0x7fdb900f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:07:58.951 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:58.949+0000 7fdb95229700 1 --2- 172.21.15.178:0/1437704921 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdb900f6510 0x7fdb900f6930 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7fdb84009fb0 tx=0x7fdb84009b30).stop 2024-02-11T17:07:58.951 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:58.949+0000 7fdb95229700 1 -- 172.21.15.178:0/1437704921 shutdown_connections 2024-02-11T17:07:58.951 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:58.949+0000 7fdb95229700 1 --2- 172.21.15.178:0/1437704921 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdb900f6510 0x7fdb900f6930 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:07:58.951 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:58.949+0000 7fdb95229700 1 -- 172.21.15.178:0/1437704921 >> 172.21.15.178:0/1437704921 conn(0x7fdb900f1c70 msgr2=0x7fdb900f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:07:58.952 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:58.949+0000 7fdb95229700 1 -- 172.21.15.178:0/1437704921 shutdown_connections 2024-02-11T17:07:58.952 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:58.949+0000 7fdb95229700 1 -- 172.21.15.178:0/1437704921 wait complete. 2024-02-11T17:07:58.953 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:58.950+0000 7fdb95229700 1 Processor -- start 2024-02-11T17:07:58.953 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:58.950+0000 7fdb95229700 1 -- start start 2024-02-11T17:07:58.953 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:58.950+0000 7fdb95229700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdb900f6510 0x7fdb90106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:07:58.953 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:58.950+0000 7fdb95229700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb90106c20 con 0x7fdb900f6510 2024-02-11T17:07:58.953 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:58.950+0000 7fdb8ed9d700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdb900f6510 0x7fdb90106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:07:58.953 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:58.950+0000 7fdb8ed9d700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdb900f6510 0x7fdb90106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:47560/0 (socket says 172.21.15.178:47560) 2024-02-11T17:07:58.953 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:58.950+0000 7fdb8ed9d700 1 -- 172.21.15.178:0/3264084653 learned_addr learned my addr 172.21.15.178:0/3264084653 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:07:58.954 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:58.951+0000 7fdb8ed9d700 1 -- 172.21.15.178:0/3264084653 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb84005040 con 0x7fdb900f6510 2024-02-11T17:07:58.954 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:58.951+0000 7fdb8ed9d700 1 --2- 172.21.15.178:0/3264084653 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdb900f6510 0x7fdb90106730 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7fdb84014000 tx=0x7fdb8400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:07:58.954 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:58.951+0000 7fdb777fe700 1 -- 172.21.15.178:0/3264084653 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdb840095f0 con 0x7fdb900f6510 2024-02-11T17:07:58.954 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:58.951+0000 7fdb95229700 1 -- 172.21.15.178:0/3264084653 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb90106e20 con 0x7fdb900f6510 2024-02-11T17:07:58.954 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:58.951+0000 7fdb777fe700 1 -- 172.21.15.178:0/3264084653 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdb84007e70 con 0x7fdb900f6510 2024-02-11T17:07:58.954 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:58.951+0000 7fdb777fe700 1 -- 172.21.15.178:0/3264084653 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdb84014ea0 con 0x7fdb900f6510 2024-02-11T17:07:58.954 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:58.951+0000 7fdb95229700 1 -- 172.21.15.178:0/3264084653 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb901072b0 con 0x7fdb900f6510 2024-02-11T17:07:58.955 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:58.952+0000 7fdb777fe700 1 -- 172.21.15.178:0/3264084653 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdb84025070 con 0x7fdb900f6510 2024-02-11T17:07:58.955 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:58.952+0000 7fdb95229700 1 -- 172.21.15.178:0/3264084653 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb90045ad0 con 0x7fdb900f6510 2024-02-11T17:07:58.955 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:58.952+0000 7fdb777fe700 1 --2- 172.21.15.178:0/3264084653 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fdb7802ff60 0x7fdb78032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:07:58.955 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:58.952+0000 7fdb777fe700 1 -- 172.21.15.178:0/3264084653 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdb840409f0 con 0x7fdb900f6510 2024-02-11T17:07:58.955 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:58.952+0000 7fdb8e59c700 1 --2- 172.21.15.178:0/3264084653 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fdb7802ff60 0x7fdb78032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:07:58.955 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:58.953+0000 7fdb8e59c700 1 --2- 172.21.15.178:0/3264084653 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fdb7802ff60 0x7fdb78032420 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fdb800096a0 tx=0x7fdb80006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:07:58.958 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:58.957+0000 7fdb777fe700 1 -- 172.21.15.178:0/3264084653 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdb8400f4e0 con 0x7fdb900f6510 2024-02-11T17:07:59.163 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:59.162+0000 7fdb95229700 1 -- 172.21.15.178:0/3264084653 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdb900fbf60 con 0x7fdb900f6510 2024-02-11T17:07:59.164 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:59.162+0000 7fdb777fe700 1 -- 172.21.15.178:0/3264084653 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdb8400f300 con 0x7fdb900f6510 2024-02-11T17:07:59.164 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:07:59.165 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:07:59.166 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:59.164+0000 7fdb95229700 1 -- 172.21.15.178:0/3264084653 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fdb7802ff60 msgr2=0x7fdb78032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:07:59.166 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:59.164+0000 7fdb95229700 1 --2- 172.21.15.178:0/3264084653 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fdb7802ff60 0x7fdb78032420 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fdb800096a0 tx=0x7fdb80006b40).stop 2024-02-11T17:07:59.166 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:59.164+0000 7fdb95229700 1 -- 172.21.15.178:0/3264084653 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdb900f6510 msgr2=0x7fdb90106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:07:59.166 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:59.164+0000 7fdb95229700 1 --2- 172.21.15.178:0/3264084653 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdb900f6510 0x7fdb90106730 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7fdb84014000 tx=0x7fdb8400b040).stop 2024-02-11T17:07:59.166 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:59.164+0000 7fdb95229700 1 -- 172.21.15.178:0/3264084653 shutdown_connections 2024-02-11T17:07:59.166 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:59.164+0000 7fdb95229700 1 --2- 172.21.15.178:0/3264084653 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fdb7802ff60 0x7fdb78032420 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:07:59.166 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:59.164+0000 7fdb95229700 1 --2- 172.21.15.178:0/3264084653 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdb900f6510 0x7fdb90106730 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:07:59.166 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:59.164+0000 7fdb95229700 1 -- 172.21.15.178:0/3264084653 >> 172.21.15.178:0/3264084653 conn(0x7fdb900f1c70 msgr2=0x7fdb900f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:07:59.166 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:59.164+0000 7fdb95229700 1 -- 172.21.15.178:0/3264084653 shutdown_connections 2024-02-11T17:07:59.166 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:07:59.164+0000 7fdb95229700 1 -- 172.21.15.178:0/3264084653 wait complete. 2024-02-11T17:07:59.167 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:07:59.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:07:59 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:07:59.163870+0000 mon.smithi037 (mon.0) 256 : audit [DBG] from='client.? 172.21.15.178:0/3264084653' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:08:00.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:08:00 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:07:59.203429+0000 mgr.smithi037.eofnaq (mgr.14162) 246 : cluster [DBG] pgmap v232: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:08:00.865 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:08:00.865 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:08:02.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:08:02 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:08:01.203793+0000 mgr.smithi037.eofnaq (mgr.14162) 247 : cluster [DBG] pgmap v233: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:08:02.884 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:02.880+0000 7f1016925700 1 -- 172.21.15.178:0/430516363 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f10100f4540 msgr2=0x7f10100f4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:08:02.884 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:02.880+0000 7f1016925700 1 --2- 172.21.15.178:0/430516363 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f10100f4540 0x7f10100f4960 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7f100c009fb0 tx=0x7f100c009b30).stop 2024-02-11T17:08:02.884 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:02.880+0000 7f1016925700 1 -- 172.21.15.178:0/430516363 shutdown_connections 2024-02-11T17:08:02.884 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:02.880+0000 7f1016925700 1 --2- 172.21.15.178:0/430516363 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f10100f4540 0x7f10100f4960 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:08:02.884 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:02.880+0000 7f1016925700 1 -- 172.21.15.178:0/430516363 >> 172.21.15.178:0/430516363 conn(0x7f10100effc0 msgr2=0x7f10100f23e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:08:02.884 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:02.880+0000 7f1016925700 1 -- 172.21.15.178:0/430516363 shutdown_connections 2024-02-11T17:08:02.884 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:02.881+0000 7f1016925700 1 -- 172.21.15.178:0/430516363 wait complete. 2024-02-11T17:08:02.884 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:02.881+0000 7f1016925700 1 Processor -- start 2024-02-11T17:08:02.884 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:02.881+0000 7f1016925700 1 -- start start 2024-02-11T17:08:02.885 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:02.881+0000 7f1016925700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f10100f4540 0x7f1010106480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:08:02.885 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:02.881+0000 7f1016925700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1010106970 con 0x7f10100f4540 2024-02-11T17:08:02.885 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:02.881+0000 7f1015923700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f10100f4540 0x7f1010106480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:08:02.885 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:02.881+0000 7f1015923700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f10100f4540 0x7f1010106480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:52520/0 (socket says 172.21.15.178:52520) 2024-02-11T17:08:02.885 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:02.881+0000 7f1015923700 1 -- 172.21.15.178:0/755870650 learned_addr learned my addr 172.21.15.178:0/755870650 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:08:02.885 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:02.882+0000 7f1015923700 1 -- 172.21.15.178:0/755870650 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f100c005040 con 0x7f10100f4540 2024-02-11T17:08:02.885 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:02.882+0000 7f1015923700 1 --2- 172.21.15.178:0/755870650 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f10100f4540 0x7f1010106480 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f100c013510 tx=0x7f100c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:08:02.885 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:02.882+0000 7f10027fc700 1 -- 172.21.15.178:0/755870650 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f100c009320 con 0x7f10100f4540 2024-02-11T17:08:02.885 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:02.882+0000 7f1016925700 1 -- 172.21.15.178:0/755870650 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1010106b70 con 0x7f10100f4540 2024-02-11T17:08:02.885 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:02.882+0000 7f1016925700 1 -- 172.21.15.178:0/755870650 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1010107000 con 0x7f10100f4540 2024-02-11T17:08:02.885 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:02.882+0000 7f10027fc700 1 -- 172.21.15.178:0/755870650 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f100c013f00 con 0x7f10100f4540 2024-02-11T17:08:02.886 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:02.882+0000 7f10027fc700 1 -- 172.21.15.178:0/755870650 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f100c014d60 con 0x7f10100f4540 2024-02-11T17:08:02.887 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:02.883+0000 7f10027fc700 1 -- 172.21.15.178:0/755870650 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f100c0095f0 con 0x7f10100f4540 2024-02-11T17:08:02.887 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:02.883+0000 7f1016925700 1 -- 172.21.15.178:0/755870650 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1010045ad0 con 0x7f10100f4540 2024-02-11T17:08:02.887 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:02.883+0000 7f10027fc700 1 --2- 172.21.15.178:0/755870650 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f0ffc02ff60 0x7f0ffc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:08:02.887 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:02.883+0000 7f10027fc700 1 -- 172.21.15.178:0/755870650 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f100c040ff0 con 0x7f10100f4540 2024-02-11T17:08:02.887 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:02.883+0000 7f1015122700 1 --2- 172.21.15.178:0/755870650 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f0ffc02ff60 0x7f0ffc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:08:02.887 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:02.884+0000 7f1015122700 1 --2- 172.21.15.178:0/755870650 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f0ffc02ff60 0x7f0ffc032420 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f10040096a0 tx=0x7f1004006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:08:02.889 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:02.888+0000 7f10027fc700 1 -- 172.21.15.178:0/755870650 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f100c014420 con 0x7f10100f4540 2024-02-11T17:08:03.095 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:03.093+0000 7f1016925700 1 -- 172.21.15.178:0/755870650 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1010107eb0 con 0x7f10100f4540 2024-02-11T17:08:03.096 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:03.094+0000 7f10027fc700 1 -- 172.21.15.178:0/755870650 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f100c012980 con 0x7f10100f4540 2024-02-11T17:08:03.096 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:08:03.096 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:08:03.098 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:03.096+0000 7f1016925700 1 -- 172.21.15.178:0/755870650 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f0ffc02ff60 msgr2=0x7f0ffc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:08:03.098 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:03.096+0000 7f1016925700 1 --2- 172.21.15.178:0/755870650 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f0ffc02ff60 0x7f0ffc032420 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f10040096a0 tx=0x7f1004006b40).stop 2024-02-11T17:08:03.098 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:03.096+0000 7f1016925700 1 -- 172.21.15.178:0/755870650 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f10100f4540 msgr2=0x7f1010106480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:08:03.098 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:03.096+0000 7f1016925700 1 --2- 172.21.15.178:0/755870650 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f10100f4540 0x7f1010106480 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f100c013510 tx=0x7f100c00b040).stop 2024-02-11T17:08:03.098 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:03.096+0000 7f1016925700 1 -- 172.21.15.178:0/755870650 shutdown_connections 2024-02-11T17:08:03.098 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:03.096+0000 7f1016925700 1 --2- 172.21.15.178:0/755870650 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f0ffc02ff60 0x7f0ffc032420 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:08:03.099 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:03.096+0000 7f1016925700 1 --2- 172.21.15.178:0/755870650 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f10100f4540 0x7f1010106480 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:08:03.099 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:03.096+0000 7f1016925700 1 -- 172.21.15.178:0/755870650 >> 172.21.15.178:0/755870650 conn(0x7f10100effc0 msgr2=0x7f10100f0ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:08:03.099 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:03.096+0000 7f1016925700 1 -- 172.21.15.178:0/755870650 shutdown_connections 2024-02-11T17:08:03.099 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:03.096+0000 7f1016925700 1 -- 172.21.15.178:0/755870650 wait complete. 2024-02-11T17:08:03.099 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:08:03.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:08:03 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:08:03.095477+0000 mon.smithi037 (mon.0) 257 : audit [DBG] from='client.? 172.21.15.178:0/755870650' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:08:04.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:08:04 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:08:03.204165+0000 mgr.smithi037.eofnaq (mgr.14162) 248 : cluster [DBG] pgmap v234: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:08:04.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:08:04 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:08:03.278330+0000 mon.smithi037 (mon.0) 258 : audit [INF] from='mgr.14162 172.21.15.37:0/2745465012' entity='mgr.smithi037.eofnaq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi037.eofnaq/mirror_snapshot_schedule"}]: dispatch 2024-02-11T17:08:04.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:08:04 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:08:03.281051+0000 mon.smithi037 (mon.0) 259 : audit [INF] from='mgr.14162 172.21.15.37:0/2745465012' entity='mgr.smithi037.eofnaq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi037.eofnaq/trash_purge_schedule"}]: dispatch 2024-02-11T17:08:04.812 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:08:04.813 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:08:06.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:08:06 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:08:05.204496+0000 mgr.smithi037.eofnaq (mgr.14162) 249 : cluster [DBG] pgmap v235: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:08:06.660 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:06.656+0000 7fac1677e700 1 -- 172.21.15.178:0/1967762409 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fac100f5410 msgr2=0x7fac100f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:08:06.660 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:06.656+0000 7fac1677e700 1 --2- 172.21.15.178:0/1967762409 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fac100f5410 0x7fac100f5830 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7fabf8009fb0 tx=0x7fabf8009b30).stop 2024-02-11T17:08:06.660 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:06.656+0000 7fac1677e700 1 -- 172.21.15.178:0/1967762409 shutdown_connections 2024-02-11T17:08:06.660 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:06.656+0000 7fac1677e700 1 --2- 172.21.15.178:0/1967762409 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fac100f5410 0x7fac100f5830 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:08:06.660 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:06.656+0000 7fac1677e700 1 -- 172.21.15.178:0/1967762409 >> 172.21.15.178:0/1967762409 conn(0x7fac100f0b70 msgr2=0x7fac100f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:08:06.660 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:06.656+0000 7fac1677e700 1 -- 172.21.15.178:0/1967762409 shutdown_connections 2024-02-11T17:08:06.660 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:06.656+0000 7fac1677e700 1 -- 172.21.15.178:0/1967762409 wait complete. 2024-02-11T17:08:06.660 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:06.657+0000 7fac1677e700 1 Processor -- start 2024-02-11T17:08:06.660 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:06.657+0000 7fac1677e700 1 -- start start 2024-02-11T17:08:06.660 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:06.657+0000 7fac1677e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fac100f5410 0x7fac10109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:08:06.661 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:06.657+0000 7fac1677e700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac10060220 con 0x7fac100f5410 2024-02-11T17:08:06.661 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:06.658+0000 7fac0ffff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fac100f5410 0x7fac10109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:08:06.661 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:06.658+0000 7fac0ffff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fac100f5410 0x7fac10109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:52542/0 (socket says 172.21.15.178:52542) 2024-02-11T17:08:06.661 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:06.658+0000 7fac0ffff700 1 -- 172.21.15.178:0/1169324412 learned_addr learned my addr 172.21.15.178:0/1169324412 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:08:06.661 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:06.658+0000 7fac0ffff700 1 -- 172.21.15.178:0/1169324412 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabf8005040 con 0x7fac100f5410 2024-02-11T17:08:06.661 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:06.658+0000 7fac0ffff700 1 --2- 172.21.15.178:0/1169324412 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fac100f5410 0x7fac10109450 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7fabf8013510 tx=0x7fabf800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:08:06.661 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:06.658+0000 7fac0cff9700 1 -- 172.21.15.178:0/1169324412 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fabf8009320 con 0x7fac100f5410 2024-02-11T17:08:06.661 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:06.658+0000 7fac1677e700 1 -- 172.21.15.178:0/1169324412 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac10060420 con 0x7fac100f5410 2024-02-11T17:08:06.661 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:06.658+0000 7fac0cff9700 1 -- 172.21.15.178:0/1169324412 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fabf8013f00 con 0x7fac100f5410 2024-02-11T17:08:06.661 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:06.658+0000 7fac0cff9700 1 -- 172.21.15.178:0/1169324412 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fabf8014d90 con 0x7fac100f5410 2024-02-11T17:08:06.661 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:06.658+0000 7fac1677e700 1 -- 172.21.15.178:0/1169324412 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac10109b00 con 0x7fac100f5410 2024-02-11T17:08:06.662 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:06.659+0000 7fac0cff9700 1 -- 172.21.15.178:0/1169324412 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fabf80095f0 con 0x7fac100f5410 2024-02-11T17:08:06.662 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:06.659+0000 7fac1677e700 1 -- 172.21.15.178:0/1169324412 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac10059100 con 0x7fac100f5410 2024-02-11T17:08:06.662 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:06.659+0000 7fac0cff9700 1 --2- 172.21.15.178:0/1169324412 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fabfc02ff60 0x7fabfc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:08:06.662 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:06.659+0000 7fac0cff9700 1 -- 172.21.15.178:0/1169324412 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fabf8040fe0 con 0x7fac100f5410 2024-02-11T17:08:06.662 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:06.660+0000 7fac0f7fe700 1 --2- 172.21.15.178:0/1169324412 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fabfc02ff60 0x7fabfc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:08:06.662 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:06.660+0000 7fac0f7fe700 1 --2- 172.21.15.178:0/1169324412 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fabfc02ff60 0x7fabfc032420 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fac000096a0 tx=0x7fac00006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:08:06.666 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:06.664+0000 7fac0cff9700 1 -- 172.21.15.178:0/1169324412 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fabf801d7f0 con 0x7fac100f5410 2024-02-11T17:08:06.869 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:06.867+0000 7fac1677e700 1 -- 172.21.15.178:0/1169324412 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fac10045ad0 con 0x7fac100f5410 2024-02-11T17:08:06.870 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:06.868+0000 7fac0cff9700 1 -- 172.21.15.178:0/1169324412 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fabf8010230 con 0x7fac100f5410 2024-02-11T17:08:06.870 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:08:06.870 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:08:06.872 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:06.870+0000 7fac1677e700 1 -- 172.21.15.178:0/1169324412 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fabfc02ff60 msgr2=0x7fabfc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:08:06.872 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:06.870+0000 7fac1677e700 1 --2- 172.21.15.178:0/1169324412 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fabfc02ff60 0x7fabfc032420 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fac000096a0 tx=0x7fac00006b40).stop 2024-02-11T17:08:06.872 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:06.870+0000 7fac1677e700 1 -- 172.21.15.178:0/1169324412 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fac100f5410 msgr2=0x7fac10109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:08:06.872 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:06.870+0000 7fac1677e700 1 --2- 172.21.15.178:0/1169324412 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fac100f5410 0x7fac10109450 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7fabf8013510 tx=0x7fabf800b040).stop 2024-02-11T17:08:06.872 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:06.870+0000 7fac1677e700 1 -- 172.21.15.178:0/1169324412 shutdown_connections 2024-02-11T17:08:06.872 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:06.870+0000 7fac1677e700 1 --2- 172.21.15.178:0/1169324412 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fabfc02ff60 0x7fabfc032420 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:08:06.872 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:06.870+0000 7fac1677e700 1 --2- 172.21.15.178:0/1169324412 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fac100f5410 0x7fac10109450 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:08:06.872 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:06.870+0000 7fac1677e700 1 -- 172.21.15.178:0/1169324412 >> 172.21.15.178:0/1169324412 conn(0x7fac100f0b70 msgr2=0x7fac100f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:08:06.872 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:06.870+0000 7fac1677e700 1 -- 172.21.15.178:0/1169324412 shutdown_connections 2024-02-11T17:08:06.872 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:06.870+0000 7fac1677e700 1 -- 172.21.15.178:0/1169324412 wait complete. 2024-02-11T17:08:06.873 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:08:07.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:08:07 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:08:06.869519+0000 mon.smithi037 (mon.0) 260 : audit [DBG] from='client.? 172.21.15.178:0/1169324412' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:08:08.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:08:08 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:08:07.204815+0000 mgr.smithi037.eofnaq (mgr.14162) 250 : cluster [DBG] pgmap v236: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:08:08.686 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:08:08.686 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:08:10.630 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:10.628+0000 7fbb4cb87700 1 -- 172.21.15.178:0/2639176429 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb480f5410 msgr2=0x7fbb480f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:08:10.630 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:10.628+0000 7fbb4cb87700 1 --2- 172.21.15.178:0/2639176429 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb480f5410 0x7fbb480f5830 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7fbb30009fb0 tx=0x7fbb30009b30).stop 2024-02-11T17:08:10.630 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:10.628+0000 7fbb4cb87700 1 -- 172.21.15.178:0/2639176429 shutdown_connections 2024-02-11T17:08:10.631 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:10.628+0000 7fbb4cb87700 1 --2- 172.21.15.178:0/2639176429 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb480f5410 0x7fbb480f5830 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:08:10.631 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:10.628+0000 7fbb4cb87700 1 -- 172.21.15.178:0/2639176429 >> 172.21.15.178:0/2639176429 conn(0x7fbb480f0b70 msgr2=0x7fbb480f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:08:10.631 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:10.628+0000 7fbb4cb87700 1 -- 172.21.15.178:0/2639176429 shutdown_connections 2024-02-11T17:08:10.631 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:10.628+0000 7fbb4cb87700 1 -- 172.21.15.178:0/2639176429 wait complete. 2024-02-11T17:08:10.631 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:10.628+0000 7fbb4cb87700 1 Processor -- start 2024-02-11T17:08:10.631 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:10.628+0000 7fbb4cb87700 1 -- start start 2024-02-11T17:08:10.631 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:10.628+0000 7fbb4cb87700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb480f5410 0x7fbb48109400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:08:10.631 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:10.628+0000 7fbb4cb87700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb48060240 con 0x7fbb480f5410 2024-02-11T17:08:10.631 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:10.629+0000 7fbb4659c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb480f5410 0x7fbb48109400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:08:10.631 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:10.629+0000 7fbb4659c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb480f5410 0x7fbb48109400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:57318/0 (socket says 172.21.15.178:57318) 2024-02-11T17:08:10.631 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:10.629+0000 7fbb4659c700 1 -- 172.21.15.178:0/2267977867 learned_addr learned my addr 172.21.15.178:0/2267977867 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:08:10.633 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:10.629+0000 7fbb4659c700 1 -- 172.21.15.178:0/2267977867 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb30005040 con 0x7fbb480f5410 2024-02-11T17:08:10.633 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:10.629+0000 7fbb4659c700 1 --2- 172.21.15.178:0/2267977867 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb480f5410 0x7fbb48109400 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7fbb30013510 tx=0x7fbb3000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:08:10.633 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:10.629+0000 7fbb3effd700 1 -- 172.21.15.178:0/2267977867 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb30009320 con 0x7fbb480f5410 2024-02-11T17:08:10.633 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:10.629+0000 7fbb3effd700 1 -- 172.21.15.178:0/2267977867 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbb30013f00 con 0x7fbb480f5410 2024-02-11T17:08:10.633 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:10.629+0000 7fbb4cb87700 1 -- 172.21.15.178:0/2267977867 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb48060440 con 0x7fbb480f5410 2024-02-11T17:08:10.633 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:10.629+0000 7fbb3effd700 1 -- 172.21.15.178:0/2267977867 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb30014d90 con 0x7fbb480f5410 2024-02-11T17:08:10.633 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:10.629+0000 7fbb4cb87700 1 -- 172.21.15.178:0/2267977867 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb48109ab0 con 0x7fbb480f5410 2024-02-11T17:08:10.633 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:10.630+0000 7fbb3effd700 1 -- 172.21.15.178:0/2267977867 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbb300263a0 con 0x7fbb480f5410 2024-02-11T17:08:10.633 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:10.630+0000 7fbb3effd700 1 --2- 172.21.15.178:0/2267977867 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fbb3402ff60 0x7fbb34032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:08:10.633 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:10.630+0000 7fbb3effd700 1 -- 172.21.15.178:0/2267977867 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbb30040fe0 con 0x7fbb480f5410 2024-02-11T17:08:10.634 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:10.630+0000 7fbb4cb87700 1 -- 172.21.15.178:0/2267977867 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb48045ad0 con 0x7fbb480f5410 2024-02-11T17:08:10.634 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:10.630+0000 7fbb45d9b700 1 --2- 172.21.15.178:0/2267977867 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fbb3402ff60 0x7fbb34032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:08:10.634 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:10.631+0000 7fbb45d9b700 1 --2- 172.21.15.178:0/2267977867 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fbb3402ff60 0x7fbb34032420 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7fbb380096a0 tx=0x7fbb38006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:08:10.637 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:10.635+0000 7fbb3effd700 1 -- 172.21.15.178:0/2267977867 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbb30012920 con 0x7fbb480f5410 2024-02-11T17:08:10.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:08:10 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:08:09.205083+0000 mgr.smithi037.eofnaq (mgr.14162) 251 : cluster [DBG] pgmap v237: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:08:10.844 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:10.842+0000 7fbb4cb87700 1 -- 172.21.15.178:0/2267977867 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbb4810a190 con 0x7fbb480f5410 2024-02-11T17:08:10.844 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:10.842+0000 7fbb3effd700 1 -- 172.21.15.178:0/2267977867 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbb3002b020 con 0x7fbb480f5410 2024-02-11T17:08:10.845 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:08:10.845 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:08:10.846 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:10.844+0000 7fbb4cb87700 1 -- 172.21.15.178:0/2267977867 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fbb3402ff60 msgr2=0x7fbb34032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:08:10.846 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:10.844+0000 7fbb4cb87700 1 --2- 172.21.15.178:0/2267977867 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fbb3402ff60 0x7fbb34032420 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7fbb380096a0 tx=0x7fbb38006b40).stop 2024-02-11T17:08:10.846 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:10.844+0000 7fbb4cb87700 1 -- 172.21.15.178:0/2267977867 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb480f5410 msgr2=0x7fbb48109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:08:10.846 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:10.844+0000 7fbb4cb87700 1 --2- 172.21.15.178:0/2267977867 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb480f5410 0x7fbb48109400 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7fbb30013510 tx=0x7fbb3000b040).stop 2024-02-11T17:08:10.846 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:10.844+0000 7fbb4cb87700 1 -- 172.21.15.178:0/2267977867 shutdown_connections 2024-02-11T17:08:10.846 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:10.844+0000 7fbb4cb87700 1 --2- 172.21.15.178:0/2267977867 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fbb3402ff60 0x7fbb34032420 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:08:10.846 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:10.844+0000 7fbb4cb87700 1 --2- 172.21.15.178:0/2267977867 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb480f5410 0x7fbb48109400 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:08:10.847 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:10.844+0000 7fbb4cb87700 1 -- 172.21.15.178:0/2267977867 >> 172.21.15.178:0/2267977867 conn(0x7fbb480f0b70 msgr2=0x7fbb480f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:08:10.847 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:10.844+0000 7fbb4cb87700 1 -- 172.21.15.178:0/2267977867 shutdown_connections 2024-02-11T17:08:10.847 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:10.844+0000 7fbb4cb87700 1 -- 172.21.15.178:0/2267977867 wait complete. 2024-02-11T17:08:10.847 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:08:11.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:08:11 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:08:10.843901+0000 mon.smithi037 (mon.0) 261 : audit [DBG] from='client.? 172.21.15.178:0/2267977867' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:08:12.632 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:08:12.633 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:08:12.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:08:12 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:08:11.205406+0000 mgr.smithi037.eofnaq (mgr.14162) 252 : cluster [DBG] pgmap v238: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:08:14.597 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:14.595+0000 7fdb03e8b700 1 -- 172.21.15.178:0/2678773313 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdafc0f5410 msgr2=0x7fdafc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:08:14.598 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:14.595+0000 7fdb03e8b700 1 --2- 172.21.15.178:0/2678773313 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdafc0f5410 0x7fdafc0f5830 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7fdaec009fb0 tx=0x7fdaec009b30).stop 2024-02-11T17:08:14.598 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:14.595+0000 7fdb03e8b700 1 -- 172.21.15.178:0/2678773313 shutdown_connections 2024-02-11T17:08:14.598 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:14.595+0000 7fdb03e8b700 1 --2- 172.21.15.178:0/2678773313 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdafc0f5410 0x7fdafc0f5830 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:08:14.598 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:14.595+0000 7fdb03e8b700 1 -- 172.21.15.178:0/2678773313 >> 172.21.15.178:0/2678773313 conn(0x7fdafc0f0b70 msgr2=0x7fdafc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:08:14.598 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:14.595+0000 7fdb03e8b700 1 -- 172.21.15.178:0/2678773313 shutdown_connections 2024-02-11T17:08:14.598 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:14.595+0000 7fdb03e8b700 1 -- 172.21.15.178:0/2678773313 wait complete. 2024-02-11T17:08:14.598 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:14.596+0000 7fdb03e8b700 1 Processor -- start 2024-02-11T17:08:14.598 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:14.596+0000 7fdb03e8b700 1 -- start start 2024-02-11T17:08:14.598 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:14.596+0000 7fdb03e8b700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdafc0f5410 0x7fdafc105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:08:14.598 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:14.596+0000 7fdb03e8b700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdafc105b20 con 0x7fdafc0f5410 2024-02-11T17:08:14.600 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:14.596+0000 7fdb01c27700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdafc0f5410 0x7fdafc105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:08:14.600 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:14.596+0000 7fdb01c27700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdafc0f5410 0x7fdafc105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:57326/0 (socket says 172.21.15.178:57326) 2024-02-11T17:08:14.600 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:14.596+0000 7fdb01c27700 1 -- 172.21.15.178:0/2077871539 learned_addr learned my addr 172.21.15.178:0/2077871539 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:08:14.600 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:14.597+0000 7fdb01c27700 1 -- 172.21.15.178:0/2077871539 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaec005040 con 0x7fdafc0f5410 2024-02-11T17:08:14.600 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:14.597+0000 7fdb01c27700 1 --2- 172.21.15.178:0/2077871539 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdafc0f5410 0x7fdafc105630 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7fdaec013510 tx=0x7fdaec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:08:14.600 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:14.597+0000 7fdaf27fc700 1 -- 172.21.15.178:0/2077871539 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdaec009320 con 0x7fdafc0f5410 2024-02-11T17:08:14.600 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:14.597+0000 7fdb03e8b700 1 -- 172.21.15.178:0/2077871539 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdafc105d20 con 0x7fdafc0f5410 2024-02-11T17:08:14.601 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:14.597+0000 7fdaf27fc700 1 -- 172.21.15.178:0/2077871539 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdaec013f00 con 0x7fdafc0f5410 2024-02-11T17:08:14.601 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:14.597+0000 7fdaf27fc700 1 -- 172.21.15.178:0/2077871539 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdaec014d90 con 0x7fdafc0f5410 2024-02-11T17:08:14.601 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:14.597+0000 7fdb03e8b700 1 -- 172.21.15.178:0/2077871539 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdafc1061b0 con 0x7fdafc0f5410 2024-02-11T17:08:14.601 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:14.598+0000 7fdb03e8b700 1 -- 172.21.15.178:0/2077871539 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdafc045ad0 con 0x7fdafc0f5410 2024-02-11T17:08:14.601 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:14.598+0000 7fdaf27fc700 1 -- 172.21.15.178:0/2077871539 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdaec0095f0 con 0x7fdafc0f5410 2024-02-11T17:08:14.601 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:14.598+0000 7fdaf27fc700 1 --2- 172.21.15.178:0/2077871539 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fdae802ff60 0x7fdae8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:08:14.601 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:14.598+0000 7fdaf27fc700 1 -- 172.21.15.178:0/2077871539 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdaec040ff0 con 0x7fdafc0f5410 2024-02-11T17:08:14.601 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:14.598+0000 7fdb01426700 1 --2- 172.21.15.178:0/2077871539 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fdae802ff60 0x7fdae8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:08:14.601 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:14.599+0000 7fdb01426700 1 --2- 172.21.15.178:0/2077871539 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fdae802ff60 0x7fdae8032420 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fdaf80096a0 tx=0x7fdaf8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:08:14.604 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:14.603+0000 7fdaf27fc700 1 -- 172.21.15.178:0/2077871539 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdaec014420 con 0x7fdafc0f5410 2024-02-11T17:08:14.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:08:14 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:08:13.205694+0000 mgr.smithi037.eofnaq (mgr.14162) 253 : cluster [DBG] pgmap v239: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:08:14.810 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:14.808+0000 7fdb03e8b700 1 -- 172.21.15.178:0/2077871539 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdafc0fae60 con 0x7fdafc0f5410 2024-02-11T17:08:14.810 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:14.809+0000 7fdaf27fc700 1 -- 172.21.15.178:0/2077871539 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdaec010230 con 0x7fdafc0f5410 2024-02-11T17:08:14.811 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:08:14.811 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:08:14.812 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:14.810+0000 7fdb03e8b700 1 -- 172.21.15.178:0/2077871539 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fdae802ff60 msgr2=0x7fdae8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:08:14.812 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:14.810+0000 7fdb03e8b700 1 --2- 172.21.15.178:0/2077871539 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fdae802ff60 0x7fdae8032420 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fdaf80096a0 tx=0x7fdaf8006b40).stop 2024-02-11T17:08:14.812 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:14.810+0000 7fdb03e8b700 1 -- 172.21.15.178:0/2077871539 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdafc0f5410 msgr2=0x7fdafc105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:08:14.812 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:14.810+0000 7fdb03e8b700 1 --2- 172.21.15.178:0/2077871539 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdafc0f5410 0x7fdafc105630 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7fdaec013510 tx=0x7fdaec00b040).stop 2024-02-11T17:08:14.812 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:14.810+0000 7fdb03e8b700 1 -- 172.21.15.178:0/2077871539 shutdown_connections 2024-02-11T17:08:14.813 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:14.810+0000 7fdb03e8b700 1 --2- 172.21.15.178:0/2077871539 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fdae802ff60 0x7fdae8032420 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:08:14.813 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:14.810+0000 7fdb03e8b700 1 --2- 172.21.15.178:0/2077871539 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdafc0f5410 0x7fdafc105630 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:08:14.813 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:14.810+0000 7fdb03e8b700 1 -- 172.21.15.178:0/2077871539 >> 172.21.15.178:0/2077871539 conn(0x7fdafc0f0b70 msgr2=0x7fdafc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:08:14.813 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:14.811+0000 7fdb03e8b700 1 -- 172.21.15.178:0/2077871539 shutdown_connections 2024-02-11T17:08:14.813 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:14.811+0000 7fdb03e8b700 1 -- 172.21.15.178:0/2077871539 wait complete. 2024-02-11T17:08:14.813 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:08:15.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:08:15 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:08:14.810113+0000 mon.smithi037 (mon.0) 262 : audit [DBG] from='client.? 172.21.15.178:0/2077871539' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:08:16.578 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:08:16.578 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:08:16.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:08:16 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:08:15.205960+0000 mgr.smithi037.eofnaq (mgr.14162) 254 : cluster [DBG] pgmap v240: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:08:18.659 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:18.655+0000 7fd86db59700 1 -- 172.21.15.178:0/4273515157 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd868060220 msgr2=0x7fd8680f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:08:18.659 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:18.655+0000 7fd86db59700 1 --2- 172.21.15.178:0/4273515157 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd868060220 0x7fd8680f9150 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7fd858009fb0 tx=0x7fd858009b30).stop 2024-02-11T17:08:18.659 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:18.655+0000 7fd86db59700 1 -- 172.21.15.178:0/4273515157 shutdown_connections 2024-02-11T17:08:18.659 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:18.655+0000 7fd86db59700 1 --2- 172.21.15.178:0/4273515157 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd868060220 0x7fd8680f9150 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:08:18.659 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:18.655+0000 7fd86db59700 1 -- 172.21.15.178:0/4273515157 >> 172.21.15.178:0/4273515157 conn(0x7fd8680f0b70 msgr2=0x7fd8680f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:08:18.659 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:18.655+0000 7fd86db59700 1 -- 172.21.15.178:0/4273515157 shutdown_connections 2024-02-11T17:08:18.659 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:18.655+0000 7fd86db59700 1 -- 172.21.15.178:0/4273515157 wait complete. 2024-02-11T17:08:18.659 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:18.656+0000 7fd86db59700 1 Processor -- start 2024-02-11T17:08:18.659 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:18.656+0000 7fd86db59700 1 -- start start 2024-02-11T17:08:18.659 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:18.656+0000 7fd86db59700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd868060220 0x7fd8681066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:08:18.660 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:18.656+0000 7fd86db59700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd868106bd0 con 0x7fd868060220 2024-02-11T17:08:18.660 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:18.657+0000 7fd8677fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd868060220 0x7fd8681066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:08:18.660 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:18.657+0000 7fd8677fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd868060220 0x7fd8681066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:57338/0 (socket says 172.21.15.178:57338) 2024-02-11T17:08:18.660 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:18.657+0000 7fd8677fe700 1 -- 172.21.15.178:0/1805423378 learned_addr learned my addr 172.21.15.178:0/1805423378 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:08:18.660 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:18.657+0000 7fd8677fe700 1 -- 172.21.15.178:0/1805423378 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd858005040 con 0x7fd868060220 2024-02-11T17:08:18.660 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:18.657+0000 7fd8677fe700 1 --2- 172.21.15.178:0/1805423378 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd868060220 0x7fd8681066e0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7fd858013510 tx=0x7fd85800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:08:18.660 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:18.657+0000 7fd85ffff700 1 -- 172.21.15.178:0/1805423378 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd858009320 con 0x7fd868060220 2024-02-11T17:08:18.660 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:18.657+0000 7fd86db59700 1 -- 172.21.15.178:0/1805423378 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd868106dd0 con 0x7fd868060220 2024-02-11T17:08:18.660 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:18.657+0000 7fd85ffff700 1 -- 172.21.15.178:0/1805423378 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd858013f00 con 0x7fd868060220 2024-02-11T17:08:18.660 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:18.657+0000 7fd85ffff700 1 -- 172.21.15.178:0/1805423378 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd858014dc0 con 0x7fd868060220 2024-02-11T17:08:18.660 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:18.657+0000 7fd86db59700 1 -- 172.21.15.178:0/1805423378 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd868107260 con 0x7fd868060220 2024-02-11T17:08:18.661 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:18.658+0000 7fd85ffff700 1 -- 172.21.15.178:0/1805423378 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd8580095f0 con 0x7fd868060220 2024-02-11T17:08:18.661 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:18.658+0000 7fd86db59700 1 -- 172.21.15.178:0/1805423378 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd868045ad0 con 0x7fd868060220 2024-02-11T17:08:18.661 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:18.658+0000 7fd85ffff700 1 --2- 172.21.15.178:0/1805423378 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fd85402ff60 0x7fd854032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:08:18.661 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:18.658+0000 7fd85ffff700 1 -- 172.21.15.178:0/1805423378 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd8580410f0 con 0x7fd868060220 2024-02-11T17:08:18.662 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:18.659+0000 7fd866ffd700 1 --2- 172.21.15.178:0/1805423378 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fd85402ff60 0x7fd854032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:08:18.662 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:18.659+0000 7fd866ffd700 1 --2- 172.21.15.178:0/1805423378 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fd85402ff60 0x7fd854032420 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7fd8500096a0 tx=0x7fd850006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:08:18.665 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:18.663+0000 7fd85ffff700 1 -- 172.21.15.178:0/1805423378 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd858014420 con 0x7fd868060220 2024-02-11T17:08:18.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:08:18 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:08:17.206284+0000 mgr.smithi037.eofnaq (mgr.14162) 255 : cluster [DBG] pgmap v241: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:08:18.871 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:18.869+0000 7fd86db59700 1 -- 172.21.15.178:0/1805423378 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd868108110 con 0x7fd868060220 2024-02-11T17:08:18.871 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:18.870+0000 7fd85ffff700 1 -- 172.21.15.178:0/1805423378 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd858010230 con 0x7fd868060220 2024-02-11T17:08:18.871 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:08:18.872 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:08:18.873 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:18.871+0000 7fd86db59700 1 -- 172.21.15.178:0/1805423378 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fd85402ff60 msgr2=0x7fd854032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:08:18.873 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:18.871+0000 7fd86db59700 1 --2- 172.21.15.178:0/1805423378 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fd85402ff60 0x7fd854032420 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7fd8500096a0 tx=0x7fd850006b40).stop 2024-02-11T17:08:18.873 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:18.871+0000 7fd86db59700 1 -- 172.21.15.178:0/1805423378 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd868060220 msgr2=0x7fd8681066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:08:18.873 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:18.871+0000 7fd86db59700 1 --2- 172.21.15.178:0/1805423378 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd868060220 0x7fd8681066e0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7fd858013510 tx=0x7fd85800b040).stop 2024-02-11T17:08:18.873 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:18.871+0000 7fd86db59700 1 -- 172.21.15.178:0/1805423378 shutdown_connections 2024-02-11T17:08:18.873 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:18.871+0000 7fd86db59700 1 --2- 172.21.15.178:0/1805423378 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fd85402ff60 0x7fd854032420 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:08:18.873 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:18.871+0000 7fd86db59700 1 --2- 172.21.15.178:0/1805423378 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd868060220 0x7fd8681066e0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:08:18.874 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:18.871+0000 7fd86db59700 1 -- 172.21.15.178:0/1805423378 >> 172.21.15.178:0/1805423378 conn(0x7fd8680f0b70 msgr2=0x7fd8680f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:08:18.874 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:18.871+0000 7fd86db59700 1 -- 172.21.15.178:0/1805423378 shutdown_connections 2024-02-11T17:08:18.874 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:18.871+0000 7fd86db59700 1 -- 172.21.15.178:0/1805423378 wait complete. 2024-02-11T17:08:18.874 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:08:19.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:08:19 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:08:18.871105+0000 mon.smithi037 (mon.0) 263 : audit [DBG] from='client.? 172.21.15.178:0/1805423378' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:08:20.611 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:08:20.611 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:08:20.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:08:20 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:08:19.206616+0000 mgr.smithi037.eofnaq (mgr.14162) 256 : cluster [DBG] pgmap v242: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:08:22.491 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:22.464+0000 7fb5e81b7700 1 -- 172.21.15.178:0/1726458161 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb5e00f5410 msgr2=0x7fb5e00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:08:22.492 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:22.464+0000 7fb5e81b7700 1 --2- 172.21.15.178:0/1726458161 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb5e00f5410 0x7fb5e00f5830 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7fb5d0009fb0 tx=0x7fb5d0009b30).stop 2024-02-11T17:08:22.492 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:22.465+0000 7fb5e81b7700 1 -- 172.21.15.178:0/1726458161 shutdown_connections 2024-02-11T17:08:22.492 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:22.465+0000 7fb5e81b7700 1 --2- 172.21.15.178:0/1726458161 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb5e00f5410 0x7fb5e00f5830 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:08:22.492 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:22.465+0000 7fb5e81b7700 1 -- 172.21.15.178:0/1726458161 >> 172.21.15.178:0/1726458161 conn(0x7fb5e00f0b70 msgr2=0x7fb5e00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:08:22.492 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:22.465+0000 7fb5e81b7700 1 -- 172.21.15.178:0/1726458161 shutdown_connections 2024-02-11T17:08:22.492 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:22.465+0000 7fb5e81b7700 1 -- 172.21.15.178:0/1726458161 wait complete. 2024-02-11T17:08:22.492 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:22.465+0000 7fb5e81b7700 1 Processor -- start 2024-02-11T17:08:22.492 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:22.465+0000 7fb5e81b7700 1 -- start start 2024-02-11T17:08:22.492 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:22.465+0000 7fb5e81b7700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb5e00f5410 0x7fb5e0106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:08:22.492 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:22.465+0000 7fb5e81b7700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e0106bf0 con 0x7fb5e00f5410 2024-02-11T17:08:22.492 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:22.466+0000 7fb5e5f53700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb5e00f5410 0x7fb5e0106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:08:22.492 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:22.466+0000 7fb5e5f53700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb5e00f5410 0x7fb5e0106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:46388/0 (socket says 172.21.15.178:46388) 2024-02-11T17:08:22.492 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:22.466+0000 7fb5e5f53700 1 -- 172.21.15.178:0/998617727 learned_addr learned my addr 172.21.15.178:0/998617727 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:08:22.493 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:22.466+0000 7fb5e5f53700 1 -- 172.21.15.178:0/998617727 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5d0005040 con 0x7fb5e00f5410 2024-02-11T17:08:22.493 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:22.466+0000 7fb5e5f53700 1 --2- 172.21.15.178:0/998617727 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb5e00f5410 0x7fb5e0106700 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7fb5d0013510 tx=0x7fb5d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:08:22.493 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:22.466+0000 7fb5d67fc700 1 -- 172.21.15.178:0/998617727 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb5d0009320 con 0x7fb5e00f5410 2024-02-11T17:08:22.493 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:22.466+0000 7fb5e81b7700 1 -- 172.21.15.178:0/998617727 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5e0106df0 con 0x7fb5e00f5410 2024-02-11T17:08:22.493 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:22.466+0000 7fb5d67fc700 1 -- 172.21.15.178:0/998617727 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb5d0013f00 con 0x7fb5e00f5410 2024-02-11T17:08:22.493 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:22.466+0000 7fb5d67fc700 1 -- 172.21.15.178:0/998617727 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb5d0014d90 con 0x7fb5e00f5410 2024-02-11T17:08:22.493 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:22.466+0000 7fb5e81b7700 1 -- 172.21.15.178:0/998617727 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5e0107280 con 0x7fb5e00f5410 2024-02-11T17:08:22.493 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:22.467+0000 7fb5d67fc700 1 -- 172.21.15.178:0/998617727 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb5d00095f0 con 0x7fb5e00f5410 2024-02-11T17:08:22.493 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:22.467+0000 7fb5e81b7700 1 -- 172.21.15.178:0/998617727 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5e0045ad0 con 0x7fb5e00f5410 2024-02-11T17:08:22.493 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:22.467+0000 7fb5d67fc700 1 --2- 172.21.15.178:0/998617727 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb5cc02ff60 0x7fb5cc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:08:22.493 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:22.467+0000 7fb5d67fc700 1 -- 172.21.15.178:0/998617727 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb5d00410c0 con 0x7fb5e00f5410 2024-02-11T17:08:22.493 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:22.467+0000 7fb5e5752700 1 --2- 172.21.15.178:0/998617727 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb5cc02ff60 0x7fb5cc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:08:22.493 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:22.468+0000 7fb5e5752700 1 --2- 172.21.15.178:0/998617727 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb5cc02ff60 0x7fb5cc032420 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fb5dc0096a0 tx=0x7fb5dc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:08:22.493 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:22.472+0000 7fb5d67fc700 1 -- 172.21.15.178:0/998617727 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb5d0014420 con 0x7fb5e00f5410 2024-02-11T17:08:22.675 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:22.674+0000 7fb5e81b7700 1 -- 172.21.15.178:0/998617727 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb5e00fae60 con 0x7fb5e00f5410 2024-02-11T17:08:22.676 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:22.674+0000 7fb5d67fc700 1 -- 172.21.15.178:0/998617727 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb5d001d7f0 con 0x7fb5e00f5410 2024-02-11T17:08:22.676 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:08:22.676 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:08:22.678 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:22.676+0000 7fb5e81b7700 1 -- 172.21.15.178:0/998617727 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb5cc02ff60 msgr2=0x7fb5cc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:08:22.678 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:22.676+0000 7fb5e81b7700 1 --2- 172.21.15.178:0/998617727 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb5cc02ff60 0x7fb5cc032420 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fb5dc0096a0 tx=0x7fb5dc006b40).stop 2024-02-11T17:08:22.678 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:22.676+0000 7fb5e81b7700 1 -- 172.21.15.178:0/998617727 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb5e00f5410 msgr2=0x7fb5e0106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:08:22.678 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:22.676+0000 7fb5e81b7700 1 --2- 172.21.15.178:0/998617727 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb5e00f5410 0x7fb5e0106700 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7fb5d0013510 tx=0x7fb5d000b040).stop 2024-02-11T17:08:22.678 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:22.676+0000 7fb5e81b7700 1 -- 172.21.15.178:0/998617727 shutdown_connections 2024-02-11T17:08:22.678 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:22.676+0000 7fb5e81b7700 1 --2- 172.21.15.178:0/998617727 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb5cc02ff60 0x7fb5cc032420 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:08:22.678 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:22.676+0000 7fb5e81b7700 1 --2- 172.21.15.178:0/998617727 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb5e00f5410 0x7fb5e0106700 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:08:22.678 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:22.676+0000 7fb5e81b7700 1 -- 172.21.15.178:0/998617727 >> 172.21.15.178:0/998617727 conn(0x7fb5e00f0b70 msgr2=0x7fb5e00f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:08:22.678 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:22.676+0000 7fb5e81b7700 1 -- 172.21.15.178:0/998617727 shutdown_connections 2024-02-11T17:08:22.678 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:22.676+0000 7fb5e81b7700 1 -- 172.21.15.178:0/998617727 wait complete. 2024-02-11T17:08:22.679 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:08:22.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:08:22 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:08:21.206959+0000 mgr.smithi037.eofnaq (mgr.14162) 257 : cluster [DBG] pgmap v243: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:08:23.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:08:23 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:08:22.675724+0000 mon.smithi037 (mon.0) 264 : audit [DBG] from='client.? 172.21.15.178:0/998617727' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:08:24.489 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:08:24.489 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:08:24.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:08:24 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:08:23.207318+0000 mgr.smithi037.eofnaq (mgr.14162) 258 : cluster [DBG] pgmap v244: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:08:26.425 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:26.423+0000 7f7c8d7b3700 1 -- 172.21.15.178:0/2995536315 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c880f6a10 msgr2=0x7f7c880f6e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:08:26.425 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:26.423+0000 7f7c8d7b3700 1 --2- 172.21.15.178:0/2995536315 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c880f6a10 0x7f7c880f6e30 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7f7c7c009fb0 tx=0x7f7c7c009b30).stop 2024-02-11T17:08:26.426 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:26.423+0000 7f7c8d7b3700 1 -- 172.21.15.178:0/2995536315 shutdown_connections 2024-02-11T17:08:26.426 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:26.423+0000 7f7c8d7b3700 1 --2- 172.21.15.178:0/2995536315 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c880f6a10 0x7f7c880f6e30 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:08:26.426 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:26.423+0000 7f7c8d7b3700 1 -- 172.21.15.178:0/2995536315 >> 172.21.15.178:0/2995536315 conn(0x7f7c880f02e0 msgr2=0x7f7c880f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:08:26.426 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:26.423+0000 7f7c8d7b3700 1 -- 172.21.15.178:0/2995536315 shutdown_connections 2024-02-11T17:08:26.426 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:26.423+0000 7f7c8d7b3700 1 -- 172.21.15.178:0/2995536315 wait complete. 2024-02-11T17:08:26.427 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:26.424+0000 7f7c8d7b3700 1 Processor -- start 2024-02-11T17:08:26.427 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:26.424+0000 7f7c8d7b3700 1 -- start start 2024-02-11T17:08:26.427 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:26.424+0000 7f7c8d7b3700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c880f6a10 0x7f7c88109390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:08:26.428 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:26.424+0000 7f7c8d7b3700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c880f5390 con 0x7f7c880f6a10 2024-02-11T17:08:26.428 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:26.425+0000 7f7c86ffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c880f6a10 0x7f7c88109390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:08:26.428 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:26.425+0000 7f7c86ffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c880f6a10 0x7f7c88109390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:46402/0 (socket says 172.21.15.178:46402) 2024-02-11T17:08:26.428 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:26.425+0000 7f7c86ffd700 1 -- 172.21.15.178:0/2905142496 learned_addr learned my addr 172.21.15.178:0/2905142496 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:08:26.428 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:26.425+0000 7f7c86ffd700 1 -- 172.21.15.178:0/2905142496 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c7c005040 con 0x7f7c880f6a10 2024-02-11T17:08:26.428 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:26.425+0000 7f7c86ffd700 1 --2- 172.21.15.178:0/2905142496 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c880f6a10 0x7f7c88109390 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f7c7c014000 tx=0x7f7c7c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:08:26.428 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:26.425+0000 7f7c6f7fe700 1 -- 172.21.15.178:0/2905142496 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c7c0095f0 con 0x7f7c880f6a10 2024-02-11T17:08:26.428 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:26.425+0000 7f7c8d7b3700 1 -- 172.21.15.178:0/2905142496 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c880f5590 con 0x7f7c880f6a10 2024-02-11T17:08:26.428 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:26.425+0000 7f7c6f7fe700 1 -- 172.21.15.178:0/2905142496 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7c7c007e70 con 0x7f7c880f6a10 2024-02-11T17:08:26.428 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:26.425+0000 7f7c6f7fe700 1 -- 172.21.15.178:0/2905142496 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c7c014ea0 con 0x7f7c880f6a10 2024-02-11T17:08:26.428 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:26.425+0000 7f7c8d7b3700 1 -- 172.21.15.178:0/2905142496 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c88109ac0 con 0x7f7c880f6a10 2024-02-11T17:08:26.429 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:26.426+0000 7f7c6f7fe700 1 -- 172.21.15.178:0/2905142496 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7c7c025070 con 0x7f7c880f6a10 2024-02-11T17:08:26.429 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:26.426+0000 7f7c8d7b3700 1 -- 172.21.15.178:0/2905142496 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c88045ad0 con 0x7f7c880f6a10 2024-02-11T17:08:26.429 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:26.426+0000 7f7c6f7fe700 1 --2- 172.21.15.178:0/2905142496 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f7c7002ff60 0x7f7c70032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:08:26.429 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:26.426+0000 7f7c6f7fe700 1 -- 172.21.15.178:0/2905142496 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7c7c040920 con 0x7f7c880f6a10 2024-02-11T17:08:26.429 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:26.427+0000 7f7c867fc700 1 --2- 172.21.15.178:0/2905142496 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f7c7002ff60 0x7f7c70032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:08:26.430 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:26.427+0000 7f7c867fc700 1 --2- 172.21.15.178:0/2905142496 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f7c7002ff60 0x7f7c70032420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f7c780096a0 tx=0x7f7c78006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:08:26.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:08:26 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:08:25.207737+0000 mgr.smithi037.eofnaq (mgr.14162) 259 : cluster [DBG] pgmap v245: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:08:26.433 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:26.431+0000 7f7c6f7fe700 1 -- 172.21.15.178:0/2905142496 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7c7c00f4e0 con 0x7f7c880f6a10 2024-02-11T17:08:26.641 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:26.639+0000 7f7c8d7b3700 1 -- 172.21.15.178:0/2905142496 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7c880fae60 con 0x7f7c880f6a10 2024-02-11T17:08:26.642 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:26.640+0000 7f7c6f7fe700 1 -- 172.21.15.178:0/2905142496 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7c7c00f300 con 0x7f7c880f6a10 2024-02-11T17:08:26.642 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:08:26.642 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:08:26.644 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:26.642+0000 7f7c8d7b3700 1 -- 172.21.15.178:0/2905142496 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f7c7002ff60 msgr2=0x7f7c70032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:08:26.644 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:26.642+0000 7f7c8d7b3700 1 --2- 172.21.15.178:0/2905142496 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f7c7002ff60 0x7f7c70032420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f7c780096a0 tx=0x7f7c78006b40).stop 2024-02-11T17:08:26.644 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:26.642+0000 7f7c8d7b3700 1 -- 172.21.15.178:0/2905142496 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c880f6a10 msgr2=0x7f7c88109390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:08:26.644 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:26.642+0000 7f7c8d7b3700 1 --2- 172.21.15.178:0/2905142496 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c880f6a10 0x7f7c88109390 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f7c7c014000 tx=0x7f7c7c00b040).stop 2024-02-11T17:08:26.644 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:26.642+0000 7f7c8d7b3700 1 -- 172.21.15.178:0/2905142496 shutdown_connections 2024-02-11T17:08:26.644 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:26.642+0000 7f7c8d7b3700 1 --2- 172.21.15.178:0/2905142496 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f7c7002ff60 0x7f7c70032420 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:08:26.644 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:26.642+0000 7f7c8d7b3700 1 --2- 172.21.15.178:0/2905142496 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c880f6a10 0x7f7c88109390 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:08:26.644 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:26.642+0000 7f7c8d7b3700 1 -- 172.21.15.178:0/2905142496 >> 172.21.15.178:0/2905142496 conn(0x7f7c880f02e0 msgr2=0x7f7c880f0da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:08:26.644 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:26.642+0000 7f7c8d7b3700 1 -- 172.21.15.178:0/2905142496 shutdown_connections 2024-02-11T17:08:26.644 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:26.642+0000 7f7c8d7b3700 1 -- 172.21.15.178:0/2905142496 wait complete. 2024-02-11T17:08:26.645 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:08:27.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:08:27 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:08:26.641590+0000 mon.smithi037 (mon.0) 265 : audit [DBG] from='client.? 172.21.15.178:0/2905142496' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:08:28.397 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:08:28.397 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:08:28.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:08:28 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:08:27.208031+0000 mgr.smithi037.eofnaq (mgr.14162) 260 : cluster [DBG] pgmap v246: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:08:30.374 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:30.371+0000 7f60212ad700 1 -- 172.21.15.178:0/78756092 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f601c0f4860 msgr2=0x7f601c0f4c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:08:30.374 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:30.371+0000 7f60212ad700 1 --2- 172.21.15.178:0/78756092 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f601c0f4860 0x7f601c0f4c80 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7f6004009fb0 tx=0x7f6004009b30).stop 2024-02-11T17:08:30.374 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:30.371+0000 7f60212ad700 1 -- 172.21.15.178:0/78756092 shutdown_connections 2024-02-11T17:08:30.375 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:30.371+0000 7f60212ad700 1 --2- 172.21.15.178:0/78756092 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f601c0f4860 0x7f601c0f4c80 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:08:30.375 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:30.371+0000 7f60212ad700 1 -- 172.21.15.178:0/78756092 >> 172.21.15.178:0/78756092 conn(0x7f601c0f02e0 msgr2=0x7f601c0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:08:30.375 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:30.371+0000 7f60212ad700 1 -- 172.21.15.178:0/78756092 shutdown_connections 2024-02-11T17:08:30.375 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:30.371+0000 7f60212ad700 1 -- 172.21.15.178:0/78756092 wait complete. 2024-02-11T17:08:30.375 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:30.372+0000 7f60212ad700 1 Processor -- start 2024-02-11T17:08:30.375 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:30.372+0000 7f60212ad700 1 -- start start 2024-02-11T17:08:30.375 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:30.372+0000 7f60212ad700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f601c0f4860 0x7f601c1068e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:08:30.375 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:30.372+0000 7f60212ad700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f601c106dd0 con 0x7f601c0f4860 2024-02-11T17:08:30.375 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:30.373+0000 7f601affd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f601c0f4860 0x7f601c1068e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:08:30.375 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:30.373+0000 7f601affd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f601c0f4860 0x7f601c1068e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:38334/0 (socket says 172.21.15.178:38334) 2024-02-11T17:08:30.375 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:30.373+0000 7f601affd700 1 -- 172.21.15.178:0/3681707082 learned_addr learned my addr 172.21.15.178:0/3681707082 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:08:30.375 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:30.373+0000 7f601affd700 1 -- 172.21.15.178:0/3681707082 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6004005040 con 0x7f601c0f4860 2024-02-11T17:08:30.375 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:30.373+0000 7f601affd700 1 --2- 172.21.15.178:0/3681707082 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f601c0f4860 0x7f601c1068e0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7f6004013510 tx=0x7f600400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:08:30.377 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:30.373+0000 7f60137fe700 1 -- 172.21.15.178:0/3681707082 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6004009320 con 0x7f601c0f4860 2024-02-11T17:08:30.377 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:30.373+0000 7f60212ad700 1 -- 172.21.15.178:0/3681707082 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f601c106fd0 con 0x7f601c0f4860 2024-02-11T17:08:30.377 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:30.373+0000 7f60137fe700 1 -- 172.21.15.178:0/3681707082 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6004013f00 con 0x7f601c0f4860 2024-02-11T17:08:30.377 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:30.373+0000 7f60137fe700 1 -- 172.21.15.178:0/3681707082 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6004014dc0 con 0x7f601c0f4860 2024-02-11T17:08:30.377 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:30.373+0000 7f60212ad700 1 -- 172.21.15.178:0/3681707082 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f601c107460 con 0x7f601c0f4860 2024-02-11T17:08:30.377 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:30.374+0000 7f60137fe700 1 -- 172.21.15.178:0/3681707082 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6004014420 con 0x7f601c0f4860 2024-02-11T17:08:30.377 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:30.374+0000 7f60212ad700 1 -- 172.21.15.178:0/3681707082 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f601c045ad0 con 0x7f601c0f4860 2024-02-11T17:08:30.377 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:30.374+0000 7f60137fe700 1 --2- 172.21.15.178:0/3681707082 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f600802ff60 0x7f6008032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:08:30.377 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:30.374+0000 7f60137fe700 1 -- 172.21.15.178:0/3681707082 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f60040410f0 con 0x7f601c0f4860 2024-02-11T17:08:30.377 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:30.374+0000 7f601a7fc700 1 --2- 172.21.15.178:0/3681707082 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f600802ff60 0x7f6008032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:08:30.377 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:30.375+0000 7f601a7fc700 1 --2- 172.21.15.178:0/3681707082 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f600802ff60 0x7f6008032420 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f600c0096a0 tx=0x7f600c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:08:30.381 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:30.379+0000 7f60137fe700 1 -- 172.21.15.178:0/3681707082 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f60040095f0 con 0x7f601c0f4860 2024-02-11T17:08:30.584 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:30.583+0000 7f60212ad700 1 -- 172.21.15.178:0/3681707082 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f601c05f370 con 0x7f601c0f4860 2024-02-11T17:08:30.585 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:30.583+0000 7f60137fe700 1 -- 172.21.15.178:0/3681707082 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f600401cb10 con 0x7f601c0f4860 2024-02-11T17:08:30.585 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:08:30.585 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:08:30.587 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:30.585+0000 7f60212ad700 1 -- 172.21.15.178:0/3681707082 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f600802ff60 msgr2=0x7f6008032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:08:30.587 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:30.585+0000 7f60212ad700 1 --2- 172.21.15.178:0/3681707082 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f600802ff60 0x7f6008032420 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f600c0096a0 tx=0x7f600c006b40).stop 2024-02-11T17:08:30.587 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:30.585+0000 7f60212ad700 1 -- 172.21.15.178:0/3681707082 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f601c0f4860 msgr2=0x7f601c1068e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:08:30.587 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:30.585+0000 7f60212ad700 1 --2- 172.21.15.178:0/3681707082 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f601c0f4860 0x7f601c1068e0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7f6004013510 tx=0x7f600400b040).stop 2024-02-11T17:08:30.587 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:30.585+0000 7f60212ad700 1 -- 172.21.15.178:0/3681707082 shutdown_connections 2024-02-11T17:08:30.587 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:30.585+0000 7f60212ad700 1 --2- 172.21.15.178:0/3681707082 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f600802ff60 0x7f6008032420 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:08:30.587 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:30.585+0000 7f60212ad700 1 --2- 172.21.15.178:0/3681707082 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f601c0f4860 0x7f601c1068e0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:08:30.587 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:30.585+0000 7f60212ad700 1 -- 172.21.15.178:0/3681707082 >> 172.21.15.178:0/3681707082 conn(0x7f601c0f02e0 msgr2=0x7f601c0f0e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:08:30.587 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:30.585+0000 7f60212ad700 1 -- 172.21.15.178:0/3681707082 shutdown_connections 2024-02-11T17:08:30.587 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:30.585+0000 7f60212ad700 1 -- 172.21.15.178:0/3681707082 wait complete. 2024-02-11T17:08:30.588 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:08:30.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:08:30 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:08:29.208257+0000 mgr.smithi037.eofnaq (mgr.14162) 261 : cluster [DBG] pgmap v247: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:08:31.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:08:31 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:08:30.584824+0000 mon.smithi037 (mon.0) 266 : audit [DBG] from='client.? 172.21.15.178:0/3681707082' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:08:32.294 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:08:32.294 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:08:32.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:08:32 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:08:31.208629+0000 mgr.smithi037.eofnaq (mgr.14162) 262 : cluster [DBG] pgmap v248: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:08:34.172 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:34.169+0000 7f31172c2700 1 -- 172.21.15.178:0/3692840346 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31100f51a0 msgr2=0x7f31100f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:08:34.172 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:34.169+0000 7f31172c2700 1 --2- 172.21.15.178:0/3692840346 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31100f51a0 0x7f31100f55c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7f310c009fb0 tx=0x7f310c009b30).stop 2024-02-11T17:08:34.172 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:34.169+0000 7f31172c2700 1 -- 172.21.15.178:0/3692840346 shutdown_connections 2024-02-11T17:08:34.173 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:34.169+0000 7f31172c2700 1 --2- 172.21.15.178:0/3692840346 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31100f51a0 0x7f31100f55c0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:08:34.173 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:34.169+0000 7f31172c2700 1 -- 172.21.15.178:0/3692840346 >> 172.21.15.178:0/3692840346 conn(0x7f31100f09a0 msgr2=0x7f31100f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:08:34.173 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:34.169+0000 7f31172c2700 1 -- 172.21.15.178:0/3692840346 shutdown_connections 2024-02-11T17:08:34.173 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:34.169+0000 7f31172c2700 1 -- 172.21.15.178:0/3692840346 wait complete. 2024-02-11T17:08:34.173 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:34.169+0000 7f31172c2700 1 Processor -- start 2024-02-11T17:08:34.173 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:34.169+0000 7f31172c2700 1 -- start start 2024-02-11T17:08:34.173 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:34.170+0000 7f31172c2700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31100f51a0 0x7f31101053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:08:34.173 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:34.170+0000 7f31172c2700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31101058a0 con 0x7f31100f51a0 2024-02-11T17:08:34.173 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:34.170+0000 7f31162c0700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31100f51a0 0x7f31101053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:08:34.173 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:34.170+0000 7f31162c0700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31100f51a0 0x7f31101053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:38346/0 (socket says 172.21.15.178:38346) 2024-02-11T17:08:34.173 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:34.170+0000 7f31162c0700 1 -- 172.21.15.178:0/3097923890 learned_addr learned my addr 172.21.15.178:0/3097923890 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:08:34.174 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:34.170+0000 7f31162c0700 1 -- 172.21.15.178:0/3097923890 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f310c005040 con 0x7f31100f51a0 2024-02-11T17:08:34.174 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:34.170+0000 7f31162c0700 1 --2- 172.21.15.178:0/3097923890 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31100f51a0 0x7f31101053b0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f310c009830 tx=0x7f310c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:08:34.174 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:34.170+0000 7f3106ffd700 1 -- 172.21.15.178:0/3097923890 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f310c009320 con 0x7f31100f51a0 2024-02-11T17:08:34.174 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:34.171+0000 7f3106ffd700 1 -- 172.21.15.178:0/3097923890 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f310c0136c0 con 0x7f31100f51a0 2024-02-11T17:08:34.174 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:34.171+0000 7f3106ffd700 1 -- 172.21.15.178:0/3097923890 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f310c01ee90 con 0x7f31100f51a0 2024-02-11T17:08:34.174 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:34.171+0000 7f31172c2700 1 -- 172.21.15.178:0/3097923890 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3110105aa0 con 0x7f31100f51a0 2024-02-11T17:08:34.174 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:34.171+0000 7f31172c2700 1 -- 172.21.15.178:0/3097923890 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3110105f30 con 0x7f31100f51a0 2024-02-11T17:08:34.176 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:34.171+0000 7f3106ffd700 1 -- 172.21.15.178:0/3097923890 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f310c01e420 con 0x7f31100f51a0 2024-02-11T17:08:34.176 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:34.171+0000 7f31172c2700 1 -- 172.21.15.178:0/3097923890 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3110045ad0 con 0x7f31100f51a0 2024-02-11T17:08:34.176 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:34.172+0000 7f3106ffd700 1 --2- 172.21.15.178:0/3097923890 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f30fc02ff10 0x7f30fc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:08:34.176 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:34.172+0000 7f3106ffd700 1 -- 172.21.15.178:0/3097923890 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f310c040f70 con 0x7f31100f51a0 2024-02-11T17:08:34.177 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:34.172+0000 7f3115abf700 1 --2- 172.21.15.178:0/3097923890 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f30fc02ff10 0x7f30fc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:08:34.177 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:34.172+0000 7f3115abf700 1 --2- 172.21.15.178:0/3097923890 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f30fc02ff10 0x7f30fc0323d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f31000096a0 tx=0x7f3100006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:08:34.178 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:34.177+0000 7f3106ffd700 1 -- 172.21.15.178:0/3097923890 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f310c06d2b0 con 0x7f31100f51a0 2024-02-11T17:08:34.382 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:34.380+0000 7f31172c2700 1 -- 172.21.15.178:0/3097923890 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3110106de0 con 0x7f31100f51a0 2024-02-11T17:08:34.383 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:34.381+0000 7f3106ffd700 1 -- 172.21.15.178:0/3097923890 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f310c02c2d0 con 0x7f31100f51a0 2024-02-11T17:08:34.383 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:08:34.383 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:08:34.384 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:34.383+0000 7f31172c2700 1 -- 172.21.15.178:0/3097923890 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f30fc02ff10 msgr2=0x7f30fc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:08:34.384 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:34.383+0000 7f31172c2700 1 --2- 172.21.15.178:0/3097923890 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f30fc02ff10 0x7f30fc0323d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f31000096a0 tx=0x7f3100006b40).stop 2024-02-11T17:08:34.384 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:34.383+0000 7f31172c2700 1 -- 172.21.15.178:0/3097923890 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31100f51a0 msgr2=0x7f31101053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:08:34.385 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:34.383+0000 7f31172c2700 1 --2- 172.21.15.178:0/3097923890 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31100f51a0 0x7f31101053b0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f310c009830 tx=0x7f310c00b040).stop 2024-02-11T17:08:34.385 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:34.383+0000 7f31172c2700 1 -- 172.21.15.178:0/3097923890 shutdown_connections 2024-02-11T17:08:34.385 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:34.383+0000 7f31172c2700 1 --2- 172.21.15.178:0/3097923890 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f30fc02ff10 0x7f30fc0323d0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:08:34.385 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:34.383+0000 7f31172c2700 1 --2- 172.21.15.178:0/3097923890 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31100f51a0 0x7f31101053b0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:08:34.385 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:34.383+0000 7f31172c2700 1 -- 172.21.15.178:0/3097923890 >> 172.21.15.178:0/3097923890 conn(0x7f31100f09a0 msgr2=0x7f31100f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:08:34.385 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:34.383+0000 7f31172c2700 1 -- 172.21.15.178:0/3097923890 shutdown_connections 2024-02-11T17:08:34.385 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:34.383+0000 7f31172c2700 1 -- 172.21.15.178:0/3097923890 wait complete. 2024-02-11T17:08:34.385 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:08:34.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:08:34 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:08:33.208966+0000 mgr.smithi037.eofnaq (mgr.14162) 263 : cluster [DBG] pgmap v249: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:08:35.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:08:35 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:08:34.382418+0000 mon.smithi037 (mon.0) 267 : audit [DBG] from='client.? 172.21.15.178:0/3097923890' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:08:36.156 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:08:36.156 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:08:36.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:08:36 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:08:35.209358+0000 mgr.smithi037.eofnaq (mgr.14162) 264 : cluster [DBG] pgmap v250: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:08:38.156 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:38.154+0000 7f1d9892f700 1 -- 172.21.15.178:0/249965289 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d940f5410 msgr2=0x7f1d940f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:08:38.156 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:38.154+0000 7f1d9892f700 1 --2- 172.21.15.178:0/249965289 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d940f5410 0x7f1d940f5830 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7f1d84009fb0 tx=0x7f1d84009b30).stop 2024-02-11T17:08:38.156 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:38.154+0000 7f1d9892f700 1 -- 172.21.15.178:0/249965289 shutdown_connections 2024-02-11T17:08:38.156 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:38.154+0000 7f1d9892f700 1 --2- 172.21.15.178:0/249965289 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d940f5410 0x7f1d940f5830 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:08:38.156 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:38.154+0000 7f1d9892f700 1 -- 172.21.15.178:0/249965289 >> 172.21.15.178:0/249965289 conn(0x7f1d940f0b70 msgr2=0x7f1d940f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:08:38.157 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:38.154+0000 7f1d9892f700 1 -- 172.21.15.178:0/249965289 shutdown_connections 2024-02-11T17:08:38.157 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:38.154+0000 7f1d9892f700 1 -- 172.21.15.178:0/249965289 wait complete. 2024-02-11T17:08:38.157 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:38.155+0000 7f1d9892f700 1 Processor -- start 2024-02-11T17:08:38.157 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:38.155+0000 7f1d9892f700 1 -- start start 2024-02-11T17:08:38.157 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:38.155+0000 7f1d9892f700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d940f5410 0x7f1d94109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:08:38.157 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:38.155+0000 7f1d9892f700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d94060240 con 0x7f1d940f5410 2024-02-11T17:08:38.157 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:38.155+0000 7f1d9259c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d940f5410 0x7f1d94109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:08:38.157 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:38.155+0000 7f1d9259c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d940f5410 0x7f1d94109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:38374/0 (socket says 172.21.15.178:38374) 2024-02-11T17:08:38.157 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:38.155+0000 7f1d9259c700 1 -- 172.21.15.178:0/2049543341 learned_addr learned my addr 172.21.15.178:0/2049543341 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:08:38.157 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:38.155+0000 7f1d9259c700 1 -- 172.21.15.178:0/2049543341 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d84005040 con 0x7f1d940f5410 2024-02-11T17:08:38.157 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:38.155+0000 7f1d9259c700 1 --2- 172.21.15.178:0/2049543341 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d940f5410 0x7f1d94109430 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f1d84014000 tx=0x7f1d8400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:08:38.157 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:38.156+0000 7f1d82ffd700 1 -- 172.21.15.178:0/2049543341 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d840095f0 con 0x7f1d940f5410 2024-02-11T17:08:38.157 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:38.156+0000 7f1d9892f700 1 -- 172.21.15.178:0/2049543341 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d94060440 con 0x7f1d940f5410 2024-02-11T17:08:38.157 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:38.156+0000 7f1d82ffd700 1 -- 172.21.15.178:0/2049543341 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1d84007e70 con 0x7f1d940f5410 2024-02-11T17:08:38.158 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:38.156+0000 7f1d82ffd700 1 -- 172.21.15.178:0/2049543341 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d84014ea0 con 0x7f1d940f5410 2024-02-11T17:08:38.158 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:38.156+0000 7f1d9892f700 1 -- 172.21.15.178:0/2049543341 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d94109ae0 con 0x7f1d940f5410 2024-02-11T17:08:38.158 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:38.156+0000 7f1d82ffd700 1 -- 172.21.15.178:0/2049543341 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1d84025070 con 0x7f1d940f5410 2024-02-11T17:08:38.158 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:38.157+0000 7f1d82ffd700 1 --2- 172.21.15.178:0/2049543341 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f1d7c02ff60 0x7f1d7c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:08:38.159 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:38.157+0000 7f1d9892f700 1 -- 172.21.15.178:0/2049543341 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d94045ad0 con 0x7f1d940f5410 2024-02-11T17:08:38.159 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:38.157+0000 7f1d82ffd700 1 -- 172.21.15.178:0/2049543341 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1d84040c90 con 0x7f1d940f5410 2024-02-11T17:08:38.159 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:38.157+0000 7f1d91d9b700 1 --2- 172.21.15.178:0/2049543341 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f1d7c02ff60 0x7f1d7c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:08:38.159 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:38.157+0000 7f1d91d9b700 1 --2- 172.21.15.178:0/2049543341 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f1d7c02ff60 0x7f1d7c032420 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f1d880096a0 tx=0x7f1d88006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:08:38.162 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:38.161+0000 7f1d82ffd700 1 -- 172.21.15.178:0/2049543341 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1d840079c0 con 0x7f1d940f5410 2024-02-11T17:08:38.367 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:38.367+0000 7f1d9892f700 1 -- 172.21.15.178:0/2049543341 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1d940faf50 con 0x7f1d940f5410 2024-02-11T17:08:38.368 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:38.367+0000 7f1d82ffd700 1 -- 172.21.15.178:0/2049543341 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1d84012330 con 0x7f1d940f5410 2024-02-11T17:08:38.368 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:08:38.368 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:08:38.370 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:38.369+0000 7f1d9892f700 1 -- 172.21.15.178:0/2049543341 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f1d7c02ff60 msgr2=0x7f1d7c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:08:38.370 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:38.369+0000 7f1d9892f700 1 --2- 172.21.15.178:0/2049543341 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f1d7c02ff60 0x7f1d7c032420 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f1d880096a0 tx=0x7f1d88006b40).stop 2024-02-11T17:08:38.370 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:38.369+0000 7f1d9892f700 1 -- 172.21.15.178:0/2049543341 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d940f5410 msgr2=0x7f1d94109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:08:38.370 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:38.369+0000 7f1d9892f700 1 --2- 172.21.15.178:0/2049543341 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d940f5410 0x7f1d94109430 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f1d84014000 tx=0x7f1d8400b040).stop 2024-02-11T17:08:38.370 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:38.369+0000 7f1d9892f700 1 -- 172.21.15.178:0/2049543341 shutdown_connections 2024-02-11T17:08:38.371 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:38.369+0000 7f1d9892f700 1 --2- 172.21.15.178:0/2049543341 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f1d7c02ff60 0x7f1d7c032420 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:08:38.371 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:38.369+0000 7f1d9892f700 1 --2- 172.21.15.178:0/2049543341 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d940f5410 0x7f1d94109430 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:08:38.371 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:38.369+0000 7f1d9892f700 1 -- 172.21.15.178:0/2049543341 >> 172.21.15.178:0/2049543341 conn(0x7f1d940f0b70 msgr2=0x7f1d940f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:08:38.371 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:38.369+0000 7f1d9892f700 1 -- 172.21.15.178:0/2049543341 shutdown_connections 2024-02-11T17:08:38.371 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:38.369+0000 7f1d9892f700 1 -- 172.21.15.178:0/2049543341 wait complete. 2024-02-11T17:08:38.371 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:08:38.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:08:38 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:08:37.209755+0000 mgr.smithi037.eofnaq (mgr.14162) 265 : cluster [DBG] pgmap v251: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:08:39.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:08:39 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:08:38.367815+0000 mon.smithi037 (mon.0) 268 : audit [DBG] from='client.? 172.21.15.178:0/2049543341' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:08:39.982 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:08:39.983 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:08:40.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:08:40 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:08:39.210050+0000 mgr.smithi037.eofnaq (mgr.14162) 266 : cluster [DBG] pgmap v252: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:08:41.875 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:41.872+0000 7f77c7289700 1 -- 172.21.15.178:0/2070005955 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f77c00f47e0 msgr2=0x7f77c00f4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:08:41.875 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:41.872+0000 7f77c7289700 1 --2- 172.21.15.178:0/2070005955 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f77c00f47e0 0x7f77c00f4c00 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f77bc009fb0 tx=0x7f77bc009b30).stop 2024-02-11T17:08:41.875 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:41.872+0000 7f77c7289700 1 -- 172.21.15.178:0/2070005955 shutdown_connections 2024-02-11T17:08:41.876 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:41.872+0000 7f77c7289700 1 --2- 172.21.15.178:0/2070005955 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f77c00f47e0 0x7f77c00f4c00 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:08:41.876 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:41.872+0000 7f77c7289700 1 -- 172.21.15.178:0/2070005955 >> 172.21.15.178:0/2070005955 conn(0x7f77c00f0260 msgr2=0x7f77c00f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:08:41.876 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:41.872+0000 7f77c7289700 1 -- 172.21.15.178:0/2070005955 shutdown_connections 2024-02-11T17:08:41.876 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:41.872+0000 7f77c7289700 1 -- 172.21.15.178:0/2070005955 wait complete. 2024-02-11T17:08:41.876 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:41.872+0000 7f77c7289700 1 Processor -- start 2024-02-11T17:08:41.876 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:41.872+0000 7f77c7289700 1 -- start start 2024-02-11T17:08:41.876 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:41.873+0000 7f77c7289700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f77c00f47e0 0x7f77c01068d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:08:41.876 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:41.873+0000 7f77c7289700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77c0106dc0 con 0x7f77c00f47e0 2024-02-11T17:08:41.876 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:41.873+0000 7f77c5025700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f77c00f47e0 0x7f77c01068d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:08:41.876 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:41.873+0000 7f77c5025700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f77c00f47e0 0x7f77c01068d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:60528/0 (socket says 172.21.15.178:60528) 2024-02-11T17:08:41.876 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:41.873+0000 7f77c5025700 1 -- 172.21.15.178:0/4284096617 learned_addr learned my addr 172.21.15.178:0/4284096617 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:08:41.876 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:41.873+0000 7f77c5025700 1 -- 172.21.15.178:0/4284096617 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77bc005040 con 0x7f77c00f47e0 2024-02-11T17:08:41.877 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:41.873+0000 7f77c5025700 1 --2- 172.21.15.178:0/4284096617 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f77c00f47e0 0x7f77c01068d0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f77bc013510 tx=0x7f77bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:08:41.877 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:41.873+0000 7f77b5ffb700 1 -- 172.21.15.178:0/4284096617 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f77bc009320 con 0x7f77c00f47e0 2024-02-11T17:08:41.877 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:41.873+0000 7f77c7289700 1 -- 172.21.15.178:0/4284096617 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77c0106fc0 con 0x7f77c00f47e0 2024-02-11T17:08:41.877 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:41.873+0000 7f77b5ffb700 1 -- 172.21.15.178:0/4284096617 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f77bc013f00 con 0x7f77c00f47e0 2024-02-11T17:08:41.877 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:41.873+0000 7f77b5ffb700 1 -- 172.21.15.178:0/4284096617 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f77bc014dc0 con 0x7f77c00f47e0 2024-02-11T17:08:41.877 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:41.874+0000 7f77c7289700 1 -- 172.21.15.178:0/4284096617 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77c0107450 con 0x7f77c00f47e0 2024-02-11T17:08:41.877 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:41.874+0000 7f77b5ffb700 1 -- 172.21.15.178:0/4284096617 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f77bc0095f0 con 0x7f77c00f47e0 2024-02-11T17:08:41.877 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:41.874+0000 7f77c7289700 1 -- 172.21.15.178:0/4284096617 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77c0045ad0 con 0x7f77c00f47e0 2024-02-11T17:08:41.877 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:41.875+0000 7f77b5ffb700 1 --2- 172.21.15.178:0/4284096617 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f77ac02ff60 0x7f77ac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:08:41.877 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:41.875+0000 7f77b5ffb700 1 -- 172.21.15.178:0/4284096617 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f77bc041040 con 0x7f77c00f47e0 2024-02-11T17:08:41.877 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:41.875+0000 7f77c4824700 1 --2- 172.21.15.178:0/4284096617 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f77ac02ff60 0x7f77ac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:08:41.878 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:41.875+0000 7f77c4824700 1 --2- 172.21.15.178:0/4284096617 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f77ac02ff60 0x7f77ac032420 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f77b00096a0 tx=0x7f77b0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:08:41.880 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:41.879+0000 7f77b5ffb700 1 -- 172.21.15.178:0/4284096617 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f77bc014420 con 0x7f77c00f47e0 2024-02-11T17:08:42.084 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:42.083+0000 7f77c7289700 1 -- 172.21.15.178:0/4284096617 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f77c005eb50 con 0x7f77c00f47e0 2024-02-11T17:08:42.084 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:42.083+0000 7f77b5ffb700 1 -- 172.21.15.178:0/4284096617 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f77bc010230 con 0x7f77c00f47e0 2024-02-11T17:08:42.084 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:08:42.085 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:08:42.085 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:42.085+0000 7f77c7289700 1 -- 172.21.15.178:0/4284096617 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f77ac02ff60 msgr2=0x7f77ac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:08:42.086 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:42.085+0000 7f77c7289700 1 --2- 172.21.15.178:0/4284096617 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f77ac02ff60 0x7f77ac032420 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f77b00096a0 tx=0x7f77b0006b40).stop 2024-02-11T17:08:42.086 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:42.085+0000 7f77c7289700 1 -- 172.21.15.178:0/4284096617 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f77c00f47e0 msgr2=0x7f77c01068d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:08:42.086 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:42.085+0000 7f77c7289700 1 --2- 172.21.15.178:0/4284096617 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f77c00f47e0 0x7f77c01068d0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f77bc013510 tx=0x7f77bc00b040).stop 2024-02-11T17:08:42.086 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:42.085+0000 7f77c7289700 1 -- 172.21.15.178:0/4284096617 shutdown_connections 2024-02-11T17:08:42.086 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:42.085+0000 7f77c7289700 1 --2- 172.21.15.178:0/4284096617 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f77ac02ff60 0x7f77ac032420 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:08:42.086 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:42.085+0000 7f77c7289700 1 --2- 172.21.15.178:0/4284096617 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f77c00f47e0 0x7f77c01068d0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:08:42.086 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:42.085+0000 7f77c7289700 1 -- 172.21.15.178:0/4284096617 >> 172.21.15.178:0/4284096617 conn(0x7f77c00f0260 msgr2=0x7f77c00f0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:08:42.086 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:42.085+0000 7f77c7289700 1 -- 172.21.15.178:0/4284096617 shutdown_connections 2024-02-11T17:08:42.086 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:42.085+0000 7f77c7289700 1 -- 172.21.15.178:0/4284096617 wait complete. 2024-02-11T17:08:42.086 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:08:42.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:08:42 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:08:41.210427+0000 mgr.smithi037.eofnaq (mgr.14162) 267 : cluster [DBG] pgmap v253: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:08:42.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:08:42 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:08:42.083656+0000 mon.smithi037 (mon.0) 269 : audit [DBG] from='client.? 172.21.15.178:0/4284096617' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:08:43.798 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:08:43.798 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:08:44.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:08:44 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:08:43.210837+0000 mgr.smithi037.eofnaq (mgr.14162) 268 : cluster [DBG] pgmap v254: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:08:45.829 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:45.827+0000 7f0a7409a700 1 -- 172.21.15.178:0/777628808 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0a6c0f6510 msgr2=0x7f0a6c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:08:45.829 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:45.827+0000 7f0a7409a700 1 --2- 172.21.15.178:0/777628808 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0a6c0f6510 0x7f0a6c0f6930 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f0a5c009fb0 tx=0x7f0a5c009b30).stop 2024-02-11T17:08:45.829 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:45.827+0000 7f0a7409a700 1 -- 172.21.15.178:0/777628808 shutdown_connections 2024-02-11T17:08:45.829 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:45.827+0000 7f0a7409a700 1 --2- 172.21.15.178:0/777628808 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0a6c0f6510 0x7f0a6c0f6930 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:08:45.829 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:45.827+0000 7f0a7409a700 1 -- 172.21.15.178:0/777628808 >> 172.21.15.178:0/777628808 conn(0x7f0a6c0f1c70 msgr2=0x7f0a6c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:08:45.829 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:45.827+0000 7f0a7409a700 1 -- 172.21.15.178:0/777628808 shutdown_connections 2024-02-11T17:08:45.830 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:45.827+0000 7f0a7409a700 1 -- 172.21.15.178:0/777628808 wait complete. 2024-02-11T17:08:45.831 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:45.828+0000 7f0a7409a700 1 Processor -- start 2024-02-11T17:08:45.831 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:45.828+0000 7f0a7409a700 1 -- start start 2024-02-11T17:08:45.831 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:45.828+0000 7f0a7409a700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0a6c0f6510 0x7f0a6c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:08:45.831 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:45.828+0000 7f0a7409a700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6c106c20 con 0x7f0a6c0f6510 2024-02-11T17:08:45.831 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:45.828+0000 7f0a71e36700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0a6c0f6510 0x7f0a6c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:08:45.831 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:45.828+0000 7f0a71e36700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0a6c0f6510 0x7f0a6c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:60558/0 (socket says 172.21.15.178:60558) 2024-02-11T17:08:45.831 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:45.828+0000 7f0a71e36700 1 -- 172.21.15.178:0/3185464930 learned_addr learned my addr 172.21.15.178:0/3185464930 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:08:45.831 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:45.828+0000 7f0a71e36700 1 -- 172.21.15.178:0/3185464930 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a5c005040 con 0x7f0a6c0f6510 2024-02-11T17:08:45.832 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:45.829+0000 7f0a71e36700 1 --2- 172.21.15.178:0/3185464930 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0a6c0f6510 0x7f0a6c106730 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f0a5c013510 tx=0x7f0a5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:08:45.832 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:45.829+0000 7f0a627fc700 1 -- 172.21.15.178:0/3185464930 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a5c009320 con 0x7f0a6c0f6510 2024-02-11T17:08:45.832 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:45.829+0000 7f0a627fc700 1 -- 172.21.15.178:0/3185464930 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0a5c013f00 con 0x7f0a6c0f6510 2024-02-11T17:08:45.832 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:45.829+0000 7f0a7409a700 1 -- 172.21.15.178:0/3185464930 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a6c106e20 con 0x7f0a6c0f6510 2024-02-11T17:08:45.832 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:45.829+0000 7f0a627fc700 1 -- 172.21.15.178:0/3185464930 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a5c014d90 con 0x7f0a6c0f6510 2024-02-11T17:08:45.832 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:45.829+0000 7f0a7409a700 1 -- 172.21.15.178:0/3185464930 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a6c1072b0 con 0x7f0a6c0f6510 2024-02-11T17:08:45.832 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:45.830+0000 7f0a7409a700 1 -- 172.21.15.178:0/3185464930 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a6c045ad0 con 0x7f0a6c0f6510 2024-02-11T17:08:45.832 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:45.830+0000 7f0a627fc700 1 -- 172.21.15.178:0/3185464930 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0a5c0095f0 con 0x7f0a6c0f6510 2024-02-11T17:08:45.833 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:45.830+0000 7f0a627fc700 1 --2- 172.21.15.178:0/3185464930 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f0a5802ff60 0x7f0a58032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:08:45.833 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:45.830+0000 7f0a627fc700 1 -- 172.21.15.178:0/3185464930 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0a5c0410c0 con 0x7f0a6c0f6510 2024-02-11T17:08:45.833 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:45.830+0000 7f0a71635700 1 --2- 172.21.15.178:0/3185464930 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f0a5802ff60 0x7f0a58032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:08:45.833 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:45.830+0000 7f0a71635700 1 --2- 172.21.15.178:0/3185464930 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f0a5802ff60 0x7f0a58032420 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f0a680096a0 tx=0x7f0a68006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:08:45.836 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:45.834+0000 7f0a627fc700 1 -- 172.21.15.178:0/3185464930 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0a5c014420 con 0x7f0a6c0f6510 2024-02-11T17:08:46.043 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:46.042+0000 7f0a7409a700 1 -- 172.21.15.178:0/3185464930 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0a6c0fbf60 con 0x7f0a6c0f6510 2024-02-11T17:08:46.044 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:46.042+0000 7f0a627fc700 1 -- 172.21.15.178:0/3185464930 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0a5c010230 con 0x7f0a6c0f6510 2024-02-11T17:08:46.045 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:08:46.045 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:08:46.047 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:46.044+0000 7f0a7409a700 1 -- 172.21.15.178:0/3185464930 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f0a5802ff60 msgr2=0x7f0a58032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:08:46.047 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:46.044+0000 7f0a7409a700 1 --2- 172.21.15.178:0/3185464930 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f0a5802ff60 0x7f0a58032420 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f0a680096a0 tx=0x7f0a68006b40).stop 2024-02-11T17:08:46.047 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:46.044+0000 7f0a7409a700 1 -- 172.21.15.178:0/3185464930 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0a6c0f6510 msgr2=0x7f0a6c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:08:46.047 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:46.044+0000 7f0a7409a700 1 --2- 172.21.15.178:0/3185464930 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0a6c0f6510 0x7f0a6c106730 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f0a5c013510 tx=0x7f0a5c00b040).stop 2024-02-11T17:08:46.047 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:46.044+0000 7f0a7409a700 1 -- 172.21.15.178:0/3185464930 shutdown_connections 2024-02-11T17:08:46.047 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:46.044+0000 7f0a7409a700 1 --2- 172.21.15.178:0/3185464930 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f0a5802ff60 0x7f0a58032420 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:08:46.048 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:46.044+0000 7f0a7409a700 1 --2- 172.21.15.178:0/3185464930 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0a6c0f6510 0x7f0a6c106730 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:08:46.048 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:46.044+0000 7f0a7409a700 1 -- 172.21.15.178:0/3185464930 >> 172.21.15.178:0/3185464930 conn(0x7f0a6c0f1c70 msgr2=0x7f0a6c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:08:46.048 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:46.044+0000 7f0a7409a700 1 -- 172.21.15.178:0/3185464930 shutdown_connections 2024-02-11T17:08:46.048 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:46.044+0000 7f0a7409a700 1 -- 172.21.15.178:0/3185464930 wait complete. 2024-02-11T17:08:46.048 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:08:46.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:08:46 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:08:45.211237+0000 mgr.smithi037.eofnaq (mgr.14162) 269 : cluster [DBG] pgmap v255: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:08:46.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:08:46 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:08:46.043722+0000 mon.smithi037 (mon.0) 270 : audit [DBG] from='client.? 172.21.15.178:0/3185464930' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:08:47.842 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:08:47.842 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:08:48.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:08:48 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:08:47.211618+0000 mgr.smithi037.eofnaq (mgr.14162) 270 : cluster [DBG] pgmap v256: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:08:49.687 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:49.682+0000 7f938a3a3700 1 -- 172.21.15.178:0/3897125823 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9384060280 msgr2=0x7f93840f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:08:49.687 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:49.682+0000 7f938a3a3700 1 --2- 172.21.15.178:0/3897125823 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9384060280 0x7f93840f90f0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7f9374009fb0 tx=0x7f9374009b30).stop 2024-02-11T17:08:49.687 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:49.682+0000 7f938a3a3700 1 -- 172.21.15.178:0/3897125823 shutdown_connections 2024-02-11T17:08:49.688 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:49.682+0000 7f938a3a3700 1 --2- 172.21.15.178:0/3897125823 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9384060280 0x7f93840f90f0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:08:49.688 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:49.682+0000 7f938a3a3700 1 -- 172.21.15.178:0/3897125823 >> 172.21.15.178:0/3897125823 conn(0x7f93840f0b70 msgr2=0x7f93840f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:08:49.688 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:49.682+0000 7f938a3a3700 1 -- 172.21.15.178:0/3897125823 shutdown_connections 2024-02-11T17:08:49.688 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:49.682+0000 7f938a3a3700 1 -- 172.21.15.178:0/3897125823 wait complete. 2024-02-11T17:08:49.688 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:49.682+0000 7f938a3a3700 1 Processor -- start 2024-02-11T17:08:49.688 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:49.683+0000 7f938a3a3700 1 -- start start 2024-02-11T17:08:49.689 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:49.683+0000 7f938a3a3700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9384060280 0x7f93841066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:08:49.689 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:49.683+0000 7f938a3a3700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9384106bd0 con 0x7f9384060280 2024-02-11T17:08:49.689 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:49.683+0000 7f9383fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9384060280 0x7f93841066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:08:49.689 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:49.683+0000 7f9383fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9384060280 0x7f93841066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:35532/0 (socket says 172.21.15.178:35532) 2024-02-11T17:08:49.689 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:49.683+0000 7f9383fff700 1 -- 172.21.15.178:0/118616617 learned_addr learned my addr 172.21.15.178:0/118616617 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:08:49.689 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:49.683+0000 7f9383fff700 1 -- 172.21.15.178:0/118616617 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9374005040 con 0x7f9384060280 2024-02-11T17:08:49.689 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:49.683+0000 7f9383fff700 1 --2- 172.21.15.178:0/118616617 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9384060280 0x7f93841066e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f9374013510 tx=0x7f937400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:08:49.689 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:49.684+0000 7f9380ff9700 1 -- 172.21.15.178:0/118616617 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9374009320 con 0x7f9384060280 2024-02-11T17:08:49.689 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:49.684+0000 7f938a3a3700 1 -- 172.21.15.178:0/118616617 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9384106dd0 con 0x7f9384060280 2024-02-11T17:08:49.689 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:49.684+0000 7f9380ff9700 1 -- 172.21.15.178:0/118616617 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9374013f00 con 0x7f9384060280 2024-02-11T17:08:49.689 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:49.684+0000 7f9380ff9700 1 -- 172.21.15.178:0/118616617 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9374014dc0 con 0x7f9384060280 2024-02-11T17:08:49.689 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:49.684+0000 7f938a3a3700 1 -- 172.21.15.178:0/118616617 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9384107260 con 0x7f9384060280 2024-02-11T17:08:49.689 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:49.685+0000 7f9380ff9700 1 -- 172.21.15.178:0/118616617 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f93740095f0 con 0x7f9384060280 2024-02-11T17:08:49.689 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:49.685+0000 7f938a3a3700 1 -- 172.21.15.178:0/118616617 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9384045ad0 con 0x7f9384060280 2024-02-11T17:08:49.689 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:49.685+0000 7f9380ff9700 1 --2- 172.21.15.178:0/118616617 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f937002ff60 0x7f9370032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:08:49.690 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:49.685+0000 7f9380ff9700 1 -- 172.21.15.178:0/118616617 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9374041000 con 0x7f9384060280 2024-02-11T17:08:49.690 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:49.685+0000 7f93837fe700 1 --2- 172.21.15.178:0/118616617 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f937002ff60 0x7f9370032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:08:49.690 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:49.685+0000 7f93837fe700 1 --2- 172.21.15.178:0/118616617 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f937002ff60 0x7f9370032420 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f936c0096a0 tx=0x7f936c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:08:49.691 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:49.689+0000 7f9380ff9700 1 -- 172.21.15.178:0/118616617 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9374014420 con 0x7f9384060280 2024-02-11T17:08:49.897 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:49.895+0000 7f938a3a3700 1 -- 172.21.15.178:0/118616617 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9384108110 con 0x7f9384060280 2024-02-11T17:08:49.897 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:49.896+0000 7f9380ff9700 1 -- 172.21.15.178:0/118616617 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9374010230 con 0x7f9384060280 2024-02-11T17:08:49.897 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:08:49.897 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:08:49.899 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:49.897+0000 7f938a3a3700 1 -- 172.21.15.178:0/118616617 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f937002ff60 msgr2=0x7f9370032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:08:49.899 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:49.897+0000 7f938a3a3700 1 --2- 172.21.15.178:0/118616617 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f937002ff60 0x7f9370032420 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f936c0096a0 tx=0x7f936c006b40).stop 2024-02-11T17:08:49.899 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:49.897+0000 7f938a3a3700 1 -- 172.21.15.178:0/118616617 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9384060280 msgr2=0x7f93841066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:08:49.899 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:49.897+0000 7f938a3a3700 1 --2- 172.21.15.178:0/118616617 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9384060280 0x7f93841066e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f9374013510 tx=0x7f937400b040).stop 2024-02-11T17:08:49.899 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:49.897+0000 7f938a3a3700 1 -- 172.21.15.178:0/118616617 shutdown_connections 2024-02-11T17:08:49.899 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:49.897+0000 7f938a3a3700 1 --2- 172.21.15.178:0/118616617 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f937002ff60 0x7f9370032420 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:08:49.899 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:49.897+0000 7f938a3a3700 1 --2- 172.21.15.178:0/118616617 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9384060280 0x7f93841066e0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:08:49.899 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:49.897+0000 7f938a3a3700 1 -- 172.21.15.178:0/118616617 >> 172.21.15.178:0/118616617 conn(0x7f93840f0b70 msgr2=0x7f93840f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:08:49.900 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:49.897+0000 7f938a3a3700 1 -- 172.21.15.178:0/118616617 shutdown_connections 2024-02-11T17:08:49.900 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:49.897+0000 7f938a3a3700 1 -- 172.21.15.178:0/118616617 wait complete. 2024-02-11T17:08:49.900 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:08:50.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:08:50 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:08:49.211987+0000 mgr.smithi037.eofnaq (mgr.14162) 271 : cluster [DBG] pgmap v257: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:08:50.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:08:50 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:08:49.897062+0000 mon.smithi037 (mon.0) 271 : audit [DBG] from='client.? 172.21.15.178:0/118616617' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:08:51.675 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:08:51.675 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:08:52.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:08:52 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:08:51.212329+0000 mgr.smithi037.eofnaq (mgr.14162) 272 : cluster [DBG] pgmap v258: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:08:53.591 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:53.589+0000 7fb3cebaa700 1 -- 172.21.15.178:0/2269942540 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3c80f62a0 msgr2=0x7fb3c80f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:08:53.591 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:53.589+0000 7fb3cebaa700 1 --2- 172.21.15.178:0/2269942540 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3c80f62a0 0x7fb3c80f66c0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7fb3c4009fb0 tx=0x7fb3c4009b30).stop 2024-02-11T17:08:53.591 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:53.589+0000 7fb3cebaa700 1 -- 172.21.15.178:0/2269942540 shutdown_connections 2024-02-11T17:08:53.591 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:53.589+0000 7fb3cebaa700 1 --2- 172.21.15.178:0/2269942540 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3c80f62a0 0x7fb3c80f66c0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:08:53.591 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:53.589+0000 7fb3cebaa700 1 -- 172.21.15.178:0/2269942540 >> 172.21.15.178:0/2269942540 conn(0x7fb3c80f1ac0 msgr2=0x7fb3c80f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:08:53.591 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:53.589+0000 7fb3cebaa700 1 -- 172.21.15.178:0/2269942540 shutdown_connections 2024-02-11T17:08:53.592 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:53.589+0000 7fb3cebaa700 1 -- 172.21.15.178:0/2269942540 wait complete. 2024-02-11T17:08:53.594 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:53.590+0000 7fb3cebaa700 1 Processor -- start 2024-02-11T17:08:53.594 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:53.590+0000 7fb3cebaa700 1 -- start start 2024-02-11T17:08:53.594 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:53.590+0000 7fb3cebaa700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3c80f62a0 0x7fb3c8106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:08:53.594 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:53.590+0000 7fb3cebaa700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c81069f0 con 0x7fb3c80f62a0 2024-02-11T17:08:53.594 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:53.590+0000 7fb3cdba8700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3c80f62a0 0x7fb3c8106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:08:53.594 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:53.590+0000 7fb3cdba8700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3c80f62a0 0x7fb3c8106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:35564/0 (socket says 172.21.15.178:35564) 2024-02-11T17:08:53.595 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:53.590+0000 7fb3cdba8700 1 -- 172.21.15.178:0/2076256483 learned_addr learned my addr 172.21.15.178:0/2076256483 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:08:53.595 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:53.590+0000 7fb3cdba8700 1 -- 172.21.15.178:0/2076256483 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3c4005040 con 0x7fb3c80f62a0 2024-02-11T17:08:53.595 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:53.591+0000 7fb3cdba8700 1 --2- 172.21.15.178:0/2076256483 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3c80f62a0 0x7fb3c8106500 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7fb3c4013510 tx=0x7fb3c400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:08:53.595 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:53.591+0000 7fb3be7fc700 1 -- 172.21.15.178:0/2076256483 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3c4009320 con 0x7fb3c80f62a0 2024-02-11T17:08:53.595 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:53.591+0000 7fb3cebaa700 1 -- 172.21.15.178:0/2076256483 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3c8106bf0 con 0x7fb3c80f62a0 2024-02-11T17:08:53.595 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:53.591+0000 7fb3be7fc700 1 -- 172.21.15.178:0/2076256483 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb3c4013f00 con 0x7fb3c80f62a0 2024-02-11T17:08:53.595 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:53.591+0000 7fb3be7fc700 1 -- 172.21.15.178:0/2076256483 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3c4014dc0 con 0x7fb3c80f62a0 2024-02-11T17:08:53.595 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:53.591+0000 7fb3cebaa700 1 -- 172.21.15.178:0/2076256483 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3c8107080 con 0x7fb3c80f62a0 2024-02-11T17:08:53.595 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:53.592+0000 7fb3be7fc700 1 -- 172.21.15.178:0/2076256483 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb3c40095f0 con 0x7fb3c80f62a0 2024-02-11T17:08:53.595 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:53.592+0000 7fb3cebaa700 1 -- 172.21.15.178:0/2076256483 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3c8045ad0 con 0x7fb3c80f62a0 2024-02-11T17:08:53.595 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:53.592+0000 7fb3be7fc700 1 --2- 172.21.15.178:0/2076256483 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb3b402ff60 0x7fb3b4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:08:53.595 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:53.592+0000 7fb3be7fc700 1 -- 172.21.15.178:0/2076256483 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb3c4041000 con 0x7fb3c80f62a0 2024-02-11T17:08:53.595 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:53.592+0000 7fb3cd3a7700 1 --2- 172.21.15.178:0/2076256483 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb3b402ff60 0x7fb3b4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:08:53.595 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:53.593+0000 7fb3cd3a7700 1 --2- 172.21.15.178:0/2076256483 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb3b402ff60 0x7fb3b4032420 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fb3b80096a0 tx=0x7fb3b8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:08:53.598 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:53.597+0000 7fb3be7fc700 1 -- 172.21.15.178:0/2076256483 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb3c401c790 con 0x7fb3c80f62a0 2024-02-11T17:08:53.802 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:53.800+0000 7fb3cebaa700 1 -- 172.21.15.178:0/2076256483 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb3c8107f30 con 0x7fb3c80f62a0 2024-02-11T17:08:53.802 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:53.801+0000 7fb3be7fc700 1 -- 172.21.15.178:0/2076256483 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb3c4010230 con 0x7fb3c80f62a0 2024-02-11T17:08:53.802 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:08:53.802 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:08:53.804 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:53.802+0000 7fb3cebaa700 1 -- 172.21.15.178:0/2076256483 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb3b402ff60 msgr2=0x7fb3b4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:08:53.804 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:53.802+0000 7fb3cebaa700 1 --2- 172.21.15.178:0/2076256483 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb3b402ff60 0x7fb3b4032420 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fb3b80096a0 tx=0x7fb3b8006b40).stop 2024-02-11T17:08:53.804 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:53.802+0000 7fb3cebaa700 1 -- 172.21.15.178:0/2076256483 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3c80f62a0 msgr2=0x7fb3c8106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:08:53.804 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:53.802+0000 7fb3cebaa700 1 --2- 172.21.15.178:0/2076256483 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3c80f62a0 0x7fb3c8106500 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7fb3c4013510 tx=0x7fb3c400b040).stop 2024-02-11T17:08:53.804 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:53.802+0000 7fb3cebaa700 1 -- 172.21.15.178:0/2076256483 shutdown_connections 2024-02-11T17:08:53.805 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:53.802+0000 7fb3cebaa700 1 --2- 172.21.15.178:0/2076256483 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb3b402ff60 0x7fb3b4032420 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:08:53.805 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:53.802+0000 7fb3cebaa700 1 --2- 172.21.15.178:0/2076256483 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3c80f62a0 0x7fb3c8106500 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:08:53.805 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:53.803+0000 7fb3cebaa700 1 -- 172.21.15.178:0/2076256483 >> 172.21.15.178:0/2076256483 conn(0x7fb3c80f1ac0 msgr2=0x7fb3c80f2530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:08:53.805 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:53.803+0000 7fb3cebaa700 1 -- 172.21.15.178:0/2076256483 shutdown_connections 2024-02-11T17:08:53.805 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:53.803+0000 7fb3cebaa700 1 -- 172.21.15.178:0/2076256483 wait complete. 2024-02-11T17:08:53.805 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:08:54.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:08:54 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:08:53.212710+0000 mgr.smithi037.eofnaq (mgr.14162) 273 : cluster [DBG] pgmap v259: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:08:54.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:08:54 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:08:53.801998+0000 mon.smithi037 (mon.0) 272 : audit [DBG] from='client.? 172.21.15.178:0/2076256483' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:08:55.564 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:08:55.564 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:08:56.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:08:56 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:08:55.213089+0000 mgr.smithi037.eofnaq (mgr.14162) 274 : cluster [DBG] pgmap v260: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:08:57.754 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:57.723+0000 7fe0e67b7700 1 -- 172.21.15.178:0/3949869813 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe0e00f62a0 msgr2=0x7fe0e00f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:08:57.754 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:57.723+0000 7fe0e67b7700 1 --2- 172.21.15.178:0/3949869813 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe0e00f62a0 0x7fe0e00f66c0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7fe0dc009fb0 tx=0x7fe0dc009b30).stop 2024-02-11T17:08:57.754 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:57.723+0000 7fe0e67b7700 1 -- 172.21.15.178:0/3949869813 shutdown_connections 2024-02-11T17:08:57.754 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:57.723+0000 7fe0e67b7700 1 --2- 172.21.15.178:0/3949869813 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe0e00f62a0 0x7fe0e00f66c0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:08:57.754 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:57.723+0000 7fe0e67b7700 1 -- 172.21.15.178:0/3949869813 >> 172.21.15.178:0/3949869813 conn(0x7fe0e00f1aa0 msgr2=0x7fe0e00f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:08:57.754 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:57.723+0000 7fe0e67b7700 1 -- 172.21.15.178:0/3949869813 shutdown_connections 2024-02-11T17:08:57.754 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:57.723+0000 7fe0e67b7700 1 -- 172.21.15.178:0/3949869813 wait complete. 2024-02-11T17:08:57.754 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:57.723+0000 7fe0e67b7700 1 Processor -- start 2024-02-11T17:08:57.754 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:57.723+0000 7fe0e67b7700 1 -- start start 2024-02-11T17:08:57.754 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:57.723+0000 7fe0e67b7700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe0e00f62a0 0x7fe0e01064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:08:57.754 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:57.723+0000 7fe0e67b7700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e01069a0 con 0x7fe0e00f62a0 2024-02-11T17:08:57.755 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:57.724+0000 7fe0e57b5700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe0e00f62a0 0x7fe0e01064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:08:57.755 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:57.724+0000 7fe0e57b5700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe0e00f62a0 0x7fe0e01064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:35572/0 (socket says 172.21.15.178:35572) 2024-02-11T17:08:57.755 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:57.724+0000 7fe0e57b5700 1 -- 172.21.15.178:0/3560268804 learned_addr learned my addr 172.21.15.178:0/3560268804 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:08:57.755 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:57.724+0000 7fe0e57b5700 1 -- 172.21.15.178:0/3560268804 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0dc005040 con 0x7fe0e00f62a0 2024-02-11T17:08:57.755 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:57.724+0000 7fe0e57b5700 1 --2- 172.21.15.178:0/3560268804 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe0e00f62a0 0x7fe0e01064b0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7fe0dc013510 tx=0x7fe0dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:08:57.755 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:57.724+0000 7fe0d5ffb700 1 -- 172.21.15.178:0/3560268804 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe0dc009320 con 0x7fe0e00f62a0 2024-02-11T17:08:57.755 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:57.724+0000 7fe0e67b7700 1 -- 172.21.15.178:0/3560268804 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0e0106ba0 con 0x7fe0e00f62a0 2024-02-11T17:08:57.755 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:57.724+0000 7fe0d5ffb700 1 -- 172.21.15.178:0/3560268804 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe0dc013f00 con 0x7fe0e00f62a0 2024-02-11T17:08:57.755 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:57.724+0000 7fe0d5ffb700 1 -- 172.21.15.178:0/3560268804 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe0dc014dc0 con 0x7fe0e00f62a0 2024-02-11T17:08:57.755 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:57.724+0000 7fe0e67b7700 1 -- 172.21.15.178:0/3560268804 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0e0107030 con 0x7fe0e00f62a0 2024-02-11T17:08:57.755 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:57.725+0000 7fe0e67b7700 1 -- 172.21.15.178:0/3560268804 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0e0045ad0 con 0x7fe0e00f62a0 2024-02-11T17:08:57.755 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:57.725+0000 7fe0d5ffb700 1 -- 172.21.15.178:0/3560268804 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe0dc0095f0 con 0x7fe0e00f62a0 2024-02-11T17:08:57.755 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:57.725+0000 7fe0d5ffb700 1 --2- 172.21.15.178:0/3560268804 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fe0cc02ff60 0x7fe0cc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:08:57.755 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:57.726+0000 7fe0d5ffb700 1 -- 172.21.15.178:0/3560268804 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe0dc041020 con 0x7fe0e00f62a0 2024-02-11T17:08:57.756 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:57.726+0000 7fe0e4fb4700 1 --2- 172.21.15.178:0/3560268804 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fe0cc02ff60 0x7fe0cc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:08:57.756 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:57.726+0000 7fe0e4fb4700 1 --2- 172.21.15.178:0/3560268804 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fe0cc02ff60 0x7fe0cc032420 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fe0d00096a0 tx=0x7fe0d0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:08:57.756 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:57.730+0000 7fe0d5ffb700 1 -- 172.21.15.178:0/3560268804 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe0dc014420 con 0x7fe0e00f62a0 2024-02-11T17:08:57.939 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:57.938+0000 7fe0e67b7700 1 -- 172.21.15.178:0/3560268804 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe0e0107e80 con 0x7fe0e00f62a0 2024-02-11T17:08:57.940 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:57.938+0000 7fe0d5ffb700 1 -- 172.21.15.178:0/3560268804 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe0dc01c920 con 0x7fe0e00f62a0 2024-02-11T17:08:57.940 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:08:57.941 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:08:57.942 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:57.940+0000 7fe0e67b7700 1 -- 172.21.15.178:0/3560268804 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fe0cc02ff60 msgr2=0x7fe0cc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:08:57.942 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:57.940+0000 7fe0e67b7700 1 --2- 172.21.15.178:0/3560268804 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fe0cc02ff60 0x7fe0cc032420 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fe0d00096a0 tx=0x7fe0d0006b40).stop 2024-02-11T17:08:57.942 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:57.940+0000 7fe0e67b7700 1 -- 172.21.15.178:0/3560268804 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe0e00f62a0 msgr2=0x7fe0e01064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:08:57.942 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:57.940+0000 7fe0e67b7700 1 --2- 172.21.15.178:0/3560268804 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe0e00f62a0 0x7fe0e01064b0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7fe0dc013510 tx=0x7fe0dc00b040).stop 2024-02-11T17:08:57.942 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:57.940+0000 7fe0e67b7700 1 -- 172.21.15.178:0/3560268804 shutdown_connections 2024-02-11T17:08:57.942 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:57.940+0000 7fe0e67b7700 1 --2- 172.21.15.178:0/3560268804 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fe0cc02ff60 0x7fe0cc032420 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:08:57.943 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:57.940+0000 7fe0e67b7700 1 --2- 172.21.15.178:0/3560268804 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe0e00f62a0 0x7fe0e01064b0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:08:57.943 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:57.940+0000 7fe0e67b7700 1 -- 172.21.15.178:0/3560268804 >> 172.21.15.178:0/3560268804 conn(0x7fe0e00f1aa0 msgr2=0x7fe0e00f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:08:57.943 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:57.940+0000 7fe0e67b7700 1 -- 172.21.15.178:0/3560268804 shutdown_connections 2024-02-11T17:08:57.943 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:08:57.940+0000 7fe0e67b7700 1 -- 172.21.15.178:0/3560268804 wait complete. 2024-02-11T17:08:57.943 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:08:58.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:08:58 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:08:57.213441+0000 mgr.smithi037.eofnaq (mgr.14162) 275 : cluster [DBG] pgmap v261: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:08:58.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:08:58 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:08:57.939714+0000 mon.smithi037 (mon.0) 273 : audit [DBG] from='client.? 172.21.15.178:0/3560268804' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:08:59.604 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:08:59.604 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:09:00.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:09:00 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:08:59.213730+0000 mgr.smithi037.eofnaq (mgr.14162) 276 : cluster [DBG] pgmap v262: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:09:01.449 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:01.445+0000 7f536b2a0700 1 -- 172.21.15.178:0/1119444750 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f53640f5410 msgr2=0x7f53640f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:09:01.449 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:01.445+0000 7f536b2a0700 1 --2- 172.21.15.178:0/1119444750 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f53640f5410 0x7f53640f5830 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7f5358009fb0 tx=0x7f5358009b30).stop 2024-02-11T17:09:01.449 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:01.445+0000 7f536b2a0700 1 -- 172.21.15.178:0/1119444750 shutdown_connections 2024-02-11T17:09:01.449 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:01.445+0000 7f536b2a0700 1 --2- 172.21.15.178:0/1119444750 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f53640f5410 0x7f53640f5830 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:09:01.449 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:01.445+0000 7f536b2a0700 1 -- 172.21.15.178:0/1119444750 >> 172.21.15.178:0/1119444750 conn(0x7f53640f0b70 msgr2=0x7f53640f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:09:01.449 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:01.445+0000 7f536b2a0700 1 -- 172.21.15.178:0/1119444750 shutdown_connections 2024-02-11T17:09:01.450 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:01.445+0000 7f536b2a0700 1 -- 172.21.15.178:0/1119444750 wait complete. 2024-02-11T17:09:01.450 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:01.446+0000 7f536b2a0700 1 Processor -- start 2024-02-11T17:09:01.450 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:01.446+0000 7f536b2a0700 1 -- start start 2024-02-11T17:09:01.450 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:01.446+0000 7f536b2a0700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f53640f5410 0x7f5364105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:09:01.450 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:01.446+0000 7f536b2a0700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5364105b20 con 0x7f53640f5410 2024-02-11T17:09:01.450 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:01.446+0000 7f536903c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f53640f5410 0x7f5364105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:09:01.450 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:01.447+0000 7f536903c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f53640f5410 0x7f5364105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:40588/0 (socket says 172.21.15.178:40588) 2024-02-11T17:09:01.450 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:01.447+0000 7f536903c700 1 -- 172.21.15.178:0/3204212636 learned_addr learned my addr 172.21.15.178:0/3204212636 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:09:01.450 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:01.447+0000 7f536903c700 1 -- 172.21.15.178:0/3204212636 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5358005040 con 0x7f53640f5410 2024-02-11T17:09:01.450 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:01.447+0000 7f536903c700 1 --2- 172.21.15.178:0/3204212636 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f53640f5410 0x7f5364105630 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f5358013510 tx=0x7f535800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:09:01.450 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:01.447+0000 7f5355ffb700 1 -- 172.21.15.178:0/3204212636 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5358009320 con 0x7f53640f5410 2024-02-11T17:09:01.450 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:01.447+0000 7f536b2a0700 1 -- 172.21.15.178:0/3204212636 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5364105d20 con 0x7f53640f5410 2024-02-11T17:09:01.450 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:01.447+0000 7f5355ffb700 1 -- 172.21.15.178:0/3204212636 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5358013f00 con 0x7f53640f5410 2024-02-11T17:09:01.450 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:01.447+0000 7f5355ffb700 1 -- 172.21.15.178:0/3204212636 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5358014d90 con 0x7f53640f5410 2024-02-11T17:09:01.450 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:01.447+0000 7f536b2a0700 1 -- 172.21.15.178:0/3204212636 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53641061b0 con 0x7f53640f5410 2024-02-11T17:09:01.451 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:01.448+0000 7f5355ffb700 1 -- 172.21.15.178:0/3204212636 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5358014420 con 0x7f53640f5410 2024-02-11T17:09:01.451 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:01.448+0000 7f536b2a0700 1 -- 172.21.15.178:0/3204212636 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5364045ad0 con 0x7f53640f5410 2024-02-11T17:09:01.451 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:01.448+0000 7f5355ffb700 1 --2- 172.21.15.178:0/3204212636 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f535002ff60 0x7f5350032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:09:01.451 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:01.448+0000 7f5355ffb700 1 -- 172.21.15.178:0/3204212636 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f53580410e0 con 0x7f53640f5410 2024-02-11T17:09:01.451 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:01.449+0000 7f536883b700 1 --2- 172.21.15.178:0/3204212636 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f535002ff60 0x7f5350032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:09:01.452 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:01.449+0000 7f536883b700 1 --2- 172.21.15.178:0/3204212636 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f535002ff60 0x7f5350032420 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f53600096a0 tx=0x7f5360006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:09:01.455 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:01.453+0000 7f5355ffb700 1 -- 172.21.15.178:0/3204212636 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f53580095f0 con 0x7f53640f5410 2024-02-11T17:09:01.662 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:01.660+0000 7f536b2a0700 1 -- 172.21.15.178:0/3204212636 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f53640fae60 con 0x7f53640f5410 2024-02-11T17:09:01.662 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:01.661+0000 7f5355ffb700 1 -- 172.21.15.178:0/3204212636 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f535801db70 con 0x7f53640f5410 2024-02-11T17:09:01.662 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:09:01.663 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:09:01.664 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:01.662+0000 7f536b2a0700 1 -- 172.21.15.178:0/3204212636 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f535002ff60 msgr2=0x7f5350032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:09:01.664 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:01.662+0000 7f536b2a0700 1 --2- 172.21.15.178:0/3204212636 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f535002ff60 0x7f5350032420 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f53600096a0 tx=0x7f5360006b40).stop 2024-02-11T17:09:01.664 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:01.662+0000 7f536b2a0700 1 -- 172.21.15.178:0/3204212636 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f53640f5410 msgr2=0x7f5364105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:09:01.664 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:01.662+0000 7f536b2a0700 1 --2- 172.21.15.178:0/3204212636 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f53640f5410 0x7f5364105630 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f5358013510 tx=0x7f535800b040).stop 2024-02-11T17:09:01.664 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:01.662+0000 7f536b2a0700 1 -- 172.21.15.178:0/3204212636 shutdown_connections 2024-02-11T17:09:01.664 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:01.662+0000 7f536b2a0700 1 --2- 172.21.15.178:0/3204212636 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f535002ff60 0x7f5350032420 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:09:01.664 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:01.662+0000 7f536b2a0700 1 --2- 172.21.15.178:0/3204212636 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f53640f5410 0x7f5364105630 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:09:01.664 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:01.662+0000 7f536b2a0700 1 -- 172.21.15.178:0/3204212636 >> 172.21.15.178:0/3204212636 conn(0x7f53640f0b70 msgr2=0x7f53640f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:09:01.664 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:01.662+0000 7f536b2a0700 1 -- 172.21.15.178:0/3204212636 shutdown_connections 2024-02-11T17:09:01.665 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:01.662+0000 7f536b2a0700 1 -- 172.21.15.178:0/3204212636 wait complete. 2024-02-11T17:09:01.665 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:09:02.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:09:02 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:09:01.214071+0000 mgr.smithi037.eofnaq (mgr.14162) 277 : cluster [DBG] pgmap v263: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:09:02.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:09:02 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:09:01.662037+0000 mon.smithi037 (mon.0) 274 : audit [DBG] from='client.? 172.21.15.178:0/3204212636' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:09:03.343 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:09:03.343 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:09:04.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:09:04 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:09:03.214441+0000 mgr.smithi037.eofnaq (mgr.14162) 278 : cluster [DBG] pgmap v264: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:09:04.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:09:04 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:09:03.279364+0000 mon.smithi037 (mon.0) 275 : audit [INF] from='mgr.14162 172.21.15.37:0/2745465012' entity='mgr.smithi037.eofnaq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi037.eofnaq/mirror_snapshot_schedule"}]: dispatch 2024-02-11T17:09:04.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:09:04 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:09:03.281904+0000 mon.smithi037 (mon.0) 276 : audit [INF] from='mgr.14162 172.21.15.37:0/2745465012' entity='mgr.smithi037.eofnaq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi037.eofnaq/trash_purge_schedule"}]: dispatch 2024-02-11T17:09:05.350 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:05.347+0000 7fab99eb7700 1 -- 172.21.15.178:0/1614097586 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fab940f47c0 msgr2=0x7fab940f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:09:05.350 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:05.347+0000 7fab99eb7700 1 --2- 172.21.15.178:0/1614097586 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fab940f47c0 0x7fab940f4be0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7fab88009fb0 tx=0x7fab88009b30).stop 2024-02-11T17:09:05.351 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:05.347+0000 7fab99eb7700 1 -- 172.21.15.178:0/1614097586 shutdown_connections 2024-02-11T17:09:05.351 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:05.347+0000 7fab99eb7700 1 --2- 172.21.15.178:0/1614097586 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fab940f47c0 0x7fab940f4be0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:09:05.351 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:05.347+0000 7fab99eb7700 1 -- 172.21.15.178:0/1614097586 >> 172.21.15.178:0/1614097586 conn(0x7fab940f02e0 msgr2=0x7fab940f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:09:05.351 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:05.347+0000 7fab99eb7700 1 -- 172.21.15.178:0/1614097586 shutdown_connections 2024-02-11T17:09:05.351 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:05.347+0000 7fab99eb7700 1 -- 172.21.15.178:0/1614097586 wait complete. 2024-02-11T17:09:05.351 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:05.347+0000 7fab99eb7700 1 Processor -- start 2024-02-11T17:09:05.351 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:05.347+0000 7fab99eb7700 1 -- start start 2024-02-11T17:09:05.351 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:05.348+0000 7fab99eb7700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fab940f47c0 0x7fab94106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:09:05.351 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:05.348+0000 7fab99eb7700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab94106bf0 con 0x7fab940f47c0 2024-02-11T17:09:05.351 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:05.348+0000 7fab937fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fab940f47c0 0x7fab94106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:09:05.351 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:05.348+0000 7fab937fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fab940f47c0 0x7fab94106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:40596/0 (socket says 172.21.15.178:40596) 2024-02-11T17:09:05.351 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:05.348+0000 7fab937fe700 1 -- 172.21.15.178:0/1505141580 learned_addr learned my addr 172.21.15.178:0/1505141580 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:09:05.352 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:05.348+0000 7fab937fe700 1 -- 172.21.15.178:0/1505141580 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab88005040 con 0x7fab940f47c0 2024-02-11T17:09:05.352 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:05.348+0000 7fab937fe700 1 --2- 172.21.15.178:0/1505141580 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fab940f47c0 0x7fab94106700 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7fab88014000 tx=0x7fab8800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:09:05.352 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:05.348+0000 7fab7bfff700 1 -- 172.21.15.178:0/1505141580 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab880095f0 con 0x7fab940f47c0 2024-02-11T17:09:05.352 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:05.348+0000 7fab99eb7700 1 -- 172.21.15.178:0/1505141580 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab94106df0 con 0x7fab940f47c0 2024-02-11T17:09:05.352 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:05.348+0000 7fab7bfff700 1 -- 172.21.15.178:0/1505141580 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fab88007e70 con 0x7fab940f47c0 2024-02-11T17:09:05.352 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:05.348+0000 7fab7bfff700 1 -- 172.21.15.178:0/1505141580 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab88014ea0 con 0x7fab940f47c0 2024-02-11T17:09:05.352 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:05.348+0000 7fab99eb7700 1 -- 172.21.15.178:0/1505141580 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab94107280 con 0x7fab940f47c0 2024-02-11T17:09:05.353 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:05.349+0000 7fab7bfff700 1 -- 172.21.15.178:0/1505141580 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fab88025070 con 0x7fab940f47c0 2024-02-11T17:09:05.353 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:05.349+0000 7fab7bfff700 1 --2- 172.21.15.178:0/1505141580 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fab7c02ffb0 0x7fab7c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:09:05.353 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:05.349+0000 7fab7bfff700 1 -- 172.21.15.178:0/1505141580 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fab88040a40 con 0x7fab940f47c0 2024-02-11T17:09:05.353 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:05.349+0000 7fab99eb7700 1 -- 172.21.15.178:0/1505141580 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab94045ad0 con 0x7fab940f47c0 2024-02-11T17:09:05.353 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:05.350+0000 7fab92ffd700 1 --2- 172.21.15.178:0/1505141580 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fab7c02ffb0 0x7fab7c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:09:05.353 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:05.350+0000 7fab92ffd700 1 --2- 172.21.15.178:0/1505141580 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fab7c02ffb0 0x7fab7c032470 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7fab840096a0 tx=0x7fab84006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:09:05.356 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:05.354+0000 7fab7bfff700 1 -- 172.21.15.178:0/1505141580 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fab88026350 con 0x7fab940f47c0 2024-02-11T17:09:05.562 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:05.560+0000 7fab99eb7700 1 -- 172.21.15.178:0/1505141580 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fab940fc050 con 0x7fab940f47c0 2024-02-11T17:09:05.563 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:05.561+0000 7fab7bfff700 1 -- 172.21.15.178:0/1505141580 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fab8802d090 con 0x7fab940f47c0 2024-02-11T17:09:05.563 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:09:05.563 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:09:05.564 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:05.563+0000 7fab99eb7700 1 -- 172.21.15.178:0/1505141580 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fab7c02ffb0 msgr2=0x7fab7c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:09:05.565 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:05.563+0000 7fab99eb7700 1 --2- 172.21.15.178:0/1505141580 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fab7c02ffb0 0x7fab7c032470 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7fab840096a0 tx=0x7fab84006b40).stop 2024-02-11T17:09:05.565 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:05.563+0000 7fab99eb7700 1 -- 172.21.15.178:0/1505141580 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fab940f47c0 msgr2=0x7fab94106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:09:05.565 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:05.563+0000 7fab99eb7700 1 --2- 172.21.15.178:0/1505141580 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fab940f47c0 0x7fab94106700 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7fab88014000 tx=0x7fab8800b040).stop 2024-02-11T17:09:05.565 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:05.563+0000 7fab99eb7700 1 -- 172.21.15.178:0/1505141580 shutdown_connections 2024-02-11T17:09:05.565 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:05.563+0000 7fab99eb7700 1 --2- 172.21.15.178:0/1505141580 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fab7c02ffb0 0x7fab7c032470 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:09:05.565 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:05.563+0000 7fab99eb7700 1 --2- 172.21.15.178:0/1505141580 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fab940f47c0 0x7fab94106700 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:09:05.565 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:05.563+0000 7fab99eb7700 1 -- 172.21.15.178:0/1505141580 >> 172.21.15.178:0/1505141580 conn(0x7fab940f02e0 msgr2=0x7fab940f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:09:05.565 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:05.563+0000 7fab99eb7700 1 -- 172.21.15.178:0/1505141580 shutdown_connections 2024-02-11T17:09:05.565 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:05.563+0000 7fab99eb7700 1 -- 172.21.15.178:0/1505141580 wait complete. 2024-02-11T17:09:05.566 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:09:06.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:09:06 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:09:05.214820+0000 mgr.smithi037.eofnaq (mgr.14162) 279 : cluster [DBG] pgmap v265: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:09:06.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:09:06 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:09:05.562441+0000 mon.smithi037 (mon.0) 277 : audit [DBG] from='client.? 172.21.15.178:0/1505141580' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:09:07.261 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:09:07.261 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:09:08.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:09:08 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:09:07.215149+0000 mgr.smithi037.eofnaq (mgr.14162) 280 : cluster [DBG] pgmap v266: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:09:09.147 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:09.145+0000 7f0afd727700 1 -- 172.21.15.178:0/1214840880 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0af80f51a0 msgr2=0x7f0af80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:09:09.147 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:09.145+0000 7f0afd727700 1 --2- 172.21.15.178:0/1214840880 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0af80f51a0 0x7f0af80f55c0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7f0ae8009fb0 tx=0x7f0ae8009b30).stop 2024-02-11T17:09:09.147 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:09.145+0000 7f0afd727700 1 -- 172.21.15.178:0/1214840880 shutdown_connections 2024-02-11T17:09:09.147 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:09.145+0000 7f0afd727700 1 --2- 172.21.15.178:0/1214840880 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0af80f51a0 0x7f0af80f55c0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:09:09.147 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:09.145+0000 7f0afd727700 1 -- 172.21.15.178:0/1214840880 >> 172.21.15.178:0/1214840880 conn(0x7f0af80f09a0 msgr2=0x7f0af80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:09:09.147 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:09.145+0000 7f0afd727700 1 -- 172.21.15.178:0/1214840880 shutdown_connections 2024-02-11T17:09:09.147 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:09.145+0000 7f0afd727700 1 -- 172.21.15.178:0/1214840880 wait complete. 2024-02-11T17:09:09.147 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:09.145+0000 7f0afd727700 1 Processor -- start 2024-02-11T17:09:09.149 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:09.146+0000 7f0afd727700 1 -- start start 2024-02-11T17:09:09.149 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:09.146+0000 7f0afd727700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0af80f51a0 0x7f0af80fd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:09:09.149 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:09.146+0000 7f0afd727700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0af80fb5a0 con 0x7f0af80f51a0 2024-02-11T17:09:09.149 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:09.146+0000 7f0af7fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0af80f51a0 0x7f0af80fd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:09:09.150 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:09.146+0000 7f0af7fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0af80f51a0 0x7f0af80fd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:40610/0 (socket says 172.21.15.178:40610) 2024-02-11T17:09:09.150 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:09.146+0000 7f0af7fff700 1 -- 172.21.15.178:0/2655232864 learned_addr learned my addr 172.21.15.178:0/2655232864 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:09:09.150 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:09.146+0000 7f0af7fff700 1 -- 172.21.15.178:0/2655232864 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ae8005040 con 0x7f0af80f51a0 2024-02-11T17:09:09.150 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:09.146+0000 7f0af7fff700 1 --2- 172.21.15.178:0/2655232864 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0af80f51a0 0x7f0af80fd120 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f0ae8013510 tx=0x7f0ae800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:09:09.150 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:09.147+0000 7f0af4ff9700 1 -- 172.21.15.178:0/2655232864 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ae8009320 con 0x7f0af80f51a0 2024-02-11T17:09:09.150 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:09.147+0000 7f0afd727700 1 -- 172.21.15.178:0/2655232864 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0af80fb740 con 0x7f0af80f51a0 2024-02-11T17:09:09.150 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:09.147+0000 7f0af4ff9700 1 -- 172.21.15.178:0/2655232864 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0ae8013f00 con 0x7f0af80f51a0 2024-02-11T17:09:09.150 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:09.147+0000 7f0af4ff9700 1 -- 172.21.15.178:0/2655232864 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ae8014dc0 con 0x7f0af80f51a0 2024-02-11T17:09:09.150 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:09.147+0000 7f0afd727700 1 -- 172.21.15.178:0/2655232864 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0af80fbbd0 con 0x7f0af80f51a0 2024-02-11T17:09:09.150 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:09.147+0000 7f0af4ff9700 1 -- 172.21.15.178:0/2655232864 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0ae80095f0 con 0x7f0af80f51a0 2024-02-11T17:09:09.150 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:09.147+0000 7f0afd727700 1 -- 172.21.15.178:0/2655232864 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0af8059100 con 0x7f0af80f51a0 2024-02-11T17:09:09.150 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:09.148+0000 7f0af4ff9700 1 --2- 172.21.15.178:0/2655232864 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f0ae002ff60 0x7f0ae0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:09:09.150 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:09.148+0000 7f0af4ff9700 1 -- 172.21.15.178:0/2655232864 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0ae8041000 con 0x7f0af80f51a0 2024-02-11T17:09:09.150 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:09.148+0000 7f0af77fe700 1 --2- 172.21.15.178:0/2655232864 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f0ae002ff60 0x7f0ae0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:09:09.151 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:09.148+0000 7f0af77fe700 1 --2- 172.21.15.178:0/2655232864 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f0ae002ff60 0x7f0ae0032420 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f0aec0096a0 tx=0x7f0aec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:09:09.154 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:09.152+0000 7f0af4ff9700 1 -- 172.21.15.178:0/2655232864 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0ae8014420 con 0x7f0af80f51a0 2024-02-11T17:09:09.355 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:09.353+0000 7f0afd727700 1 -- 172.21.15.178:0/2655232864 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0af80fabe0 con 0x7f0af80f51a0 2024-02-11T17:09:09.356 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:09.354+0000 7f0af4ff9700 1 -- 172.21.15.178:0/2655232864 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0ae8010230 con 0x7f0af80f51a0 2024-02-11T17:09:09.356 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:09:09.356 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:09:09.358 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:09.356+0000 7f0afd727700 1 -- 172.21.15.178:0/2655232864 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f0ae002ff60 msgr2=0x7f0ae0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:09:09.358 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:09.356+0000 7f0afd727700 1 --2- 172.21.15.178:0/2655232864 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f0ae002ff60 0x7f0ae0032420 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f0aec0096a0 tx=0x7f0aec006b40).stop 2024-02-11T17:09:09.358 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:09.356+0000 7f0afd727700 1 -- 172.21.15.178:0/2655232864 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0af80f51a0 msgr2=0x7f0af80fd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:09:09.358 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:09.356+0000 7f0afd727700 1 --2- 172.21.15.178:0/2655232864 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0af80f51a0 0x7f0af80fd120 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f0ae8013510 tx=0x7f0ae800b040).stop 2024-02-11T17:09:09.358 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:09.356+0000 7f0afd727700 1 -- 172.21.15.178:0/2655232864 shutdown_connections 2024-02-11T17:09:09.358 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:09.356+0000 7f0afd727700 1 --2- 172.21.15.178:0/2655232864 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f0ae002ff60 0x7f0ae0032420 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:09:09.358 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:09.356+0000 7f0afd727700 1 --2- 172.21.15.178:0/2655232864 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0af80f51a0 0x7f0af80fd120 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:09:09.358 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:09.356+0000 7f0afd727700 1 -- 172.21.15.178:0/2655232864 >> 172.21.15.178:0/2655232864 conn(0x7f0af80f09a0 msgr2=0x7f0af805fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:09:09.358 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:09.356+0000 7f0afd727700 1 -- 172.21.15.178:0/2655232864 shutdown_connections 2024-02-11T17:09:09.358 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:09.356+0000 7f0afd727700 1 -- 172.21.15.178:0/2655232864 wait complete. 2024-02-11T17:09:09.359 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:09:10.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:09:10 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:09:09.215506+0000 mgr.smithi037.eofnaq (mgr.14162) 281 : cluster [DBG] pgmap v267: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:09:10.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:09:10 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:09:09.355448+0000 mon.smithi037 (mon.0) 278 : audit [DBG] from='client.? 172.21.15.178:0/2655232864' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:09:11.086 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:09:11.086 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:09:12.684 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:09:12 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:09:11.215832+0000 mgr.smithi037.eofnaq (mgr.14162) 282 : cluster [DBG] pgmap v268: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:09:13.062 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:13.059+0000 7f5e761fa700 1 -- 172.21.15.178:0/238562964 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e700f6510 msgr2=0x7f5e700f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:09:13.062 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:13.059+0000 7f5e761fa700 1 --2- 172.21.15.178:0/238562964 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e700f6510 0x7f5e700f6930 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f5e60009fb0 tx=0x7f5e60009b30).stop 2024-02-11T17:09:13.062 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:13.059+0000 7f5e761fa700 1 -- 172.21.15.178:0/238562964 shutdown_connections 2024-02-11T17:09:13.062 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:13.059+0000 7f5e761fa700 1 --2- 172.21.15.178:0/238562964 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e700f6510 0x7f5e700f6930 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:09:13.062 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:13.059+0000 7f5e761fa700 1 -- 172.21.15.178:0/238562964 >> 172.21.15.178:0/238562964 conn(0x7f5e700f1c70 msgr2=0x7f5e700f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:09:13.062 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:13.059+0000 7f5e761fa700 1 -- 172.21.15.178:0/238562964 shutdown_connections 2024-02-11T17:09:13.062 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:13.059+0000 7f5e761fa700 1 -- 172.21.15.178:0/238562964 wait complete. 2024-02-11T17:09:13.062 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:13.059+0000 7f5e761fa700 1 Processor -- start 2024-02-11T17:09:13.062 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:13.059+0000 7f5e761fa700 1 -- start start 2024-02-11T17:09:13.062 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:13.059+0000 7f5e761fa700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e700f6510 0x7f5e70106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:09:13.062 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:13.059+0000 7f5e761fa700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e70106c20 con 0x7f5e700f6510 2024-02-11T17:09:13.063 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:13.060+0000 7f5e6f7fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e700f6510 0x7f5e70106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:09:13.063 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:13.060+0000 7f5e6f7fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e700f6510 0x7f5e70106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:45800/0 (socket says 172.21.15.178:45800) 2024-02-11T17:09:13.063 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:13.060+0000 7f5e6f7fe700 1 -- 172.21.15.178:0/1722502172 learned_addr learned my addr 172.21.15.178:0/1722502172 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:09:13.063 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:13.060+0000 7f5e6f7fe700 1 -- 172.21.15.178:0/1722502172 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e60005040 con 0x7f5e700f6510 2024-02-11T17:09:13.063 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:13.060+0000 7f5e6f7fe700 1 --2- 172.21.15.178:0/1722502172 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e700f6510 0x7f5e70106730 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f5e60013510 tx=0x7f5e6000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:09:13.063 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:13.060+0000 7f5e6cff9700 1 -- 172.21.15.178:0/1722502172 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e60009320 con 0x7f5e700f6510 2024-02-11T17:09:13.063 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:13.060+0000 7f5e761fa700 1 -- 172.21.15.178:0/1722502172 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e70106e20 con 0x7f5e700f6510 2024-02-11T17:09:13.063 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:13.060+0000 7f5e6cff9700 1 -- 172.21.15.178:0/1722502172 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5e60013f00 con 0x7f5e700f6510 2024-02-11T17:09:13.063 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:13.060+0000 7f5e6cff9700 1 -- 172.21.15.178:0/1722502172 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e60014d90 con 0x7f5e700f6510 2024-02-11T17:09:13.063 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:13.060+0000 7f5e761fa700 1 -- 172.21.15.178:0/1722502172 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e701072b0 con 0x7f5e700f6510 2024-02-11T17:09:13.064 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:13.061+0000 7f5e6cff9700 1 -- 172.21.15.178:0/1722502172 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5e600095f0 con 0x7f5e700f6510 2024-02-11T17:09:13.064 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:13.061+0000 7f5e761fa700 1 -- 172.21.15.178:0/1722502172 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e70045ad0 con 0x7f5e700f6510 2024-02-11T17:09:13.064 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:13.061+0000 7f5e6cff9700 1 --2- 172.21.15.178:0/1722502172 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f5e5c02ff60 0x7f5e5c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:09:13.064 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:13.061+0000 7f5e6cff9700 1 -- 172.21.15.178:0/1722502172 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5e60040fe0 con 0x7f5e700f6510 2024-02-11T17:09:13.065 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:13.062+0000 7f5e67fff700 1 --2- 172.21.15.178:0/1722502172 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f5e5c02ff60 0x7f5e5c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:09:13.065 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:13.062+0000 7f5e67fff700 1 --2- 172.21.15.178:0/1722502172 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f5e5c02ff60 0x7f5e5c032420 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f5e580096a0 tx=0x7f5e58006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:09:13.068 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:13.066+0000 7f5e6cff9700 1 -- 172.21.15.178:0/1722502172 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5e6001d7f0 con 0x7f5e700f6510 2024-02-11T17:09:13.269 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:13.267+0000 7f5e761fa700 1 -- 172.21.15.178:0/1722502172 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5e700fbf60 con 0x7f5e700f6510 2024-02-11T17:09:13.270 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:13.268+0000 7f5e6cff9700 1 -- 172.21.15.178:0/1722502172 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5e60010230 con 0x7f5e700f6510 2024-02-11T17:09:13.270 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:09:13.270 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:09:13.271 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:13.269+0000 7f5e761fa700 1 -- 172.21.15.178:0/1722502172 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f5e5c02ff60 msgr2=0x7f5e5c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:09:13.271 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:13.269+0000 7f5e761fa700 1 --2- 172.21.15.178:0/1722502172 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f5e5c02ff60 0x7f5e5c032420 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f5e580096a0 tx=0x7f5e58006b40).stop 2024-02-11T17:09:13.271 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:13.269+0000 7f5e761fa700 1 -- 172.21.15.178:0/1722502172 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e700f6510 msgr2=0x7f5e70106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:09:13.271 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:13.269+0000 7f5e761fa700 1 --2- 172.21.15.178:0/1722502172 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e700f6510 0x7f5e70106730 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f5e60013510 tx=0x7f5e6000b040).stop 2024-02-11T17:09:13.272 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:13.269+0000 7f5e761fa700 1 -- 172.21.15.178:0/1722502172 shutdown_connections 2024-02-11T17:09:13.272 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:13.269+0000 7f5e761fa700 1 --2- 172.21.15.178:0/1722502172 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f5e5c02ff60 0x7f5e5c032420 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:09:13.272 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:13.269+0000 7f5e761fa700 1 --2- 172.21.15.178:0/1722502172 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e700f6510 0x7f5e70106730 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:09:13.272 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:13.269+0000 7f5e761fa700 1 -- 172.21.15.178:0/1722502172 >> 172.21.15.178:0/1722502172 conn(0x7f5e700f1c70 msgr2=0x7f5e700f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:09:13.272 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:13.269+0000 7f5e761fa700 1 -- 172.21.15.178:0/1722502172 shutdown_connections 2024-02-11T17:09:13.272 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:13.269+0000 7f5e761fa700 1 -- 172.21.15.178:0/1722502172 wait complete. 2024-02-11T17:09:13.272 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:09:13.684 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:09:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:09:13.269300+0000 mon.smithi037 (mon.0) 279 : audit [DBG] from='client.? 172.21.15.178:0/1722502172' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:09:14.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:09:14 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:09:13.216114+0000 mgr.smithi037.eofnaq (mgr.14162) 283 : cluster [DBG] pgmap v269: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:09:15.060 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:09:15.060 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:09:16.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:09:16 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:09:15.216357+0000 mgr.smithi037.eofnaq (mgr.14162) 284 : cluster [DBG] pgmap v270: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:09:17.021 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:17.019+0000 7fe715aca700 1 -- 172.21.15.178:0/3390684228 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7100f51a0 msgr2=0x7fe7100f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:09:17.022 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:17.019+0000 7fe715aca700 1 --2- 172.21.15.178:0/3390684228 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7100f51a0 0x7fe7100f55c0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7fe704009fb0 tx=0x7fe704009b30).stop 2024-02-11T17:09:17.022 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:17.019+0000 7fe715aca700 1 -- 172.21.15.178:0/3390684228 shutdown_connections 2024-02-11T17:09:17.022 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:17.019+0000 7fe715aca700 1 --2- 172.21.15.178:0/3390684228 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7100f51a0 0x7fe7100f55c0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:09:17.022 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:17.019+0000 7fe715aca700 1 -- 172.21.15.178:0/3390684228 >> 172.21.15.178:0/3390684228 conn(0x7fe7100f09a0 msgr2=0x7fe7100f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:09:17.022 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:17.019+0000 7fe715aca700 1 -- 172.21.15.178:0/3390684228 shutdown_connections 2024-02-11T17:09:17.022 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:17.019+0000 7fe715aca700 1 -- 172.21.15.178:0/3390684228 wait complete. 2024-02-11T17:09:17.022 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:17.019+0000 7fe715aca700 1 Processor -- start 2024-02-11T17:09:17.022 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:17.019+0000 7fe715aca700 1 -- start start 2024-02-11T17:09:17.022 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:17.020+0000 7fe715aca700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7100f51a0 0x7fe7101086d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:09:17.022 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:17.020+0000 7fe715aca700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe710108bc0 con 0x7fe7100f51a0 2024-02-11T17:09:17.022 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:17.020+0000 7fe714ac8700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7100f51a0 0x7fe7101086d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:09:17.022 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:17.020+0000 7fe714ac8700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7100f51a0 0x7fe7101086d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:45824/0 (socket says 172.21.15.178:45824) 2024-02-11T17:09:17.022 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:17.020+0000 7fe714ac8700 1 -- 172.21.15.178:0/4203709472 learned_addr learned my addr 172.21.15.178:0/4203709472 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:09:17.024 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:17.020+0000 7fe714ac8700 1 -- 172.21.15.178:0/4203709472 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe704005040 con 0x7fe7100f51a0 2024-02-11T17:09:17.024 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:17.020+0000 7fe714ac8700 1 --2- 172.21.15.178:0/4203709472 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7100f51a0 0x7fe7101086d0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7fe704013510 tx=0x7fe70400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:09:17.024 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:17.020+0000 7fe70d7fa700 1 -- 172.21.15.178:0/4203709472 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe704009320 con 0x7fe7100f51a0 2024-02-11T17:09:17.024 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:17.020+0000 7fe715aca700 1 -- 172.21.15.178:0/4203709472 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe710108dc0 con 0x7fe7100f51a0 2024-02-11T17:09:17.024 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:17.020+0000 7fe70d7fa700 1 -- 172.21.15.178:0/4203709472 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe704013f00 con 0x7fe7100f51a0 2024-02-11T17:09:17.024 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:17.020+0000 7fe70d7fa700 1 -- 172.21.15.178:0/4203709472 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe704014dc0 con 0x7fe7100f51a0 2024-02-11T17:09:17.024 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:17.021+0000 7fe715aca700 1 -- 172.21.15.178:0/4203709472 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe710109250 con 0x7fe7100f51a0 2024-02-11T17:09:17.024 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:17.021+0000 7fe70d7fa700 1 -- 172.21.15.178:0/4203709472 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe704014420 con 0x7fe7100f51a0 2024-02-11T17:09:17.024 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:17.021+0000 7fe70d7fa700 1 --2- 172.21.15.178:0/4203709472 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fe70002ff60 0x7fe700032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:09:17.025 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:17.021+0000 7fe715aca700 1 -- 172.21.15.178:0/4203709472 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe710045ad0 con 0x7fe7100f51a0 2024-02-11T17:09:17.025 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:17.021+0000 7fe70d7fa700 1 -- 172.21.15.178:0/4203709472 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe704041110 con 0x7fe7100f51a0 2024-02-11T17:09:17.025 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:17.022+0000 7fe70ffff700 1 --2- 172.21.15.178:0/4203709472 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fe70002ff60 0x7fe700032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:09:17.025 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:17.022+0000 7fe70ffff700 1 --2- 172.21.15.178:0/4203709472 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fe70002ff60 0x7fe700032420 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fe6fc0096a0 tx=0x7fe6fc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:09:17.028 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:17.026+0000 7fe70d7fa700 1 -- 172.21.15.178:0/4203709472 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe7040095f0 con 0x7fe7100f51a0 2024-02-11T17:09:17.232 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:17.230+0000 7fe715aca700 1 -- 172.21.15.178:0/4203709472 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe71010a100 con 0x7fe7100f51a0 2024-02-11T17:09:17.232 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:17.231+0000 7fe70d7fa700 1 -- 172.21.15.178:0/4203709472 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe70401cb10 con 0x7fe7100f51a0 2024-02-11T17:09:17.232 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:09:17.233 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:09:17.234 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:17.232+0000 7fe715aca700 1 -- 172.21.15.178:0/4203709472 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fe70002ff60 msgr2=0x7fe700032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:09:17.234 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:17.232+0000 7fe715aca700 1 --2- 172.21.15.178:0/4203709472 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fe70002ff60 0x7fe700032420 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fe6fc0096a0 tx=0x7fe6fc006b40).stop 2024-02-11T17:09:17.234 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:17.232+0000 7fe715aca700 1 -- 172.21.15.178:0/4203709472 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7100f51a0 msgr2=0x7fe7101086d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:09:17.234 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:17.232+0000 7fe715aca700 1 --2- 172.21.15.178:0/4203709472 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7100f51a0 0x7fe7101086d0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7fe704013510 tx=0x7fe70400b040).stop 2024-02-11T17:09:17.234 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:17.232+0000 7fe715aca700 1 -- 172.21.15.178:0/4203709472 shutdown_connections 2024-02-11T17:09:17.234 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:17.232+0000 7fe715aca700 1 --2- 172.21.15.178:0/4203709472 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fe70002ff60 0x7fe700032420 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:09:17.234 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:17.232+0000 7fe715aca700 1 --2- 172.21.15.178:0/4203709472 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe7100f51a0 0x7fe7101086d0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:09:17.234 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:17.232+0000 7fe715aca700 1 -- 172.21.15.178:0/4203709472 >> 172.21.15.178:0/4203709472 conn(0x7fe7100f09a0 msgr2=0x7fe71005fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:09:17.235 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:17.232+0000 7fe715aca700 1 -- 172.21.15.178:0/4203709472 shutdown_connections 2024-02-11T17:09:17.235 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:17.232+0000 7fe715aca700 1 -- 172.21.15.178:0/4203709472 wait complete. 2024-02-11T17:09:17.235 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:09:17.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:09:17 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:09:17.232017+0000 mon.smithi037 (mon.0) 280 : audit [DBG] from='client.? 172.21.15.178:0/4203709472' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:09:18.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:09:18 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:09:17.216643+0000 mgr.smithi037.eofnaq (mgr.14162) 285 : cluster [DBG] pgmap v271: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:09:18.960 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:09:18.961 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:09:20.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:09:20 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:09:19.216988+0000 mgr.smithi037.eofnaq (mgr.14162) 286 : cluster [DBG] pgmap v272: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:09:20.978 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:20.975+0000 7fbae3785700 1 -- 172.21.15.178:0/937736412 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbadc0f5410 msgr2=0x7fbadc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:09:20.978 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:20.975+0000 7fbae3785700 1 --2- 172.21.15.178:0/937736412 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbadc0f5410 0x7fbadc0f5830 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7fbad0009fb0 tx=0x7fbad0009b30).stop 2024-02-11T17:09:20.978 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:20.975+0000 7fbae3785700 1 -- 172.21.15.178:0/937736412 shutdown_connections 2024-02-11T17:09:20.979 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:20.975+0000 7fbae3785700 1 --2- 172.21.15.178:0/937736412 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbadc0f5410 0x7fbadc0f5830 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:09:20.979 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:20.975+0000 7fbae3785700 1 -- 172.21.15.178:0/937736412 >> 172.21.15.178:0/937736412 conn(0x7fbadc0f0b70 msgr2=0x7fbadc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:09:20.979 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:20.975+0000 7fbae3785700 1 -- 172.21.15.178:0/937736412 shutdown_connections 2024-02-11T17:09:20.979 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:20.975+0000 7fbae3785700 1 -- 172.21.15.178:0/937736412 wait complete. 2024-02-11T17:09:20.979 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:20.976+0000 7fbae3785700 1 Processor -- start 2024-02-11T17:09:20.979 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:20.976+0000 7fbae3785700 1 -- start start 2024-02-11T17:09:20.979 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:20.976+0000 7fbae3785700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbadc0f5410 0x7fbadc105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:09:20.979 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:20.976+0000 7fbae3785700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbadc105b20 con 0x7fbadc0f5410 2024-02-11T17:09:20.979 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:20.976+0000 7fbae1521700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbadc0f5410 0x7fbadc105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:09:20.979 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:20.976+0000 7fbae1521700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbadc0f5410 0x7fbadc105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:49906/0 (socket says 172.21.15.178:49906) 2024-02-11T17:09:20.979 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:20.976+0000 7fbae1521700 1 -- 172.21.15.178:0/980237076 learned_addr learned my addr 172.21.15.178:0/980237076 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:09:20.979 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:20.977+0000 7fbae1521700 1 -- 172.21.15.178:0/980237076 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbad0005040 con 0x7fbadc0f5410 2024-02-11T17:09:20.979 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:20.977+0000 7fbae1521700 1 --2- 172.21.15.178:0/980237076 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbadc0f5410 0x7fbadc105630 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7fbad0014000 tx=0x7fbad000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:09:20.980 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:20.977+0000 7fbacdffb700 1 -- 172.21.15.178:0/980237076 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbad00095f0 con 0x7fbadc0f5410 2024-02-11T17:09:20.980 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:20.977+0000 7fbae3785700 1 -- 172.21.15.178:0/980237076 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbadc105d20 con 0x7fbadc0f5410 2024-02-11T17:09:20.980 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:20.977+0000 7fbacdffb700 1 -- 172.21.15.178:0/980237076 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbad0007e70 con 0x7fbadc0f5410 2024-02-11T17:09:20.980 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:20.977+0000 7fbacdffb700 1 -- 172.21.15.178:0/980237076 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbad0014ea0 con 0x7fbadc0f5410 2024-02-11T17:09:20.981 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:20.977+0000 7fbae3785700 1 -- 172.21.15.178:0/980237076 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbadc1061b0 con 0x7fbadc0f5410 2024-02-11T17:09:20.981 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:20.978+0000 7fbacdffb700 1 -- 172.21.15.178:0/980237076 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbad0025070 con 0x7fbadc0f5410 2024-02-11T17:09:20.981 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:20.978+0000 7fbae3785700 1 -- 172.21.15.178:0/980237076 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbadc045ad0 con 0x7fbadc0f5410 2024-02-11T17:09:20.981 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:20.978+0000 7fbacdffb700 1 --2- 172.21.15.178:0/980237076 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fbac802ff60 0x7fbac8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:09:20.981 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:20.978+0000 7fbacdffb700 1 -- 172.21.15.178:0/980237076 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbad00409f0 con 0x7fbadc0f5410 2024-02-11T17:09:20.981 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:20.978+0000 7fbae0d20700 1 --2- 172.21.15.178:0/980237076 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fbac802ff60 0x7fbac8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:09:20.981 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:20.979+0000 7fbae0d20700 1 --2- 172.21.15.178:0/980237076 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fbac802ff60 0x7fbac8032420 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fbad80096a0 tx=0x7fbad8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:09:20.984 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:20.983+0000 7fbacdffb700 1 -- 172.21.15.178:0/980237076 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbad000f4e0 con 0x7fbadc0f5410 2024-02-11T17:09:21.191 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:21.190+0000 7fbae3785700 1 -- 172.21.15.178:0/980237076 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbadc0fae60 con 0x7fbadc0f5410 2024-02-11T17:09:21.192 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:21.190+0000 7fbacdffb700 1 -- 172.21.15.178:0/980237076 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbad000f300 con 0x7fbadc0f5410 2024-02-11T17:09:21.192 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:09:21.192 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:09:21.194 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:21.192+0000 7fbae3785700 1 -- 172.21.15.178:0/980237076 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fbac802ff60 msgr2=0x7fbac8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:09:21.194 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:21.192+0000 7fbae3785700 1 --2- 172.21.15.178:0/980237076 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fbac802ff60 0x7fbac8032420 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fbad80096a0 tx=0x7fbad8006b40).stop 2024-02-11T17:09:21.194 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:21.192+0000 7fbae3785700 1 -- 172.21.15.178:0/980237076 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbadc0f5410 msgr2=0x7fbadc105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:09:21.194 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:21.192+0000 7fbae3785700 1 --2- 172.21.15.178:0/980237076 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbadc0f5410 0x7fbadc105630 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7fbad0014000 tx=0x7fbad000b040).stop 2024-02-11T17:09:21.194 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:21.192+0000 7fbae3785700 1 -- 172.21.15.178:0/980237076 shutdown_connections 2024-02-11T17:09:21.194 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:21.192+0000 7fbae3785700 1 --2- 172.21.15.178:0/980237076 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fbac802ff60 0x7fbac8032420 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:09:21.194 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:21.192+0000 7fbae3785700 1 --2- 172.21.15.178:0/980237076 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbadc0f5410 0x7fbadc105630 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:09:21.194 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:21.192+0000 7fbae3785700 1 -- 172.21.15.178:0/980237076 >> 172.21.15.178:0/980237076 conn(0x7fbadc0f0b70 msgr2=0x7fbadc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:09:21.195 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:21.192+0000 7fbae3785700 1 -- 172.21.15.178:0/980237076 shutdown_connections 2024-02-11T17:09:21.195 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:21.192+0000 7fbae3785700 1 -- 172.21.15.178:0/980237076 wait complete. 2024-02-11T17:09:21.195 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:09:21.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:09:21 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:09:21.191740+0000 mon.smithi037 (mon.0) 281 : audit [DBG] from='client.? 172.21.15.178:0/980237076' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:09:22.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:09:22 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:09:21.217339+0000 mgr.smithi037.eofnaq (mgr.14162) 287 : cluster [DBG] pgmap v273: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:09:22.880 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:09:22.881 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:09:24.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:09:24 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:09:23.217738+0000 mgr.smithi037.eofnaq (mgr.14162) 288 : cluster [DBG] pgmap v274: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:09:25.000 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:24.998+0000 7f20965d8700 1 -- 172.21.15.178:0/1466782649 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f20900f6510 msgr2=0x7f20900f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:09:25.001 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:24.999+0000 7f20965d8700 1 --2- 172.21.15.178:0/1466782649 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f20900f6510 0x7f20900f6930 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7f2078009fb0 tx=0x7f2078009b30).stop 2024-02-11T17:09:25.001 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:24.999+0000 7f20965d8700 1 -- 172.21.15.178:0/1466782649 shutdown_connections 2024-02-11T17:09:25.001 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:24.999+0000 7f20965d8700 1 --2- 172.21.15.178:0/1466782649 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f20900f6510 0x7f20900f6930 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:09:25.001 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:24.999+0000 7f20965d8700 1 -- 172.21.15.178:0/1466782649 >> 172.21.15.178:0/1466782649 conn(0x7f20900f1c70 msgr2=0x7f20900f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:09:25.001 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:24.999+0000 7f20965d8700 1 -- 172.21.15.178:0/1466782649 shutdown_connections 2024-02-11T17:09:25.001 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:24.999+0000 7f20965d8700 1 -- 172.21.15.178:0/1466782649 wait complete. 2024-02-11T17:09:25.001 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:24.999+0000 7f20965d8700 1 Processor -- start 2024-02-11T17:09:25.002 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:24.999+0000 7f20965d8700 1 -- start start 2024-02-11T17:09:25.002 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:24.999+0000 7f20965d8700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f20900f6510 0x7f209006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:09:25.003 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:25.000+0000 7f20965d8700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2090069e20 con 0x7f20900f6510 2024-02-11T17:09:25.003 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:25.000+0000 7f208ffff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f20900f6510 0x7f209006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:09:25.003 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:25.000+0000 7f208ffff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f20900f6510 0x7f209006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:49934/0 (socket says 172.21.15.178:49934) 2024-02-11T17:09:25.003 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:25.000+0000 7f208ffff700 1 -- 172.21.15.178:0/3289249401 learned_addr learned my addr 172.21.15.178:0/3289249401 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:09:25.003 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:25.000+0000 7f208ffff700 1 -- 172.21.15.178:0/3289249401 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2078005040 con 0x7f20900f6510 2024-02-11T17:09:25.003 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:25.000+0000 7f208ffff700 1 --2- 172.21.15.178:0/3289249401 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f20900f6510 0x7f209006b9a0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f2078014000 tx=0x7f207800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:09:25.003 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:25.000+0000 7f208cff9700 1 -- 172.21.15.178:0/3289249401 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f20780095f0 con 0x7f20900f6510 2024-02-11T17:09:25.003 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:25.000+0000 7f208cff9700 1 -- 172.21.15.178:0/3289249401 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2078007e70 con 0x7f20900f6510 2024-02-11T17:09:25.003 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:25.000+0000 7f208cff9700 1 -- 172.21.15.178:0/3289249401 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2078014ea0 con 0x7f20900f6510 2024-02-11T17:09:25.003 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:25.000+0000 7f20965d8700 1 -- 172.21.15.178:0/3289249401 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2090069fc0 con 0x7f20900f6510 2024-02-11T17:09:25.003 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:25.000+0000 7f20965d8700 1 -- 172.21.15.178:0/3289249401 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f209006a450 con 0x7f20900f6510 2024-02-11T17:09:25.004 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:25.001+0000 7f208cff9700 1 -- 172.21.15.178:0/3289249401 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2078025070 con 0x7f20900f6510 2024-02-11T17:09:25.004 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:25.001+0000 7f20965d8700 1 -- 172.21.15.178:0/3289249401 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2090059100 con 0x7f20900f6510 2024-02-11T17:09:25.004 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:25.001+0000 7f208cff9700 1 --2- 172.21.15.178:0/3289249401 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f207c02ff60 0x7f207c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:09:25.004 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:25.002+0000 7f208cff9700 1 -- 172.21.15.178:0/3289249401 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2078040970 con 0x7f20900f6510 2024-02-11T17:09:25.004 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:25.002+0000 7f208f7fe700 1 --2- 172.21.15.178:0/3289249401 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f207c02ff60 0x7f207c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:09:25.004 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:25.002+0000 7f208f7fe700 1 --2- 172.21.15.178:0/3289249401 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f207c02ff60 0x7f207c032420 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f20800096a0 tx=0x7f2080006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:09:25.008 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:25.006+0000 7f208cff9700 1 -- 172.21.15.178:0/3289249401 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f207800f4e0 con 0x7f20900f6510 2024-02-11T17:09:25.216 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:25.214+0000 7f20965d8700 1 -- 172.21.15.178:0/3289249401 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f20900fbf60 con 0x7f20900f6510 2024-02-11T17:09:25.216 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:25.215+0000 7f208cff9700 1 -- 172.21.15.178:0/3289249401 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f207800f300 con 0x7f20900f6510 2024-02-11T17:09:25.217 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:09:25.217 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:09:25.218 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:25.216+0000 7f20965d8700 1 -- 172.21.15.178:0/3289249401 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f207c02ff60 msgr2=0x7f207c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:09:25.218 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:25.216+0000 7f20965d8700 1 --2- 172.21.15.178:0/3289249401 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f207c02ff60 0x7f207c032420 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f20800096a0 tx=0x7f2080006b40).stop 2024-02-11T17:09:25.218 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:25.216+0000 7f20965d8700 1 -- 172.21.15.178:0/3289249401 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f20900f6510 msgr2=0x7f209006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:09:25.218 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:25.216+0000 7f20965d8700 1 --2- 172.21.15.178:0/3289249401 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f20900f6510 0x7f209006b9a0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f2078014000 tx=0x7f207800b040).stop 2024-02-11T17:09:25.218 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:25.216+0000 7f20965d8700 1 -- 172.21.15.178:0/3289249401 shutdown_connections 2024-02-11T17:09:25.218 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:25.216+0000 7f20965d8700 1 --2- 172.21.15.178:0/3289249401 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f207c02ff60 0x7f207c032420 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:09:25.219 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:25.216+0000 7f20965d8700 1 --2- 172.21.15.178:0/3289249401 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f20900f6510 0x7f209006b9a0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:09:25.219 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:25.216+0000 7f20965d8700 1 -- 172.21.15.178:0/3289249401 >> 172.21.15.178:0/3289249401 conn(0x7f20900f1c70 msgr2=0x7f20900f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:09:25.219 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:25.216+0000 7f20965d8700 1 -- 172.21.15.178:0/3289249401 shutdown_connections 2024-02-11T17:09:25.219 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:25.216+0000 7f20965d8700 1 -- 172.21.15.178:0/3289249401 wait complete. 2024-02-11T17:09:25.219 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:09:26.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:09:26 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:09:25.216111+0000 mon.smithi037 (mon.0) 282 : audit [DBG] from='client.? 172.21.15.178:0/3289249401' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:09:26.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:09:26 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:09:25.218182+0000 mgr.smithi037.eofnaq (mgr.14162) 289 : cluster [DBG] pgmap v275: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:09:26.897 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:09:26.898 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:09:28.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:09:28 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:09:27.218624+0000 mgr.smithi037.eofnaq (mgr.14162) 290 : cluster [DBG] pgmap v276: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:09:28.852 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:28.849+0000 7fcffd1ed700 1 -- 172.21.15.178:0/3383136533 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcff80f5410 msgr2=0x7fcff80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:09:28.853 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:28.849+0000 7fcffd1ed700 1 --2- 172.21.15.178:0/3383136533 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcff80f5410 0x7fcff80f5830 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7fcfe0009fb0 tx=0x7fcfe0009b30).stop 2024-02-11T17:09:28.853 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:28.849+0000 7fcffd1ed700 1 -- 172.21.15.178:0/3383136533 shutdown_connections 2024-02-11T17:09:28.853 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:28.849+0000 7fcffd1ed700 1 --2- 172.21.15.178:0/3383136533 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcff80f5410 0x7fcff80f5830 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:09:28.853 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:28.849+0000 7fcffd1ed700 1 -- 172.21.15.178:0/3383136533 >> 172.21.15.178:0/3383136533 conn(0x7fcff80f0b70 msgr2=0x7fcff80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:09:28.853 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:28.849+0000 7fcffd1ed700 1 -- 172.21.15.178:0/3383136533 shutdown_connections 2024-02-11T17:09:28.853 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:28.849+0000 7fcffd1ed700 1 -- 172.21.15.178:0/3383136533 wait complete. 2024-02-11T17:09:28.853 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:28.849+0000 7fcffd1ed700 1 Processor -- start 2024-02-11T17:09:28.853 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:28.849+0000 7fcffd1ed700 1 -- start start 2024-02-11T17:09:28.853 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:28.850+0000 7fcffd1ed700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcff80f5410 0x7fcff8106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:09:28.853 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:28.850+0000 7fcffd1ed700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcff8106c10 con 0x7fcff80f5410 2024-02-11T17:09:28.853 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:28.850+0000 7fcff6d9d700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcff80f5410 0x7fcff8106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:09:28.853 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:28.850+0000 7fcff6d9d700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcff80f5410 0x7fcff8106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:49952/0 (socket says 172.21.15.178:49952) 2024-02-11T17:09:28.854 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:28.850+0000 7fcff6d9d700 1 -- 172.21.15.178:0/3679010646 learned_addr learned my addr 172.21.15.178:0/3679010646 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:09:28.854 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:28.850+0000 7fcff6d9d700 1 -- 172.21.15.178:0/3679010646 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfe0005040 con 0x7fcff80f5410 2024-02-11T17:09:28.854 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:28.850+0000 7fcff6d9d700 1 --2- 172.21.15.178:0/3679010646 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcff80f5410 0x7fcff8106720 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7fcfe0013510 tx=0x7fcfe000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:09:28.854 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:28.850+0000 7fcfef7fe700 1 -- 172.21.15.178:0/3679010646 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcfe0009320 con 0x7fcff80f5410 2024-02-11T17:09:28.854 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:28.850+0000 7fcffd1ed700 1 -- 172.21.15.178:0/3679010646 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcff8106e10 con 0x7fcff80f5410 2024-02-11T17:09:28.854 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:28.850+0000 7fcfef7fe700 1 -- 172.21.15.178:0/3679010646 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcfe0013f00 con 0x7fcff80f5410 2024-02-11T17:09:28.854 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:28.850+0000 7fcfef7fe700 1 -- 172.21.15.178:0/3679010646 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcfe0014d90 con 0x7fcff80f5410 2024-02-11T17:09:28.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:28.851+0000 7fcffd1ed700 1 -- 172.21.15.178:0/3679010646 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcff81072a0 con 0x7fcff80f5410 2024-02-11T17:09:28.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:28.851+0000 7fcfef7fe700 1 -- 172.21.15.178:0/3679010646 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcfe0014420 con 0x7fcff80f5410 2024-02-11T17:09:28.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:28.851+0000 7fcffd1ed700 1 -- 172.21.15.178:0/3679010646 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcff8045ad0 con 0x7fcff80f5410 2024-02-11T17:09:28.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:28.851+0000 7fcfef7fe700 1 --2- 172.21.15.178:0/3679010646 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fcfe402ff60 0x7fcfe4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:09:28.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:28.851+0000 7fcfef7fe700 1 -- 172.21.15.178:0/3679010646 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcfe00410c0 con 0x7fcff80f5410 2024-02-11T17:09:28.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:28.852+0000 7fcff659c700 1 --2- 172.21.15.178:0/3679010646 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fcfe402ff60 0x7fcfe4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:09:28.855 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:28.852+0000 7fcff659c700 1 --2- 172.21.15.178:0/3679010646 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fcfe402ff60 0x7fcfe4032420 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fcfe80096a0 tx=0x7fcfe8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:09:28.858 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:28.856+0000 7fcfef7fe700 1 -- 172.21.15.178:0/3679010646 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcfe00095f0 con 0x7fcff80f5410 2024-02-11T17:09:29.062 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:29.060+0000 7fcffd1ed700 1 -- 172.21.15.178:0/3679010646 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcff8108150 con 0x7fcff80f5410 2024-02-11T17:09:29.062 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:29.061+0000 7fcfef7fe700 1 -- 172.21.15.178:0/3679010646 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcfe001db70 con 0x7fcff80f5410 2024-02-11T17:09:29.063 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:09:29.063 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:09:29.064 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:29.062+0000 7fcffd1ed700 1 -- 172.21.15.178:0/3679010646 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fcfe402ff60 msgr2=0x7fcfe4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:09:29.064 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:29.062+0000 7fcffd1ed700 1 --2- 172.21.15.178:0/3679010646 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fcfe402ff60 0x7fcfe4032420 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fcfe80096a0 tx=0x7fcfe8006b40).stop 2024-02-11T17:09:29.064 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:29.062+0000 7fcffd1ed700 1 -- 172.21.15.178:0/3679010646 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcff80f5410 msgr2=0x7fcff8106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:09:29.064 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:29.062+0000 7fcffd1ed700 1 --2- 172.21.15.178:0/3679010646 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcff80f5410 0x7fcff8106720 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7fcfe0013510 tx=0x7fcfe000b040).stop 2024-02-11T17:09:29.064 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:29.063+0000 7fcffd1ed700 1 -- 172.21.15.178:0/3679010646 shutdown_connections 2024-02-11T17:09:29.065 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:29.063+0000 7fcffd1ed700 1 --2- 172.21.15.178:0/3679010646 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fcfe402ff60 0x7fcfe4032420 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:09:29.065 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:29.063+0000 7fcffd1ed700 1 --2- 172.21.15.178:0/3679010646 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcff80f5410 0x7fcff8106720 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:09:29.065 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:29.063+0000 7fcffd1ed700 1 -- 172.21.15.178:0/3679010646 >> 172.21.15.178:0/3679010646 conn(0x7fcff80f0b70 msgr2=0x7fcff805fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:09:29.065 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:29.063+0000 7fcffd1ed700 1 -- 172.21.15.178:0/3679010646 shutdown_connections 2024-02-11T17:09:29.065 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:29.063+0000 7fcffd1ed700 1 -- 172.21.15.178:0/3679010646 wait complete. 2024-02-11T17:09:29.065 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:09:29.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:09:29 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:09:29.062285+0000 mon.smithi037 (mon.0) 283 : audit [DBG] from='client.? 172.21.15.178:0/3679010646' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:09:30.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:09:30 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:09:29.218952+0000 mgr.smithi037.eofnaq (mgr.14162) 291 : cluster [DBG] pgmap v277: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:09:30.687 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:09:30.688 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:09:32.637 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:32.636+0000 7fb03b1c0700 1 -- 172.21.15.178:0/2054998681 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb0340f51a0 msgr2=0x7fb0340f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:09:32.637 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:32.636+0000 7fb03b1c0700 1 --2- 172.21.15.178:0/2054998681 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb0340f51a0 0x7fb0340f55c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7fb030009fb0 tx=0x7fb030009b30).stop 2024-02-11T17:09:32.638 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:32.636+0000 7fb03b1c0700 1 -- 172.21.15.178:0/2054998681 shutdown_connections 2024-02-11T17:09:32.638 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:32.636+0000 7fb03b1c0700 1 --2- 172.21.15.178:0/2054998681 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb0340f51a0 0x7fb0340f55c0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:09:32.638 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:32.636+0000 7fb03b1c0700 1 -- 172.21.15.178:0/2054998681 >> 172.21.15.178:0/2054998681 conn(0x7fb0340f09a0 msgr2=0x7fb0340f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:09:32.638 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:32.636+0000 7fb03b1c0700 1 -- 172.21.15.178:0/2054998681 shutdown_connections 2024-02-11T17:09:32.638 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:32.636+0000 7fb03b1c0700 1 -- 172.21.15.178:0/2054998681 wait complete. 2024-02-11T17:09:32.638 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:32.636+0000 7fb03b1c0700 1 Processor -- start 2024-02-11T17:09:32.638 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:32.637+0000 7fb03b1c0700 1 -- start start 2024-02-11T17:09:32.638 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:32.637+0000 7fb03b1c0700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb0340f51a0 0x7fb0340fd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:09:32.638 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:32.637+0000 7fb03b1c0700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0340fb5a0 con 0x7fb0340f51a0 2024-02-11T17:09:32.638 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:32.637+0000 7fb03a1be700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb0340f51a0 0x7fb0340fd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:09:32.638 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:32.637+0000 7fb03a1be700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb0340f51a0 0x7fb0340fd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:46650/0 (socket says 172.21.15.178:46650) 2024-02-11T17:09:32.638 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:32.637+0000 7fb03a1be700 1 -- 172.21.15.178:0/4104528633 learned_addr learned my addr 172.21.15.178:0/4104528633 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:09:32.640 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:32.637+0000 7fb03a1be700 1 -- 172.21.15.178:0/4104528633 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb030005040 con 0x7fb0340f51a0 2024-02-11T17:09:32.640 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:32.637+0000 7fb03a1be700 1 --2- 172.21.15.178:0/4104528633 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb0340f51a0 0x7fb0340fd120 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7fb030013510 tx=0x7fb03000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:09:32.640 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:32.637+0000 7fb02affd700 1 -- 172.21.15.178:0/4104528633 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb030009320 con 0x7fb0340f51a0 2024-02-11T17:09:32.640 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:32.637+0000 7fb03b1c0700 1 -- 172.21.15.178:0/4104528633 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0340fb740 con 0x7fb0340f51a0 2024-02-11T17:09:32.640 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:32.637+0000 7fb02affd700 1 -- 172.21.15.178:0/4104528633 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb030013f00 con 0x7fb0340f51a0 2024-02-11T17:09:32.640 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:32.637+0000 7fb02affd700 1 -- 172.21.15.178:0/4104528633 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb030014dc0 con 0x7fb0340f51a0 2024-02-11T17:09:32.640 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:32.638+0000 7fb03b1c0700 1 -- 172.21.15.178:0/4104528633 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0340fbbd0 con 0x7fb0340f51a0 2024-02-11T17:09:32.640 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:32.638+0000 7fb02affd700 1 -- 172.21.15.178:0/4104528633 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb030014420 con 0x7fb0340f51a0 2024-02-11T17:09:32.640 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:32.638+0000 7fb03b1c0700 1 -- 172.21.15.178:0/4104528633 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb034059100 con 0x7fb0340f51a0 2024-02-11T17:09:32.641 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:32.638+0000 7fb02affd700 1 --2- 172.21.15.178:0/4104528633 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb02002ff60 0x7fb020032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:09:32.641 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:32.638+0000 7fb02affd700 1 -- 172.21.15.178:0/4104528633 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb030041030 con 0x7fb0340f51a0 2024-02-11T17:09:32.641 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:32.639+0000 7fb0399bd700 1 --2- 172.21.15.178:0/4104528633 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb02002ff60 0x7fb020032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:09:32.641 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:32.639+0000 7fb0399bd700 1 --2- 172.21.15.178:0/4104528633 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb02002ff60 0x7fb020032420 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fb0240096a0 tx=0x7fb024006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:09:32.644 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:32.643+0000 7fb02affd700 1 -- 172.21.15.178:0/4104528633 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb0300095f0 con 0x7fb0340f51a0 2024-02-11T17:09:32.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:09:32 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:09:31.219220+0000 mgr.smithi037.eofnaq (mgr.14162) 292 : cluster [DBG] pgmap v278: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:09:32.850 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:32.849+0000 7fb03b1c0700 1 -- 172.21.15.178:0/4104528633 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb0340fabe0 con 0x7fb0340f51a0 2024-02-11T17:09:32.850 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:32.850+0000 7fb02affd700 1 -- 172.21.15.178:0/4104528633 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb03001cb10 con 0x7fb0340f51a0 2024-02-11T17:09:32.850 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:09:32.851 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:09:32.852 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:32.851+0000 7fb03b1c0700 1 -- 172.21.15.178:0/4104528633 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb02002ff60 msgr2=0x7fb020032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:09:32.852 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:32.851+0000 7fb03b1c0700 1 --2- 172.21.15.178:0/4104528633 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb02002ff60 0x7fb020032420 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fb0240096a0 tx=0x7fb024006b40).stop 2024-02-11T17:09:32.852 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:32.851+0000 7fb03b1c0700 1 -- 172.21.15.178:0/4104528633 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb0340f51a0 msgr2=0x7fb0340fd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:09:32.852 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:32.851+0000 7fb03b1c0700 1 --2- 172.21.15.178:0/4104528633 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb0340f51a0 0x7fb0340fd120 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7fb030013510 tx=0x7fb03000b040).stop 2024-02-11T17:09:32.852 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:32.851+0000 7fb03b1c0700 1 -- 172.21.15.178:0/4104528633 shutdown_connections 2024-02-11T17:09:32.852 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:32.851+0000 7fb03b1c0700 1 --2- 172.21.15.178:0/4104528633 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb02002ff60 0x7fb020032420 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:09:32.853 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:32.851+0000 7fb03b1c0700 1 --2- 172.21.15.178:0/4104528633 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb0340f51a0 0x7fb0340fd120 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:09:32.853 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:32.851+0000 7fb03b1c0700 1 -- 172.21.15.178:0/4104528633 >> 172.21.15.178:0/4104528633 conn(0x7fb0340f09a0 msgr2=0x7fb03405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:09:32.853 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:32.851+0000 7fb03b1c0700 1 -- 172.21.15.178:0/4104528633 shutdown_connections 2024-02-11T17:09:32.853 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:32.851+0000 7fb03b1c0700 1 -- 172.21.15.178:0/4104528633 wait complete. 2024-02-11T17:09:32.853 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:09:33.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:09:33 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:09:32.850130+0000 mon.smithi037 (mon.0) 284 : audit [DBG] from='client.? 172.21.15.178:0/4104528633' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:09:34.565 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:09:34.566 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:09:34.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:09:34 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:09:33.219518+0000 mgr.smithi037.eofnaq (mgr.14162) 293 : cluster [DBG] pgmap v279: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:09:36.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:09:36 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:09:35.219788+0000 mgr.smithi037.eofnaq (mgr.14162) 294 : cluster [DBG] pgmap v280: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:09:36.517 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:36.515+0000 7f50bf2cc700 1 -- 172.21.15.178:0/3143725959 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f50b80f6510 msgr2=0x7f50b80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:09:36.517 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:36.515+0000 7f50bf2cc700 1 --2- 172.21.15.178:0/3143725959 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f50b80f6510 0x7f50b80f6930 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f50b4009fb0 tx=0x7f50b4009b30).stop 2024-02-11T17:09:36.518 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:36.515+0000 7f50bf2cc700 1 -- 172.21.15.178:0/3143725959 shutdown_connections 2024-02-11T17:09:36.518 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:36.515+0000 7f50bf2cc700 1 --2- 172.21.15.178:0/3143725959 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f50b80f6510 0x7f50b80f6930 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:09:36.518 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:36.515+0000 7f50bf2cc700 1 -- 172.21.15.178:0/3143725959 >> 172.21.15.178:0/3143725959 conn(0x7f50b80f1c70 msgr2=0x7f50b80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:09:36.518 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:36.515+0000 7f50bf2cc700 1 -- 172.21.15.178:0/3143725959 shutdown_connections 2024-02-11T17:09:36.518 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:36.515+0000 7f50bf2cc700 1 -- 172.21.15.178:0/3143725959 wait complete. 2024-02-11T17:09:36.518 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:36.515+0000 7f50bf2cc700 1 Processor -- start 2024-02-11T17:09:36.518 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:36.515+0000 7f50bf2cc700 1 -- start start 2024-02-11T17:09:36.518 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:36.516+0000 7f50bf2cc700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f50b80f6510 0x7f50b8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:09:36.518 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:36.516+0000 7f50bf2cc700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b8106c20 con 0x7f50b80f6510 2024-02-11T17:09:36.518 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:36.516+0000 7f50bd068700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f50b80f6510 0x7f50b8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:09:36.518 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:36.516+0000 7f50bd068700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f50b80f6510 0x7f50b8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:46670/0 (socket says 172.21.15.178:46670) 2024-02-11T17:09:36.518 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:36.516+0000 7f50bd068700 1 -- 172.21.15.178:0/259122964 learned_addr learned my addr 172.21.15.178:0/259122964 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:09:36.518 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:36.516+0000 7f50bd068700 1 -- 172.21.15.178:0/259122964 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50b4005040 con 0x7f50b80f6510 2024-02-11T17:09:36.518 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:36.516+0000 7f50bd068700 1 --2- 172.21.15.178:0/259122964 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f50b80f6510 0x7f50b8106730 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f50b4013510 tx=0x7f50b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:09:36.519 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:36.516+0000 7f50adffb700 1 -- 172.21.15.178:0/259122964 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f50b4009320 con 0x7f50b80f6510 2024-02-11T17:09:36.519 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:36.517+0000 7f50bf2cc700 1 -- 172.21.15.178:0/259122964 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50b8106e20 con 0x7f50b80f6510 2024-02-11T17:09:36.519 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:36.517+0000 7f50adffb700 1 -- 172.21.15.178:0/259122964 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f50b4013f00 con 0x7f50b80f6510 2024-02-11T17:09:36.519 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:36.517+0000 7f50adffb700 1 -- 172.21.15.178:0/259122964 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f50b4014d90 con 0x7f50b80f6510 2024-02-11T17:09:36.519 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:36.517+0000 7f50bf2cc700 1 -- 172.21.15.178:0/259122964 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50b81072b0 con 0x7f50b80f6510 2024-02-11T17:09:36.520 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:36.518+0000 7f50adffb700 1 -- 172.21.15.178:0/259122964 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f50b4014420 con 0x7f50b80f6510 2024-02-11T17:09:36.520 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:36.518+0000 7f50bf2cc700 1 -- 172.21.15.178:0/259122964 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50b8045ad0 con 0x7f50b80f6510 2024-02-11T17:09:36.520 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:36.518+0000 7f50adffb700 1 --2- 172.21.15.178:0/259122964 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f50a402ff60 0x7f50a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:09:36.520 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:36.518+0000 7f50adffb700 1 -- 172.21.15.178:0/259122964 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f50b4041050 con 0x7f50b80f6510 2024-02-11T17:09:36.520 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:36.518+0000 7f50bc867700 1 --2- 172.21.15.178:0/259122964 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f50a402ff60 0x7f50a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:09:36.520 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:36.518+0000 7f50bc867700 1 --2- 172.21.15.178:0/259122964 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f50a402ff60 0x7f50a4032420 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f50a80096a0 tx=0x7f50a8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:09:36.523 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:36.522+0000 7f50adffb700 1 -- 172.21.15.178:0/259122964 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f50b40095f0 con 0x7f50b80f6510 2024-02-11T17:09:36.729 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:36.728+0000 7f50bf2cc700 1 -- 172.21.15.178:0/259122964 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f50b80fbf60 con 0x7f50b80f6510 2024-02-11T17:09:36.730 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:36.729+0000 7f50adffb700 1 -- 172.21.15.178:0/259122964 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f50b401db70 con 0x7f50b80f6510 2024-02-11T17:09:36.730 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:09:36.730 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:09:36.731 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:36.731+0000 7f50bf2cc700 1 -- 172.21.15.178:0/259122964 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f50a402ff60 msgr2=0x7f50a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:09:36.732 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:36.731+0000 7f50bf2cc700 1 --2- 172.21.15.178:0/259122964 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f50a402ff60 0x7f50a4032420 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f50a80096a0 tx=0x7f50a8006b40).stop 2024-02-11T17:09:36.732 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:36.731+0000 7f50bf2cc700 1 -- 172.21.15.178:0/259122964 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f50b80f6510 msgr2=0x7f50b8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:09:36.732 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:36.731+0000 7f50bf2cc700 1 --2- 172.21.15.178:0/259122964 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f50b80f6510 0x7f50b8106730 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f50b4013510 tx=0x7f50b400b040).stop 2024-02-11T17:09:36.732 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:36.731+0000 7f50bf2cc700 1 -- 172.21.15.178:0/259122964 shutdown_connections 2024-02-11T17:09:36.732 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:36.731+0000 7f50bf2cc700 1 --2- 172.21.15.178:0/259122964 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f50a402ff60 0x7f50a4032420 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:09:36.732 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:36.731+0000 7f50bf2cc700 1 --2- 172.21.15.178:0/259122964 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f50b80f6510 0x7f50b8106730 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:09:36.732 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:36.731+0000 7f50bf2cc700 1 -- 172.21.15.178:0/259122964 >> 172.21.15.178:0/259122964 conn(0x7f50b80f1c70 msgr2=0x7f50b80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:09:36.732 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:36.731+0000 7f50bf2cc700 1 -- 172.21.15.178:0/259122964 shutdown_connections 2024-02-11T17:09:36.732 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:36.731+0000 7f50bf2cc700 1 -- 172.21.15.178:0/259122964 wait complete. 2024-02-11T17:09:36.732 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:09:37.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:09:37 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:09:36.729556+0000 mon.smithi037 (mon.0) 285 : audit [DBG] from='client.? 172.21.15.178:0/259122964' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:09:38.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:09:38 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:09:37.220210+0000 mgr.smithi037.eofnaq (mgr.14162) 295 : cluster [DBG] pgmap v281: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:09:38.475 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:09:38.476 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:09:40.408 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:40.380+0000 7f9ea5cc1700 1 -- 172.21.15.178:0/1571809245 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9ea0060160 msgr2=0x7f9ea00fb3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:09:40.408 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:40.380+0000 7f9ea5cc1700 1 --2- 172.21.15.178:0/1571809245 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9ea0060160 0x7f9ea00fb3a0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f9e94009fb0 tx=0x7f9e94009b30).stop 2024-02-11T17:09:40.408 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:40.380+0000 7f9ea5cc1700 1 -- 172.21.15.178:0/1571809245 shutdown_connections 2024-02-11T17:09:40.408 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:40.380+0000 7f9ea5cc1700 1 --2- 172.21.15.178:0/1571809245 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9ea0060160 0x7f9ea00fb3a0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:09:40.408 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:40.380+0000 7f9ea5cc1700 1 -- 172.21.15.178:0/1571809245 >> 172.21.15.178:0/1571809245 conn(0x7f9ea00f09a0 msgr2=0x7f9ea00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:09:40.408 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:40.380+0000 7f9ea5cc1700 1 -- 172.21.15.178:0/1571809245 shutdown_connections 2024-02-11T17:09:40.409 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:40.380+0000 7f9ea5cc1700 1 -- 172.21.15.178:0/1571809245 wait complete. 2024-02-11T17:09:40.409 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:40.380+0000 7f9ea5cc1700 1 Processor -- start 2024-02-11T17:09:40.409 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:40.381+0000 7f9ea5cc1700 1 -- start start 2024-02-11T17:09:40.409 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:40.381+0000 7f9ea5cc1700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9ea0060160 0x7f9ea01086f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:09:40.409 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:40.381+0000 7f9ea5cc1700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea0108be0 con 0x7f9ea0060160 2024-02-11T17:09:40.409 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:40.381+0000 7f9ea4cbf700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9ea0060160 0x7f9ea01086f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:09:40.409 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:40.381+0000 7f9ea4cbf700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9ea0060160 0x7f9ea01086f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:33360/0 (socket says 172.21.15.178:33360) 2024-02-11T17:09:40.409 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:40.381+0000 7f9ea4cbf700 1 -- 172.21.15.178:0/3716308400 learned_addr learned my addr 172.21.15.178:0/3716308400 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:09:40.409 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:40.381+0000 7f9ea4cbf700 1 -- 172.21.15.178:0/3716308400 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e94005040 con 0x7f9ea0060160 2024-02-11T17:09:40.409 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:40.382+0000 7f9ea4cbf700 1 --2- 172.21.15.178:0/3716308400 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9ea0060160 0x7f9ea01086f0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f9e94013510 tx=0x7f9e9400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:09:40.409 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:40.382+0000 7f9e9d7fa700 1 -- 172.21.15.178:0/3716308400 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e94009320 con 0x7f9ea0060160 2024-02-11T17:09:40.409 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:40.382+0000 7f9ea5cc1700 1 -- 172.21.15.178:0/3716308400 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ea0108de0 con 0x7f9ea0060160 2024-02-11T17:09:40.409 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:40.382+0000 7f9e9d7fa700 1 -- 172.21.15.178:0/3716308400 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9e94013f00 con 0x7f9ea0060160 2024-02-11T17:09:40.409 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:40.382+0000 7f9e9d7fa700 1 -- 172.21.15.178:0/3716308400 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e94014d90 con 0x7f9ea0060160 2024-02-11T17:09:40.410 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:40.382+0000 7f9ea5cc1700 1 -- 172.21.15.178:0/3716308400 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ea0109270 con 0x7f9ea0060160 2024-02-11T17:09:40.410 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:40.383+0000 7f9e9d7fa700 1 -- 172.21.15.178:0/3716308400 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9e940095f0 con 0x7f9ea0060160 2024-02-11T17:09:40.410 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:40.383+0000 7f9ea5cc1700 1 -- 172.21.15.178:0/3716308400 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ea0045ad0 con 0x7f9ea0060160 2024-02-11T17:09:40.410 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:40.383+0000 7f9e9d7fa700 1 --2- 172.21.15.178:0/3716308400 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f9e9002ff60 0x7f9e90032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:09:40.410 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:40.383+0000 7f9e9d7fa700 1 -- 172.21.15.178:0/3716308400 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9e94040fe0 con 0x7f9ea0060160 2024-02-11T17:09:40.410 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:40.383+0000 7f9e9ffff700 1 --2- 172.21.15.178:0/3716308400 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f9e9002ff60 0x7f9e90032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:09:40.410 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:40.383+0000 7f9e9ffff700 1 --2- 172.21.15.178:0/3716308400 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f9e9002ff60 0x7f9e90032420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f9e8c0096a0 tx=0x7f9e8c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:09:40.410 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:40.387+0000 7f9e9d7fa700 1 -- 172.21.15.178:0/3716308400 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9e9401d7f0 con 0x7f9ea0060160 2024-02-11T17:09:40.593 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:40.592+0000 7f9ea5cc1700 1 -- 172.21.15.178:0/3716308400 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9ea010a120 con 0x7f9ea0060160 2024-02-11T17:09:40.594 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:40.592+0000 7f9e9d7fa700 1 -- 172.21.15.178:0/3716308400 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9e94010230 con 0x7f9ea0060160 2024-02-11T17:09:40.594 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:09:40.594 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:09:40.596 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:40.594+0000 7f9ea5cc1700 1 -- 172.21.15.178:0/3716308400 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f9e9002ff60 msgr2=0x7f9e90032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:09:40.596 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:40.594+0000 7f9ea5cc1700 1 --2- 172.21.15.178:0/3716308400 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f9e9002ff60 0x7f9e90032420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f9e8c0096a0 tx=0x7f9e8c006b40).stop 2024-02-11T17:09:40.596 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:40.594+0000 7f9ea5cc1700 1 -- 172.21.15.178:0/3716308400 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9ea0060160 msgr2=0x7f9ea01086f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:09:40.596 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:40.594+0000 7f9ea5cc1700 1 --2- 172.21.15.178:0/3716308400 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9ea0060160 0x7f9ea01086f0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f9e94013510 tx=0x7f9e9400b040).stop 2024-02-11T17:09:40.596 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:40.594+0000 7f9ea5cc1700 1 -- 172.21.15.178:0/3716308400 shutdown_connections 2024-02-11T17:09:40.596 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:40.594+0000 7f9ea5cc1700 1 --2- 172.21.15.178:0/3716308400 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f9e9002ff60 0x7f9e90032420 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:09:40.596 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:40.594+0000 7f9ea5cc1700 1 --2- 172.21.15.178:0/3716308400 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9ea0060160 0x7f9ea01086f0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:09:40.596 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:40.594+0000 7f9ea5cc1700 1 -- 172.21.15.178:0/3716308400 >> 172.21.15.178:0/3716308400 conn(0x7f9ea00f09a0 msgr2=0x7f9ea00f14a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:09:40.596 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:40.594+0000 7f9ea5cc1700 1 -- 172.21.15.178:0/3716308400 shutdown_connections 2024-02-11T17:09:40.596 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:40.594+0000 7f9ea5cc1700 1 -- 172.21.15.178:0/3716308400 wait complete. 2024-02-11T17:09:40.597 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:09:40.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:09:40 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:09:39.220622+0000 mgr.smithi037.eofnaq (mgr.14162) 296 : cluster [DBG] pgmap v282: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:09:41.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:09:41 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:09:40.593698+0000 mon.smithi037 (mon.0) 286 : audit [DBG] from='client.? 172.21.15.178:0/3716308400' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:09:42.393 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:09:42.394 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:09:42.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:09:42 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:09:41.220951+0000 mgr.smithi037.eofnaq (mgr.14162) 297 : cluster [DBG] pgmap v283: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:09:44.470 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:44.428+0000 7f3438e51700 1 -- 172.21.15.178:0/1890133860 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f34340f62a0 msgr2=0x7f34340f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:09:44.471 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:44.428+0000 7f3438e51700 1 --2- 172.21.15.178:0/1890133860 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f34340f62a0 0x7f34340f66c0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7f3424009fb0 tx=0x7f3424009b30).stop 2024-02-11T17:09:44.471 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:44.428+0000 7f3438e51700 1 -- 172.21.15.178:0/1890133860 shutdown_connections 2024-02-11T17:09:44.471 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:44.428+0000 7f3438e51700 1 --2- 172.21.15.178:0/1890133860 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f34340f62a0 0x7f34340f66c0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:09:44.471 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:44.428+0000 7f3438e51700 1 -- 172.21.15.178:0/1890133860 >> 172.21.15.178:0/1890133860 conn(0x7f34340f1ac0 msgr2=0x7f34340f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:09:44.471 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:44.428+0000 7f3438e51700 1 -- 172.21.15.178:0/1890133860 shutdown_connections 2024-02-11T17:09:44.471 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:44.428+0000 7f3438e51700 1 -- 172.21.15.178:0/1890133860 wait complete. 2024-02-11T17:09:44.471 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:44.428+0000 7f3438e51700 1 Processor -- start 2024-02-11T17:09:44.471 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:44.429+0000 7f3438e51700 1 -- start start 2024-02-11T17:09:44.471 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:44.429+0000 7f3438e51700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f34340f62a0 0x7f3434106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:09:44.471 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:44.429+0000 7f3438e51700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34341069f0 con 0x7f34340f62a0 2024-02-11T17:09:44.471 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:44.429+0000 7f34337fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f34340f62a0 0x7f3434106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:09:44.471 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:44.429+0000 7f34337fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f34340f62a0 0x7f3434106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:33384/0 (socket says 172.21.15.178:33384) 2024-02-11T17:09:44.472 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:44.429+0000 7f34337fe700 1 -- 172.21.15.178:0/4172479119 learned_addr learned my addr 172.21.15.178:0/4172479119 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:09:44.472 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:44.429+0000 7f34337fe700 1 -- 172.21.15.178:0/4172479119 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3424005040 con 0x7f34340f62a0 2024-02-11T17:09:44.472 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:44.429+0000 7f34337fe700 1 --2- 172.21.15.178:0/4172479119 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f34340f62a0 0x7f3434106500 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f3424013510 tx=0x7f342400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:09:44.472 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:44.429+0000 7f341bfff700 1 -- 172.21.15.178:0/4172479119 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3424009320 con 0x7f34340f62a0 2024-02-11T17:09:44.472 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:44.429+0000 7f341bfff700 1 -- 172.21.15.178:0/4172479119 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3424013f00 con 0x7f34340f62a0 2024-02-11T17:09:44.472 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:44.429+0000 7f3438e51700 1 -- 172.21.15.178:0/4172479119 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3434106bf0 con 0x7f34340f62a0 2024-02-11T17:09:44.472 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:44.430+0000 7f341bfff700 1 -- 172.21.15.178:0/4172479119 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3424014dc0 con 0x7f34340f62a0 2024-02-11T17:09:44.472 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:44.430+0000 7f3438e51700 1 -- 172.21.15.178:0/4172479119 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3434107080 con 0x7f34340f62a0 2024-02-11T17:09:44.472 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:44.430+0000 7f341bfff700 1 -- 172.21.15.178:0/4172479119 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f342401d3d0 con 0x7f34340f62a0 2024-02-11T17:09:44.472 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:44.430+0000 7f341bfff700 1 --2- 172.21.15.178:0/4172479119 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f341c02ff60 0x7f341c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:09:44.472 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:44.430+0000 7f341bfff700 1 -- 172.21.15.178:0/4172479119 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3424041020 con 0x7f34340f62a0 2024-02-11T17:09:44.472 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:44.430+0000 7f3438e51700 1 -- 172.21.15.178:0/4172479119 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3434045ad0 con 0x7f34340f62a0 2024-02-11T17:09:44.472 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:44.431+0000 7f3432ffd700 1 --2- 172.21.15.178:0/4172479119 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f341c02ff60 0x7f341c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:09:44.472 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:44.431+0000 7f3432ffd700 1 --2- 172.21.15.178:0/4172479119 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f341c02ff60 0x7f341c032420 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f34280096a0 tx=0x7f3428006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:09:44.473 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:44.435+0000 7f341bfff700 1 -- 172.21.15.178:0/4172479119 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3424012900 con 0x7f34340f62a0 2024-02-11T17:09:44.642 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:44.640+0000 7f3438e51700 1 -- 172.21.15.178:0/4172479119 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3434107e40 con 0x7f34340f62a0 2024-02-11T17:09:44.642 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:44.641+0000 7f341bfff700 1 -- 172.21.15.178:0/4172479119 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f342402b020 con 0x7f34340f62a0 2024-02-11T17:09:44.643 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:09:44.643 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:09:44.645 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:44.642+0000 7f3438e51700 1 -- 172.21.15.178:0/4172479119 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f341c02ff60 msgr2=0x7f341c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:09:44.645 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:44.643+0000 7f3438e51700 1 --2- 172.21.15.178:0/4172479119 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f341c02ff60 0x7f341c032420 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f34280096a0 tx=0x7f3428006b40).stop 2024-02-11T17:09:44.645 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:44.643+0000 7f3438e51700 1 -- 172.21.15.178:0/4172479119 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f34340f62a0 msgr2=0x7f3434106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:09:44.645 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:44.643+0000 7f3438e51700 1 --2- 172.21.15.178:0/4172479119 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f34340f62a0 0x7f3434106500 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f3424013510 tx=0x7f342400b040).stop 2024-02-11T17:09:44.645 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:44.643+0000 7f3438e51700 1 -- 172.21.15.178:0/4172479119 shutdown_connections 2024-02-11T17:09:44.645 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:44.643+0000 7f3438e51700 1 --2- 172.21.15.178:0/4172479119 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f341c02ff60 0x7f341c032420 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:09:44.645 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:44.643+0000 7f3438e51700 1 --2- 172.21.15.178:0/4172479119 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f34340f62a0 0x7f3434106500 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:09:44.645 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:44.643+0000 7f3438e51700 1 -- 172.21.15.178:0/4172479119 >> 172.21.15.178:0/4172479119 conn(0x7f34340f1ac0 msgr2=0x7f34340f2530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:09:44.645 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:44.643+0000 7f3438e51700 1 -- 172.21.15.178:0/4172479119 shutdown_connections 2024-02-11T17:09:44.646 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:44.643+0000 7f3438e51700 1 -- 172.21.15.178:0/4172479119 wait complete. 2024-02-11T17:09:44.646 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:09:44.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:09:44 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:09:43.221307+0000 mgr.smithi037.eofnaq (mgr.14162) 298 : cluster [DBG] pgmap v284: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:09:45.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:09:45 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:09:44.642224+0000 mon.smithi037 (mon.0) 287 : audit [DBG] from='client.? 172.21.15.178:0/4172479119' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:09:46.476 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:09:46.476 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:09:46.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:09:46 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:09:45.221694+0000 mgr.smithi037.eofnaq (mgr.14162) 299 : cluster [DBG] pgmap v285: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:09:48.330 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:48.326+0000 7fba5f597700 1 -- 172.21.15.178:0/3258827731 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba580f5410 msgr2=0x7fba580f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:09:48.330 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:48.326+0000 7fba5f597700 1 --2- 172.21.15.178:0/3258827731 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba580f5410 0x7fba580f5830 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7fba48009fb0 tx=0x7fba48009b30).stop 2024-02-11T17:09:48.330 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:48.326+0000 7fba5f597700 1 -- 172.21.15.178:0/3258827731 shutdown_connections 2024-02-11T17:09:48.330 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:48.326+0000 7fba5f597700 1 --2- 172.21.15.178:0/3258827731 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba580f5410 0x7fba580f5830 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:09:48.330 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:48.326+0000 7fba5f597700 1 -- 172.21.15.178:0/3258827731 >> 172.21.15.178:0/3258827731 conn(0x7fba580f0b70 msgr2=0x7fba580f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:09:48.330 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:48.326+0000 7fba5f597700 1 -- 172.21.15.178:0/3258827731 shutdown_connections 2024-02-11T17:09:48.330 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:48.326+0000 7fba5f597700 1 -- 172.21.15.178:0/3258827731 wait complete. 2024-02-11T17:09:48.331 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:48.327+0000 7fba5f597700 1 Processor -- start 2024-02-11T17:09:48.331 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:48.327+0000 7fba5f597700 1 -- start start 2024-02-11T17:09:48.331 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:48.327+0000 7fba5f597700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba580f5410 0x7fba58105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:09:48.331 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:48.327+0000 7fba5f597700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba58105af0 con 0x7fba580f5410 2024-02-11T17:09:48.331 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:48.327+0000 7fba5d333700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba580f5410 0x7fba58105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:09:48.331 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:48.327+0000 7fba5d333700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba580f5410 0x7fba58105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:33392/0 (socket says 172.21.15.178:33392) 2024-02-11T17:09:48.331 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:48.327+0000 7fba5d333700 1 -- 172.21.15.178:0/845138133 learned_addr learned my addr 172.21.15.178:0/845138133 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:09:48.331 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:48.327+0000 7fba5d333700 1 -- 172.21.15.178:0/845138133 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba48005040 con 0x7fba580f5410 2024-02-11T17:09:48.331 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:48.328+0000 7fba5d333700 1 --2- 172.21.15.178:0/845138133 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba580f5410 0x7fba58105600 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7fba48013510 tx=0x7fba4800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:09:48.331 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:48.328+0000 7fba4dffb700 1 -- 172.21.15.178:0/845138133 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba48009320 con 0x7fba580f5410 2024-02-11T17:09:48.331 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:48.328+0000 7fba5f597700 1 -- 172.21.15.178:0/845138133 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba58105cf0 con 0x7fba580f5410 2024-02-11T17:09:48.331 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:48.328+0000 7fba4dffb700 1 -- 172.21.15.178:0/845138133 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fba48013f00 con 0x7fba580f5410 2024-02-11T17:09:48.331 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:48.328+0000 7fba4dffb700 1 -- 172.21.15.178:0/845138133 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba48014d90 con 0x7fba580f5410 2024-02-11T17:09:48.331 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:48.328+0000 7fba5f597700 1 -- 172.21.15.178:0/845138133 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba58106180 con 0x7fba580f5410 2024-02-11T17:09:48.332 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:48.329+0000 7fba4dffb700 1 -- 172.21.15.178:0/845138133 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fba480263a0 con 0x7fba580f5410 2024-02-11T17:09:48.332 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:48.329+0000 7fba5f597700 1 -- 172.21.15.178:0/845138133 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba58045ad0 con 0x7fba580f5410 2024-02-11T17:09:48.332 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:48.329+0000 7fba4dffb700 1 --2- 172.21.15.178:0/845138133 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fba4402ff60 0x7fba44032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:09:48.333 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:48.329+0000 7fba4dffb700 1 -- 172.21.15.178:0/845138133 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fba480412e0 con 0x7fba580f5410 2024-02-11T17:09:48.333 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:48.329+0000 7fba5cb32700 1 --2- 172.21.15.178:0/845138133 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fba4402ff60 0x7fba44032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:09:48.333 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:48.330+0000 7fba5cb32700 1 --2- 172.21.15.178:0/845138133 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fba4402ff60 0x7fba44032420 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fba540096a0 tx=0x7fba54006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:09:48.335 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:48.334+0000 7fba4dffb700 1 -- 172.21.15.178:0/845138133 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fba480126f0 con 0x7fba580f5410 2024-02-11T17:09:48.539 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:48.537+0000 7fba5f597700 1 -- 172.21.15.178:0/845138133 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fba580fae60 con 0x7fba580f5410 2024-02-11T17:09:48.540 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:48.538+0000 7fba4dffb700 1 -- 172.21.15.178:0/845138133 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fba4800f340 con 0x7fba580f5410 2024-02-11T17:09:48.540 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:09:48.540 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:09:48.542 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:48.540+0000 7fba5f597700 1 -- 172.21.15.178:0/845138133 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fba4402ff60 msgr2=0x7fba44032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:09:48.542 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:48.540+0000 7fba5f597700 1 --2- 172.21.15.178:0/845138133 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fba4402ff60 0x7fba44032420 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fba540096a0 tx=0x7fba54006b40).stop 2024-02-11T17:09:48.542 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:48.540+0000 7fba5f597700 1 -- 172.21.15.178:0/845138133 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba580f5410 msgr2=0x7fba58105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:09:48.542 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:48.540+0000 7fba5f597700 1 --2- 172.21.15.178:0/845138133 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba580f5410 0x7fba58105600 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7fba48013510 tx=0x7fba4800b040).stop 2024-02-11T17:09:48.542 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:48.540+0000 7fba5f597700 1 -- 172.21.15.178:0/845138133 shutdown_connections 2024-02-11T17:09:48.542 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:48.540+0000 7fba5f597700 1 --2- 172.21.15.178:0/845138133 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fba4402ff60 0x7fba44032420 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:09:48.542 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:48.540+0000 7fba5f597700 1 --2- 172.21.15.178:0/845138133 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba580f5410 0x7fba58105600 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:09:48.542 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:48.540+0000 7fba5f597700 1 -- 172.21.15.178:0/845138133 >> 172.21.15.178:0/845138133 conn(0x7fba580f0b70 msgr2=0x7fba580f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:09:48.543 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:48.540+0000 7fba5f597700 1 -- 172.21.15.178:0/845138133 shutdown_connections 2024-02-11T17:09:48.543 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:48.540+0000 7fba5f597700 1 -- 172.21.15.178:0/845138133 wait complete. 2024-02-11T17:09:48.543 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:09:48.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:09:48 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:09:47.221916+0000 mgr.smithi037.eofnaq (mgr.14162) 300 : cluster [DBG] pgmap v286: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:09:49.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:09:49 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:09:48.539522+0000 mon.smithi037 (mon.0) 288 : audit [DBG] from='client.? 172.21.15.178:0/845138133' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:09:50.259 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:09:50.259 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:09:50.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:09:50 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:09:49.222258+0000 mgr.smithi037.eofnaq (mgr.14162) 301 : cluster [DBG] pgmap v287: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:09:52.186 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:52.182+0000 7fecd76c0700 1 -- 172.21.15.178:0/3939321864 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fecd00f5410 msgr2=0x7fecd00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:09:52.186 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:52.182+0000 7fecd76c0700 1 --2- 172.21.15.178:0/3939321864 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fecd00f5410 0x7fecd00f5830 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7fecc0009fb0 tx=0x7fecc0009b30).stop 2024-02-11T17:09:52.186 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:52.182+0000 7fecd76c0700 1 -- 172.21.15.178:0/3939321864 shutdown_connections 2024-02-11T17:09:52.186 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:52.182+0000 7fecd76c0700 1 --2- 172.21.15.178:0/3939321864 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fecd00f5410 0x7fecd00f5830 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:09:52.186 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:52.182+0000 7fecd76c0700 1 -- 172.21.15.178:0/3939321864 >> 172.21.15.178:0/3939321864 conn(0x7fecd00f0b70 msgr2=0x7fecd00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:09:52.186 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:52.182+0000 7fecd76c0700 1 -- 172.21.15.178:0/3939321864 shutdown_connections 2024-02-11T17:09:52.186 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:52.182+0000 7fecd76c0700 1 -- 172.21.15.178:0/3939321864 wait complete. 2024-02-11T17:09:52.186 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:52.183+0000 7fecd76c0700 1 Processor -- start 2024-02-11T17:09:52.186 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:52.183+0000 7fecd76c0700 1 -- start start 2024-02-11T17:09:52.186 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:52.183+0000 7fecd76c0700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fecd00f5410 0x7fecd0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:09:52.187 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:52.183+0000 7fecd76c0700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecd0106c20 con 0x7fecd00f5410 2024-02-11T17:09:52.187 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:52.183+0000 7fecd545c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fecd00f5410 0x7fecd0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:09:52.187 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:52.183+0000 7fecd545c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fecd00f5410 0x7fecd0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:39174/0 (socket says 172.21.15.178:39174) 2024-02-11T17:09:52.187 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:52.183+0000 7fecd545c700 1 -- 172.21.15.178:0/4221953545 learned_addr learned my addr 172.21.15.178:0/4221953545 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:09:52.187 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:52.183+0000 7fecd545c700 1 -- 172.21.15.178:0/4221953545 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecc0005040 con 0x7fecd00f5410 2024-02-11T17:09:52.187 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:52.184+0000 7fecd545c700 1 --2- 172.21.15.178:0/4221953545 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fecd00f5410 0x7fecd0106730 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7fecc0014000 tx=0x7fecc000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:09:52.187 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:52.184+0000 7fecc5ffb700 1 -- 172.21.15.178:0/4221953545 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fecc00095f0 con 0x7fecd00f5410 2024-02-11T17:09:52.187 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:52.184+0000 7fecd76c0700 1 -- 172.21.15.178:0/4221953545 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecd0106e20 con 0x7fecd00f5410 2024-02-11T17:09:52.187 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:52.184+0000 7fecc5ffb700 1 -- 172.21.15.178:0/4221953545 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fecc0007e70 con 0x7fecd00f5410 2024-02-11T17:09:52.187 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:52.184+0000 7fecc5ffb700 1 -- 172.21.15.178:0/4221953545 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fecc0014ea0 con 0x7fecd00f5410 2024-02-11T17:09:52.187 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:52.184+0000 7fecd76c0700 1 -- 172.21.15.178:0/4221953545 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecd01072b0 con 0x7fecd00f5410 2024-02-11T17:09:52.188 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:52.185+0000 7fecc5ffb700 1 -- 172.21.15.178:0/4221953545 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fecc0025070 con 0x7fecd00f5410 2024-02-11T17:09:52.188 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:52.185+0000 7fecd76c0700 1 -- 172.21.15.178:0/4221953545 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecd0045ad0 con 0x7fecd00f5410 2024-02-11T17:09:52.188 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:52.185+0000 7fecc5ffb700 1 --2- 172.21.15.178:0/4221953545 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fecbc02ff60 0x7fecbc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:09:52.188 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:52.185+0000 7fecc5ffb700 1 -- 172.21.15.178:0/4221953545 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fecc0040a30 con 0x7fecd00f5410 2024-02-11T17:09:52.188 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:52.185+0000 7fecd4c5b700 1 --2- 172.21.15.178:0/4221953545 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fecbc02ff60 0x7fecbc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:09:52.188 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:52.186+0000 7fecd4c5b700 1 --2- 172.21.15.178:0/4221953545 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fecbc02ff60 0x7fecbc032420 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7feccc0096a0 tx=0x7feccc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:09:52.191 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:52.190+0000 7fecc5ffb700 1 -- 172.21.15.178:0/4221953545 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fecc00079c0 con 0x7fecd00f5410 2024-02-11T17:09:52.395 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:52.394+0000 7fecd76c0700 1 -- 172.21.15.178:0/4221953545 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fecd00fae60 con 0x7fecd00f5410 2024-02-11T17:09:52.396 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:52.395+0000 7fecc5ffb700 1 -- 172.21.15.178:0/4221953545 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fecc000f300 con 0x7fecd00f5410 2024-02-11T17:09:52.396 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:09:52.397 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:09:52.398 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:52.396+0000 7fecd76c0700 1 -- 172.21.15.178:0/4221953545 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fecbc02ff60 msgr2=0x7fecbc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:09:52.398 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:52.396+0000 7fecd76c0700 1 --2- 172.21.15.178:0/4221953545 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fecbc02ff60 0x7fecbc032420 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7feccc0096a0 tx=0x7feccc006b40).stop 2024-02-11T17:09:52.398 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:52.396+0000 7fecd76c0700 1 -- 172.21.15.178:0/4221953545 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fecd00f5410 msgr2=0x7fecd0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:09:52.398 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:52.396+0000 7fecd76c0700 1 --2- 172.21.15.178:0/4221953545 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fecd00f5410 0x7fecd0106730 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7fecc0014000 tx=0x7fecc000b040).stop 2024-02-11T17:09:52.398 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:52.396+0000 7fecd76c0700 1 -- 172.21.15.178:0/4221953545 shutdown_connections 2024-02-11T17:09:52.398 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:52.396+0000 7fecd76c0700 1 --2- 172.21.15.178:0/4221953545 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fecbc02ff60 0x7fecbc032420 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:09:52.398 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:52.396+0000 7fecd76c0700 1 --2- 172.21.15.178:0/4221953545 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fecd00f5410 0x7fecd0106730 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:09:52.398 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:52.396+0000 7fecd76c0700 1 -- 172.21.15.178:0/4221953545 >> 172.21.15.178:0/4221953545 conn(0x7fecd00f0b70 msgr2=0x7fecd00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:09:52.398 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:52.396+0000 7fecd76c0700 1 -- 172.21.15.178:0/4221953545 shutdown_connections 2024-02-11T17:09:52.399 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:52.396+0000 7fecd76c0700 1 -- 172.21.15.178:0/4221953545 wait complete. 2024-02-11T17:09:52.399 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:09:52.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:09:52 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:09:51.222610+0000 mgr.smithi037.eofnaq (mgr.14162) 302 : cluster [DBG] pgmap v288: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:09:53.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:09:53 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:09:52.395999+0000 mon.smithi037 (mon.0) 289 : audit [DBG] from='client.? 172.21.15.178:0/4221953545' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:09:54.121 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:09:54.121 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:09:54.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:09:54 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:09:53.222936+0000 mgr.smithi037.eofnaq (mgr.14162) 303 : cluster [DBG] pgmap v289: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:09:56.017 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:56.015+0000 7fb4abfff700 1 -- 172.21.15.178:0/2698159520 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4ac0f51a0 msgr2=0x7fb4ac0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:09:56.018 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:56.015+0000 7fb4abfff700 1 --2- 172.21.15.178:0/2698159520 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4ac0f51a0 0x7fb4ac0f55c0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7fb49c009fb0 tx=0x7fb49c009b30).stop 2024-02-11T17:09:56.018 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:56.015+0000 7fb4abfff700 1 -- 172.21.15.178:0/2698159520 shutdown_connections 2024-02-11T17:09:56.018 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:56.015+0000 7fb4abfff700 1 --2- 172.21.15.178:0/2698159520 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4ac0f51a0 0x7fb4ac0f55c0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:09:56.018 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:56.015+0000 7fb4abfff700 1 -- 172.21.15.178:0/2698159520 >> 172.21.15.178:0/2698159520 conn(0x7fb4ac0f09a0 msgr2=0x7fb4ac0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:09:56.018 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:56.015+0000 7fb4abfff700 1 -- 172.21.15.178:0/2698159520 shutdown_connections 2024-02-11T17:09:56.018 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:56.015+0000 7fb4abfff700 1 -- 172.21.15.178:0/2698159520 wait complete. 2024-02-11T17:09:56.019 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:56.016+0000 7fb4abfff700 1 Processor -- start 2024-02-11T17:09:56.019 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:56.016+0000 7fb4abfff700 1 -- start start 2024-02-11T17:09:56.019 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:56.017+0000 7fb4abfff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4ac060240 0x7fb4ac109300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:09:56.020 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:56.017+0000 7fb4abfff700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4ac1097f0 con 0x7fb4ac060240 2024-02-11T17:09:56.020 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:56.017+0000 7fb4aaffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4ac060240 0x7fb4ac109300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:09:56.020 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:56.017+0000 7fb4aaffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4ac060240 0x7fb4ac109300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:39196/0 (socket says 172.21.15.178:39196) 2024-02-11T17:09:56.020 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:56.017+0000 7fb4aaffd700 1 -- 172.21.15.178:0/515551078 learned_addr learned my addr 172.21.15.178:0/515551078 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:09:56.020 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:56.017+0000 7fb4aaffd700 1 -- 172.21.15.178:0/515551078 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb49c005040 con 0x7fb4ac060240 2024-02-11T17:09:56.020 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:56.017+0000 7fb4aaffd700 1 --2- 172.21.15.178:0/515551078 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4ac060240 0x7fb4ac109300 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7fb49c009f80 tx=0x7fb49c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:09:56.020 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:56.018+0000 7fb4937fe700 1 -- 172.21.15.178:0/515551078 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb49c0136c0 con 0x7fb4ac060240 2024-02-11T17:09:56.020 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:56.018+0000 7fb4abfff700 1 -- 172.21.15.178:0/515551078 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4ac109930 con 0x7fb4ac060240 2024-02-11T17:09:56.020 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:56.018+0000 7fb4937fe700 1 -- 172.21.15.178:0/515551078 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb49c0095f0 con 0x7fb4ac060240 2024-02-11T17:09:56.020 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:56.018+0000 7fb4937fe700 1 -- 172.21.15.178:0/515551078 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb49c014420 con 0x7fb4ac060240 2024-02-11T17:09:56.020 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:56.018+0000 7fb4abfff700 1 -- 172.21.15.178:0/515551078 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4ac109dc0 con 0x7fb4ac060240 2024-02-11T17:09:56.021 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:56.019+0000 7fb4937fe700 1 -- 172.21.15.178:0/515551078 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb49c025060 con 0x7fb4ac060240 2024-02-11T17:09:56.021 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:56.019+0000 7fb4abfff700 1 -- 172.21.15.178:0/515551078 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4ac059100 con 0x7fb4ac060240 2024-02-11T17:09:56.021 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:56.019+0000 7fb4937fe700 1 --2- 172.21.15.178:0/515551078 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb49402ff60 0x7fb494032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:09:56.021 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:56.019+0000 7fb4937fe700 1 -- 172.21.15.178:0/515551078 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb49c040d80 con 0x7fb4ac060240 2024-02-11T17:09:56.022 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:56.019+0000 7fb4aa7fc700 1 --2- 172.21.15.178:0/515551078 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb49402ff60 0x7fb494032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:09:56.022 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:56.019+0000 7fb4aa7fc700 1 --2- 172.21.15.178:0/515551078 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb49402ff60 0x7fb494032420 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fb4a00096a0 tx=0x7fb4a0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:09:56.025 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:56.024+0000 7fb4937fe700 1 -- 172.21.15.178:0/515551078 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb49c01c950 con 0x7fb4ac060240 2024-02-11T17:09:56.230 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:56.229+0000 7fb4abfff700 1 -- 172.21.15.178:0/515551078 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb4ac045ad0 con 0x7fb4ac060240 2024-02-11T17:09:56.231 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:56.229+0000 7fb4937fe700 1 -- 172.21.15.178:0/515551078 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb4ac045ad0 con 0x7fb4ac060240 2024-02-11T17:09:56.231 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:09:56.231 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:09:56.233 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:56.231+0000 7fb4abfff700 1 -- 172.21.15.178:0/515551078 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb49402ff60 msgr2=0x7fb494032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:09:56.233 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:56.231+0000 7fb4abfff700 1 --2- 172.21.15.178:0/515551078 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb49402ff60 0x7fb494032420 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fb4a00096a0 tx=0x7fb4a0006b40).stop 2024-02-11T17:09:56.233 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:56.231+0000 7fb4abfff700 1 -- 172.21.15.178:0/515551078 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4ac060240 msgr2=0x7fb4ac109300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:09:56.233 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:56.231+0000 7fb4abfff700 1 --2- 172.21.15.178:0/515551078 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4ac060240 0x7fb4ac109300 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7fb49c009f80 tx=0x7fb49c009b30).stop 2024-02-11T17:09:56.233 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:56.231+0000 7fb4abfff700 1 -- 172.21.15.178:0/515551078 shutdown_connections 2024-02-11T17:09:56.233 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:56.231+0000 7fb4abfff700 1 --2- 172.21.15.178:0/515551078 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb49402ff60 0x7fb494032420 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:09:56.233 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:56.231+0000 7fb4abfff700 1 --2- 172.21.15.178:0/515551078 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4ac060240 0x7fb4ac109300 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:09:56.233 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:56.231+0000 7fb4abfff700 1 -- 172.21.15.178:0/515551078 >> 172.21.15.178:0/515551078 conn(0x7fb4ac0f09a0 msgr2=0x7fb4ac0f1500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:09:56.234 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:56.231+0000 7fb4abfff700 1 -- 172.21.15.178:0/515551078 shutdown_connections 2024-02-11T17:09:56.234 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:56.231+0000 7fb4abfff700 1 -- 172.21.15.178:0/515551078 wait complete. 2024-02-11T17:09:56.234 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:09:56.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:09:56 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:09:55.223257+0000 mgr.smithi037.eofnaq (mgr.14162) 304 : cluster [DBG] pgmap v290: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:09:57.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:09:57 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:09:56.230661+0000 mon.smithi037 (mon.0) 290 : audit [DBG] from='client.? 172.21.15.178:0/515551078' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:09:57.890 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:09:57.890 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:09:58.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:09:58 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:09:57.223575+0000 mgr.smithi037.eofnaq (mgr.14162) 305 : cluster [DBG] pgmap v291: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:09:59.801 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:59.798+0000 7f094c18b700 1 -- 172.21.15.178:0/852395403 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f09440f5410 msgr2=0x7f09440f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:09:59.801 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:59.798+0000 7f094c18b700 1 --2- 172.21.15.178:0/852395403 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f09440f5410 0x7f09440f5830 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f0940009fb0 tx=0x7f0940009b30).stop 2024-02-11T17:09:59.801 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:59.798+0000 7f094c18b700 1 -- 172.21.15.178:0/852395403 shutdown_connections 2024-02-11T17:09:59.801 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:59.798+0000 7f094c18b700 1 --2- 172.21.15.178:0/852395403 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f09440f5410 0x7f09440f5830 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:09:59.801 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:59.798+0000 7f094c18b700 1 -- 172.21.15.178:0/852395403 >> 172.21.15.178:0/852395403 conn(0x7f09440f0b70 msgr2=0x7f09440f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:09:59.801 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:59.798+0000 7f094c18b700 1 -- 172.21.15.178:0/852395403 shutdown_connections 2024-02-11T17:09:59.802 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:59.798+0000 7f094c18b700 1 -- 172.21.15.178:0/852395403 wait complete. 2024-02-11T17:09:59.802 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:59.799+0000 7f094c18b700 1 Processor -- start 2024-02-11T17:09:59.802 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:59.799+0000 7f094c18b700 1 -- start start 2024-02-11T17:09:59.802 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:59.799+0000 7f094c18b700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f09440f5410 0x7f0944105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:09:59.802 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:59.799+0000 7f094c18b700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0944105b20 con 0x7f09440f5410 2024-02-11T17:09:59.803 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:59.800+0000 7f0949f27700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f09440f5410 0x7f0944105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:09:59.804 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:59.800+0000 7f0949f27700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f09440f5410 0x7f0944105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:34660/0 (socket says 172.21.15.178:34660) 2024-02-11T17:09:59.804 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:59.800+0000 7f0949f27700 1 -- 172.21.15.178:0/4238291306 learned_addr learned my addr 172.21.15.178:0/4238291306 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:09:59.804 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:59.800+0000 7f0949f27700 1 -- 172.21.15.178:0/4238291306 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0940005040 con 0x7f09440f5410 2024-02-11T17:09:59.804 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:59.800+0000 7f0949f27700 1 --2- 172.21.15.178:0/4238291306 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f09440f5410 0x7f0944105630 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f0940013510 tx=0x7f094000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:09:59.804 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:59.800+0000 7f093a7fc700 1 -- 172.21.15.178:0/4238291306 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0940009320 con 0x7f09440f5410 2024-02-11T17:09:59.804 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:59.800+0000 7f094c18b700 1 -- 172.21.15.178:0/4238291306 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0944105d20 con 0x7f09440f5410 2024-02-11T17:09:59.804 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:59.800+0000 7f093a7fc700 1 -- 172.21.15.178:0/4238291306 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0940013f00 con 0x7f09440f5410 2024-02-11T17:09:59.804 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:59.800+0000 7f093a7fc700 1 -- 172.21.15.178:0/4238291306 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0940014d90 con 0x7f09440f5410 2024-02-11T17:09:59.804 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:59.800+0000 7f094c18b700 1 -- 172.21.15.178:0/4238291306 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09441061b0 con 0x7f09440f5410 2024-02-11T17:09:59.804 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:59.801+0000 7f093a7fc700 1 -- 172.21.15.178:0/4238291306 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f09400095f0 con 0x7f09440f5410 2024-02-11T17:09:59.804 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:59.801+0000 7f094c18b700 1 -- 172.21.15.178:0/4238291306 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0944045ad0 con 0x7f09440f5410 2024-02-11T17:09:59.804 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:59.801+0000 7f093a7fc700 1 --2- 172.21.15.178:0/4238291306 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f093002ff60 0x7f0930032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:09:59.804 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:59.801+0000 7f093a7fc700 1 -- 172.21.15.178:0/4238291306 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0940041000 con 0x7f09440f5410 2024-02-11T17:09:59.805 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:59.801+0000 7f0949726700 1 --2- 172.21.15.178:0/4238291306 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f093002ff60 0x7f0930032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:09:59.805 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:59.802+0000 7f0949726700 1 --2- 172.21.15.178:0/4238291306 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f093002ff60 0x7f0930032420 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f09340096a0 tx=0x7f0934006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:09:59.807 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:09:59.806+0000 7f093a7fc700 1 -- 172.21.15.178:0/4238291306 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0940014420 con 0x7f09440f5410 2024-02-11T17:10:00.011 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:00.009+0000 7f094c18b700 1 -- 172.21.15.178:0/4238291306 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f09440fae60 con 0x7f09440f5410 2024-02-11T17:10:00.011 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:00.010+0000 7f093a7fc700 1 -- 172.21.15.178:0/4238291306 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0940010230 con 0x7f09440f5410 2024-02-11T17:10:00.012 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:10:00.012 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:10:00.013 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:00.011+0000 7f094c18b700 1 -- 172.21.15.178:0/4238291306 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f093002ff60 msgr2=0x7f0930032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:10:00.013 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:00.011+0000 7f094c18b700 1 --2- 172.21.15.178:0/4238291306 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f093002ff60 0x7f0930032420 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f09340096a0 tx=0x7f0934006b40).stop 2024-02-11T17:10:00.013 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:00.011+0000 7f094c18b700 1 -- 172.21.15.178:0/4238291306 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f09440f5410 msgr2=0x7f0944105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:10:00.014 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:00.011+0000 7f094c18b700 1 --2- 172.21.15.178:0/4238291306 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f09440f5410 0x7f0944105630 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f0940013510 tx=0x7f094000b040).stop 2024-02-11T17:10:00.014 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:00.011+0000 7f094c18b700 1 -- 172.21.15.178:0/4238291306 shutdown_connections 2024-02-11T17:10:00.014 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:00.011+0000 7f094c18b700 1 --2- 172.21.15.178:0/4238291306 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f093002ff60 0x7f0930032420 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:10:00.014 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:00.011+0000 7f094c18b700 1 --2- 172.21.15.178:0/4238291306 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f09440f5410 0x7f0944105630 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:10:00.014 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:00.011+0000 7f094c18b700 1 -- 172.21.15.178:0/4238291306 >> 172.21.15.178:0/4238291306 conn(0x7f09440f0b70 msgr2=0x7f09440f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:10:00.014 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:00.011+0000 7f094c18b700 1 -- 172.21.15.178:0/4238291306 shutdown_connections 2024-02-11T17:10:00.014 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:00.011+0000 7f094c18b700 1 -- 172.21.15.178:0/4238291306 wait complete. 2024-02-11T17:10:00.014 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:10:00.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:00 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:09:59.223818+0000 mgr.smithi037.eofnaq (mgr.14162) 306 : cluster [DBG] pgmap v292: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:10:00.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:00 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:10:00.011267+0000 mon.smithi037 (mon.0) 291 : audit [DBG] from='client.? 172.21.15.178:0/4238291306' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:10:01.684 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:01 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cephadm 2024-02-11T17:10:00.509679+0000 mgr.smithi037.eofnaq (mgr.14162) 307 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon prometheus.smithi037 ... 2024-02-11T17:10:01.684 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:01 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi037 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2024-02-11T17:10:01.684 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:01 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2024-02-11T17:10:01.684 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:01 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: stat: stderr time="2024-02-11T17:08:08Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-11T17:10:01.684 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:01 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: stat: stderr time="2024-02-11T17:08:45Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-11T17:10:01.684 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:01 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: stat: stderr time="2024-02-11T17:09:23Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-11T17:10:01.684 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:01 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-11T17:10:01.684 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:01 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: Traceback (most recent call last): 2024-02-11T17:10:01.684 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:01 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: File "/var/lib/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-11T17:10:01.684 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:01 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: main() 2024-02-11T17:10:01.685 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:01 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: File "/var/lib/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-11T17:10:01.685 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:01 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: r = ctx.func(ctx) 2024-02-11T17:10:01.685 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:01 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: File "/var/lib/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-11T17:10:01.685 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:01 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: return func(ctx) 2024-02-11T17:10:01.685 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:01 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: File "/var/lib/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-02-11T17:10:01.685 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:01 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-11T17:10:01.685 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:01 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: File "/var/lib/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4185, in extract_uid_gid_monitoring 2024-02-11T17:10:01.685 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:01 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2024-02-11T17:10:01.685 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:01 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: File "/var/lib/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-02-11T17:10:01.685 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:01 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: raise RuntimeError('uid/gid not found') 2024-02-11T17:10:01.685 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:01 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: RuntimeError: uid/gid not found 2024-02-11T17:10:01.685 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:01 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: Traceback (most recent call last): 2024-02-11T17:10:01.685 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:01 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-02-11T17:10:01.685 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:01 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: yield (conn, connr) 2024-02-11T17:10:01.686 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:01 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-02-11T17:10:01.686 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:01 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: code, '\n'.join(err))) 2024-02-11T17:10:01.686 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:01 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon prometheus.smithi037 ... 2024-02-11T17:10:01.686 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:01 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi037 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2024-02-11T17:10:01.686 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:01 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2024-02-11T17:10:01.686 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:01 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: stat: stderr time="2024-02-11T17:08:08Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-11T17:10:01.686 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:01 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: stat: stderr time="2024-02-11T17:08:45Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-11T17:10:01.686 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:01 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: stat: stderr time="2024-02-11T17:09:23Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-11T17:10:01.686 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:01 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-11T17:10:01.686 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:01 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: Traceback (most recent call last): 2024-02-11T17:10:01.686 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:01 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: File "/var/lib/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-11T17:10:01.686 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:01 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: main() 2024-02-11T17:10:01.686 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:01 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: File "/var/lib/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-11T17:10:01.686 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:01 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: r = ctx.func(ctx) 2024-02-11T17:10:01.686 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:01 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: File "/var/lib/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-11T17:10:01.687 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:01 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: return func(ctx) 2024-02-11T17:10:01.687 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:01 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: File "/var/lib/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-02-11T17:10:01.687 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:01 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-11T17:10:01.687 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:01 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: File "/var/lib/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4185, in extract_uid_gid_monitoring 2024-02-11T17:10:01.687 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:01 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2024-02-11T17:10:01.687 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:01 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: File "/var/lib/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-02-11T17:10:01.687 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:01 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: raise RuntimeError('uid/gid not found') 2024-02-11T17:10:01.687 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:01 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: RuntimeError: uid/gid not found 2024-02-11T17:10:01.687 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:01 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:10:00.510331+0000 mon.smithi037 (mon.0) 292 : audit [DBG] from='mgr.14162 172.21.15.37:0/2745465012' entity='mgr.smithi037.eofnaq' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-11T17:10:01.687 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:01 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:10:00.746388+0000 mon.smithi037 (mon.0) 293 : audit [INF] from='mgr.14162 172.21.15.37:0/2745465012' entity='mgr.smithi037.eofnaq' 2024-02-11T17:10:01.687 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:01 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:10:00.851601+0000 mon.smithi037 (mon.0) 294 : audit [INF] from='mgr.14162 172.21.15.37:0/2745465012' entity='mgr.smithi037.eofnaq' 2024-02-11T17:10:01.687 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:01 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:10:01.082783+0000 mon.smithi037 (mon.0) 295 : audit [INF] from='mgr.14162 172.21.15.37:0/2745465012' entity='mgr.smithi037.eofnaq' 2024-02-11T17:10:01.813 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:10:01.813 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:10:02.683 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:02 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:10:01.224054+0000 mgr.smithi037.eofnaq (mgr.14162) 308 : cluster [DBG] pgmap v293: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:10:04.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:03 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:10:02.825306+0000 mon.smithi037 (mon.0) 296 : audit [INF] from='mgr.14162 172.21.15.37:0/2745465012' entity='mgr.smithi037.eofnaq' 2024-02-11T17:10:04.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:03 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:10:03.280170+0000 mon.smithi037 (mon.0) 297 : audit [INF] from='mgr.14162 172.21.15.37:0/2745465012' entity='mgr.smithi037.eofnaq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi037.eofnaq/mirror_snapshot_schedule"}]: dispatch 2024-02-11T17:10:04.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:03 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:10:03.282639+0000 mon.smithi037 (mon.0) 298 : audit [INF] from='mgr.14162 172.21.15.37:0/2745465012' entity='mgr.smithi037.eofnaq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi037.eofnaq/trash_purge_schedule"}]: dispatch 2024-02-11T17:10:04.777 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:04.743+0000 7fb368ff9700 1 -- 172.21.15.178:0/1365897710 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3640f5410 msgr2=0x7fb3640f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:10:04.777 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:04.743+0000 7fb368ff9700 1 --2- 172.21.15.178:0/1365897710 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3640f5410 0x7fb3640f5830 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7fb34c009fb0 tx=0x7fb34c009b30).stop 2024-02-11T17:10:04.777 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:04.743+0000 7fb368ff9700 1 -- 172.21.15.178:0/1365897710 shutdown_connections 2024-02-11T17:10:04.777 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:04.743+0000 7fb368ff9700 1 --2- 172.21.15.178:0/1365897710 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3640f5410 0x7fb3640f5830 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:10:04.777 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:04.743+0000 7fb368ff9700 1 -- 172.21.15.178:0/1365897710 >> 172.21.15.178:0/1365897710 conn(0x7fb3640f0b70 msgr2=0x7fb3640f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:10:04.777 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:04.743+0000 7fb368ff9700 1 -- 172.21.15.178:0/1365897710 shutdown_connections 2024-02-11T17:10:04.777 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:04.743+0000 7fb368ff9700 1 -- 172.21.15.178:0/1365897710 wait complete. 2024-02-11T17:10:04.778 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:04.743+0000 7fb368ff9700 1 Processor -- start 2024-02-11T17:10:04.778 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:04.743+0000 7fb368ff9700 1 -- start start 2024-02-11T17:10:04.778 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:04.743+0000 7fb368ff9700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3640f5410 0x7fb364105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:10:04.778 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:04.743+0000 7fb368ff9700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb364105b20 con 0x7fb3640f5410 2024-02-11T17:10:04.778 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:04.744+0000 7fb36259c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3640f5410 0x7fb364105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:10:04.778 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:04.744+0000 7fb36259c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3640f5410 0x7fb364105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:34686/0 (socket says 172.21.15.178:34686) 2024-02-11T17:10:04.778 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:04.744+0000 7fb36259c700 1 -- 172.21.15.178:0/1586555372 learned_addr learned my addr 172.21.15.178:0/1586555372 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:10:04.778 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:04.744+0000 7fb36259c700 1 -- 172.21.15.178:0/1586555372 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb34c005040 con 0x7fb3640f5410 2024-02-11T17:10:04.778 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:04.744+0000 7fb36259c700 1 --2- 172.21.15.178:0/1586555372 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3640f5410 0x7fb364105630 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7fb34c013510 tx=0x7fb34c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:10:04.778 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:04.744+0000 7fb35affd700 1 -- 172.21.15.178:0/1586555372 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb34c009320 con 0x7fb3640f5410 2024-02-11T17:10:04.778 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:04.744+0000 7fb368ff9700 1 -- 172.21.15.178:0/1586555372 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb364105d20 con 0x7fb3640f5410 2024-02-11T17:10:04.778 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:04.744+0000 7fb35affd700 1 -- 172.21.15.178:0/1586555372 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb34c013f00 con 0x7fb3640f5410 2024-02-11T17:10:04.778 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:04.744+0000 7fb35affd700 1 -- 172.21.15.178:0/1586555372 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb34c014d90 con 0x7fb3640f5410 2024-02-11T17:10:04.778 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:04.744+0000 7fb368ff9700 1 -- 172.21.15.178:0/1586555372 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3641061b0 con 0x7fb3640f5410 2024-02-11T17:10:04.779 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:04.745+0000 7fb35affd700 1 -- 172.21.15.178:0/1586555372 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb34c0095f0 con 0x7fb3640f5410 2024-02-11T17:10:04.779 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:04.745+0000 7fb368ff9700 1 -- 172.21.15.178:0/1586555372 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb364045ad0 con 0x7fb3640f5410 2024-02-11T17:10:04.779 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:04.745+0000 7fb35affd700 1 --2- 172.21.15.178:0/1586555372 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb35002ff60 0x7fb350032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:10:04.779 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:04.745+0000 7fb35affd700 1 -- 172.21.15.178:0/1586555372 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb34c0410c0 con 0x7fb3640f5410 2024-02-11T17:10:04.779 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:04.745+0000 7fb361d9b700 1 --2- 172.21.15.178:0/1586555372 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb35002ff60 0x7fb350032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:10:04.779 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:04.746+0000 7fb361d9b700 1 --2- 172.21.15.178:0/1586555372 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb35002ff60 0x7fb350032420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fb3540096a0 tx=0x7fb354006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:10:04.779 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:04.750+0000 7fb35affd700 1 -- 172.21.15.178:0/1586555372 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb34c014420 con 0x7fb3640f5410 2024-02-11T17:10:04.963 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:04.962+0000 7fb368ff9700 1 -- 172.21.15.178:0/1586555372 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb3640fae60 con 0x7fb3640f5410 2024-02-11T17:10:04.964 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:04.962+0000 7fb35affd700 1 -- 172.21.15.178:0/1586555372 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb34c010230 con 0x7fb3640f5410 2024-02-11T17:10:04.964 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:10:04.964 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:10:04.966 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:04.964+0000 7fb368ff9700 1 -- 172.21.15.178:0/1586555372 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb35002ff60 msgr2=0x7fb350032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:10:04.966 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:04.964+0000 7fb368ff9700 1 --2- 172.21.15.178:0/1586555372 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb35002ff60 0x7fb350032420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fb3540096a0 tx=0x7fb354006b40).stop 2024-02-11T17:10:04.966 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:04.964+0000 7fb368ff9700 1 -- 172.21.15.178:0/1586555372 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3640f5410 msgr2=0x7fb364105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:10:04.966 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:04.964+0000 7fb368ff9700 1 --2- 172.21.15.178:0/1586555372 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3640f5410 0x7fb364105630 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7fb34c013510 tx=0x7fb34c00b040).stop 2024-02-11T17:10:04.966 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:04.964+0000 7fb368ff9700 1 -- 172.21.15.178:0/1586555372 shutdown_connections 2024-02-11T17:10:04.966 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:04.964+0000 7fb368ff9700 1 --2- 172.21.15.178:0/1586555372 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb35002ff60 0x7fb350032420 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:10:04.966 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:04.964+0000 7fb368ff9700 1 --2- 172.21.15.178:0/1586555372 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3640f5410 0x7fb364105630 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:10:04.966 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:04.964+0000 7fb368ff9700 1 -- 172.21.15.178:0/1586555372 >> 172.21.15.178:0/1586555372 conn(0x7fb3640f0b70 msgr2=0x7fb3640f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:10:04.966 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:04.964+0000 7fb368ff9700 1 -- 172.21.15.178:0/1586555372 shutdown_connections 2024-02-11T17:10:04.966 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:04.964+0000 7fb368ff9700 1 -- 172.21.15.178:0/1586555372 wait complete. 2024-02-11T17:10:04.967 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:10:05.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:04 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:10:03.224274+0000 mgr.smithi037.eofnaq (mgr.14162) 309 : cluster [DBG] pgmap v294: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:10:06.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:10:04.963849+0000 mon.smithi037 (mon.0) 299 : audit [DBG] from='client.? 172.21.15.178:0/1586555372' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:10:06.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:10:05.599152+0000 mon.smithi037 (mon.0) 300 : audit [INF] from='mgr.14162 172.21.15.37:0/2745465012' entity='mgr.smithi037.eofnaq' 2024-02-11T17:10:06.981 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:10:06.982 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:10:07.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:06 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:10:05.224618+0000 mgr.smithi037.eofnaq (mgr.14162) 310 : cluster [DBG] pgmap v295: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:10:07.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:06 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:10:06.739144+0000 mon.smithi037 (mon.0) 301 : audit [INF] from='mgr.14162 172.21.15.37:0/2745465012' entity='mgr.smithi037.eofnaq' 2024-02-11T17:10:07.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:06 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:10:06.745050+0000 mon.smithi037 (mon.0) 302 : audit [INF] from='mgr.14162 172.21.15.37:0/2745465012' entity='mgr.smithi037.eofnaq' 2024-02-11T17:10:08.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:07 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:10:06.739960+0000 mgr.smithi037.eofnaq (mgr.14162) 311 : cluster [DBG] pgmap v296: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:10:08.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:07 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cephadm 2024-02-11T17:10:06.747326+0000 mgr.smithi037.eofnaq (mgr.14162) 312 : cephadm [INF] Deploying daemon alertmanager.smithi037 on smithi037 2024-02-11T17:10:08.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:07 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:10:07.737814+0000 mon.smithi037 (mon.0) 303 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-02-11T17:10:08.894 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:08.891+0000 7f4c53c38700 1 -- 172.21.15.178:0/262484385 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4c4c0f5410 msgr2=0x7f4c4c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:10:08.895 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:08.891+0000 7f4c53c38700 1 --2- 172.21.15.178:0/262484385 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4c4c0f5410 0x7f4c4c0f5830 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7f4c48009fb0 tx=0x7f4c48009b30).stop 2024-02-11T17:10:08.895 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:08.891+0000 7f4c53c38700 1 -- 172.21.15.178:0/262484385 shutdown_connections 2024-02-11T17:10:08.895 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:08.891+0000 7f4c53c38700 1 --2- 172.21.15.178:0/262484385 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4c4c0f5410 0x7f4c4c0f5830 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:10:08.895 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:08.891+0000 7f4c53c38700 1 -- 172.21.15.178:0/262484385 >> 172.21.15.178:0/262484385 conn(0x7f4c4c0f0b70 msgr2=0x7f4c4c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:10:08.895 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:08.891+0000 7f4c53c38700 1 -- 172.21.15.178:0/262484385 shutdown_connections 2024-02-11T17:10:08.895 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:08.891+0000 7f4c53c38700 1 -- 172.21.15.178:0/262484385 wait complete. 2024-02-11T17:10:08.895 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:08.892+0000 7f4c53c38700 1 Processor -- start 2024-02-11T17:10:08.895 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:08.892+0000 7f4c53c38700 1 -- start start 2024-02-11T17:10:08.895 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:08.892+0000 7f4c53c38700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4c4c0f5410 0x7f4c4c105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:10:08.895 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:08.892+0000 7f4c53c38700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c4c105af0 con 0x7f4c4c0f5410 2024-02-11T17:10:08.895 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:08.892+0000 7f4c519d4700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4c4c0f5410 0x7f4c4c105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:10:08.895 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:08.892+0000 7f4c519d4700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4c4c0f5410 0x7f4c4c105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:34714/0 (socket says 172.21.15.178:34714) 2024-02-11T17:10:08.896 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:08.892+0000 7f4c519d4700 1 -- 172.21.15.178:0/3218309726 learned_addr learned my addr 172.21.15.178:0/3218309726 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:10:08.896 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:08.893+0000 7f4c519d4700 1 -- 172.21.15.178:0/3218309726 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c48005040 con 0x7f4c4c0f5410 2024-02-11T17:10:08.896 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:08.893+0000 7f4c519d4700 1 --2- 172.21.15.178:0/3218309726 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4c4c0f5410 0x7f4c4c105600 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f4c48013510 tx=0x7f4c4800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:10:08.896 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:08.893+0000 7f4c427fc700 1 -- 172.21.15.178:0/3218309726 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c48009320 con 0x7f4c4c0f5410 2024-02-11T17:10:08.896 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:08.893+0000 7f4c53c38700 1 -- 172.21.15.178:0/3218309726 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c4c105cf0 con 0x7f4c4c0f5410 2024-02-11T17:10:08.896 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:08.893+0000 7f4c53c38700 1 -- 172.21.15.178:0/3218309726 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c4c106180 con 0x7f4c4c0f5410 2024-02-11T17:10:08.896 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:08.893+0000 7f4c427fc700 1 -- 172.21.15.178:0/3218309726 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4c48013f00 con 0x7f4c4c0f5410 2024-02-11T17:10:08.896 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:08.893+0000 7f4c427fc700 1 -- 172.21.15.178:0/3218309726 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c48014d60 con 0x7f4c4c0f5410 2024-02-11T17:10:08.897 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:08.894+0000 7f4c427fc700 1 -- 172.21.15.178:0/3218309726 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4c480263a0 con 0x7f4c4c0f5410 2024-02-11T17:10:08.897 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:08.894+0000 7f4c427fc700 1 --2- 172.21.15.178:0/3218309726 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f4c3802ff60 0x7f4c38032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:10:08.897 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:08.894+0000 7f4c427fc700 1 -- 172.21.15.178:0/3218309726 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4c48040fa0 con 0x7f4c4c0f5410 2024-02-11T17:10:08.897 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:08.894+0000 7f4c53c38700 1 -- 172.21.15.178:0/3218309726 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c34004fa0 con 0x7f4c4c0f5410 2024-02-11T17:10:08.897 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:08.894+0000 7f4c511d3700 1 --2- 172.21.15.178:0/3218309726 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f4c3802ff60 0x7f4c38032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:10:08.897 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:08.895+0000 7f4c511d3700 1 --2- 172.21.15.178:0/3218309726 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f4c3802ff60 0x7f4c38032420 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f4c3c0096a0 tx=0x7f4c3c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:10:08.900 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:08.899+0000 7f4c427fc700 1 -- 172.21.15.178:0/3218309726 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4c4801d380 con 0x7f4c4c0f5410 2024-02-11T17:10:09.102 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:09.100+0000 7f4c53c38700 1 -- 172.21.15.178:0/3218309726 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4c34005670 con 0x7f4c4c0f5410 2024-02-11T17:10:09.103 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:09.101+0000 7f4c427fc700 1 -- 172.21.15.178:0/3218309726 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4c4802b020 con 0x7f4c4c0f5410 2024-02-11T17:10:09.103 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:10:09.103 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:10:09.104 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:09.102+0000 7f4c53c38700 1 -- 172.21.15.178:0/3218309726 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f4c3802ff60 msgr2=0x7f4c38032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:10:09.104 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:09.102+0000 7f4c53c38700 1 --2- 172.21.15.178:0/3218309726 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f4c3802ff60 0x7f4c38032420 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f4c3c0096a0 tx=0x7f4c3c006b40).stop 2024-02-11T17:10:09.104 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:09.102+0000 7f4c53c38700 1 -- 172.21.15.178:0/3218309726 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4c4c0f5410 msgr2=0x7f4c4c105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:10:09.105 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:09.103+0000 7f4c53c38700 1 --2- 172.21.15.178:0/3218309726 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4c4c0f5410 0x7f4c4c105600 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f4c48013510 tx=0x7f4c4800b040).stop 2024-02-11T17:10:09.105 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:09.103+0000 7f4c53c38700 1 -- 172.21.15.178:0/3218309726 shutdown_connections 2024-02-11T17:10:09.105 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:09.103+0000 7f4c53c38700 1 --2- 172.21.15.178:0/3218309726 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f4c3802ff60 0x7f4c38032420 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:10:09.105 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:09.103+0000 7f4c53c38700 1 --2- 172.21.15.178:0/3218309726 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4c4c0f5410 0x7f4c4c105600 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:10:09.105 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:09.103+0000 7f4c53c38700 1 -- 172.21.15.178:0/3218309726 >> 172.21.15.178:0/3218309726 conn(0x7f4c4c0f0b70 msgr2=0x7f4c4c0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:10:09.105 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:09.103+0000 7f4c53c38700 1 -- 172.21.15.178:0/3218309726 shutdown_connections 2024-02-11T17:10:09.105 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:09.103+0000 7f4c53c38700 1 -- 172.21.15.178:0/3218309726 wait complete. 2024-02-11T17:10:09.105 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:10:10.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:09 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:10:08.740433+0000 mgr.smithi037.eofnaq (mgr.14162) 313 : cluster [DBG] pgmap v297: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:10:10.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:09 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:10:09.102480+0000 mon.smithi037 (mon.0) 304 : audit [DBG] from='client.? 172.21.15.178:0/3218309726' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:10:10.940 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:10:10.941 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:10:11.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:11 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:10:10.740844+0000 mgr.smithi037.eofnaq (mgr.14162) 314 : cluster [DBG] pgmap v298: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:10:12.828 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:12.816+0000 7f7e0a582700 1 -- 172.21.15.178:0/837263709 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7e040f51a0 msgr2=0x7f7e040f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:10:12.828 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:12.816+0000 7f7e0a582700 1 --2- 172.21.15.178:0/837263709 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7e040f51a0 0x7f7e040f55c0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f7df4009fb0 tx=0x7f7df4009b30).stop 2024-02-11T17:10:12.828 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:12.816+0000 7f7e0a582700 1 -- 172.21.15.178:0/837263709 shutdown_connections 2024-02-11T17:10:12.828 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:12.816+0000 7f7e0a582700 1 --2- 172.21.15.178:0/837263709 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7e040f51a0 0x7f7e040f55c0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:10:12.828 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:12.817+0000 7f7e0a582700 1 -- 172.21.15.178:0/837263709 >> 172.21.15.178:0/837263709 conn(0x7f7e040f09a0 msgr2=0x7f7e040f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:10:12.828 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:12.817+0000 7f7e0a582700 1 -- 172.21.15.178:0/837263709 shutdown_connections 2024-02-11T17:10:12.829 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:12.817+0000 7f7e0a582700 1 -- 172.21.15.178:0/837263709 wait complete. 2024-02-11T17:10:12.829 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:12.817+0000 7f7e0a582700 1 Processor -- start 2024-02-11T17:10:12.829 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:12.817+0000 7f7e0a582700 1 -- start start 2024-02-11T17:10:12.829 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:12.817+0000 7f7e0a582700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7e040f51a0 0x7f7e041053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:10:12.829 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:12.818+0000 7f7e0a582700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e041058a0 con 0x7f7e040f51a0 2024-02-11T17:10:12.829 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:12.818+0000 7f7e09580700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7e040f51a0 0x7f7e041053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:10:12.829 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:12.818+0000 7f7e09580700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7e040f51a0 0x7f7e041053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:50142/0 (socket says 172.21.15.178:50142) 2024-02-11T17:10:12.829 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:12.818+0000 7f7e09580700 1 -- 172.21.15.178:0/2244335847 learned_addr learned my addr 172.21.15.178:0/2244335847 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:10:12.829 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:12.818+0000 7f7e09580700 1 -- 172.21.15.178:0/2244335847 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7df4005040 con 0x7f7e040f51a0 2024-02-11T17:10:12.829 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:12.818+0000 7f7e09580700 1 --2- 172.21.15.178:0/2244335847 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7e040f51a0 0x7f7e041053b0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f7df4013510 tx=0x7f7df400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:10:12.829 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:12.818+0000 7f7df9ffb700 1 -- 172.21.15.178:0/2244335847 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7df4009320 con 0x7f7e040f51a0 2024-02-11T17:10:12.829 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:12.818+0000 7f7e0a582700 1 -- 172.21.15.178:0/2244335847 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e04105aa0 con 0x7f7e040f51a0 2024-02-11T17:10:12.829 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:12.818+0000 7f7df9ffb700 1 -- 172.21.15.178:0/2244335847 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7df4013f00 con 0x7f7e040f51a0 2024-02-11T17:10:12.829 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:12.818+0000 7f7df9ffb700 1 -- 172.21.15.178:0/2244335847 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7df4014dc0 con 0x7f7e040f51a0 2024-02-11T17:10:12.829 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:12.819+0000 7f7e0a582700 1 -- 172.21.15.178:0/2244335847 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e04105f30 con 0x7f7e040f51a0 2024-02-11T17:10:12.830 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:12.819+0000 7f7df9ffb700 1 -- 172.21.15.178:0/2244335847 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7df401d3d0 con 0x7f7e040f51a0 2024-02-11T17:10:12.830 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:12.819+0000 7f7df9ffb700 1 --2- 172.21.15.178:0/2244335847 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f7df002ff60 0x7f7df0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:10:12.830 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:12.820+0000 7f7df9ffb700 1 -- 172.21.15.178:0/2244335847 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7df40410c0 con 0x7f7e040f51a0 2024-02-11T17:10:12.830 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:12.820+0000 7f7e0a582700 1 -- 172.21.15.178:0/2244335847 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e04045ad0 con 0x7f7e040f51a0 2024-02-11T17:10:12.830 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:12.820+0000 7f7e08d7f700 1 --2- 172.21.15.178:0/2244335847 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f7df002ff60 0x7f7df0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:10:12.830 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:12.820+0000 7f7e08d7f700 1 --2- 172.21.15.178:0/2244335847 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f7df002ff60 0x7f7df0032420 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f7e000096a0 tx=0x7f7e00006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:10:12.830 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:12.825+0000 7f7df9ffb700 1 -- 172.21.15.178:0/2244335847 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7df4012920 con 0x7f7e040f51a0 2024-02-11T17:10:13.032 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:13.030+0000 7f7e0a582700 1 -- 172.21.15.178:0/2244335847 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7e04106c50 con 0x7f7e040f51a0 2024-02-11T17:10:13.032 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:13.031+0000 7f7df9ffb700 1 -- 172.21.15.178:0/2244335847 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7df402b020 con 0x7f7e040f51a0 2024-02-11T17:10:13.032 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:10:13.033 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:10:13.034 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:13.032+0000 7f7e0a582700 1 -- 172.21.15.178:0/2244335847 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f7df002ff60 msgr2=0x7f7df0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:10:13.034 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:13.032+0000 7f7e0a582700 1 --2- 172.21.15.178:0/2244335847 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f7df002ff60 0x7f7df0032420 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f7e000096a0 tx=0x7f7e00006b40).stop 2024-02-11T17:10:13.034 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:13.032+0000 7f7e0a582700 1 -- 172.21.15.178:0/2244335847 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7e040f51a0 msgr2=0x7f7e041053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:10:13.034 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:13.032+0000 7f7e0a582700 1 --2- 172.21.15.178:0/2244335847 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7e040f51a0 0x7f7e041053b0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f7df4013510 tx=0x7f7df400b040).stop 2024-02-11T17:10:13.034 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:13.032+0000 7f7e0a582700 1 -- 172.21.15.178:0/2244335847 shutdown_connections 2024-02-11T17:10:13.034 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:13.032+0000 7f7e0a582700 1 --2- 172.21.15.178:0/2244335847 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f7df002ff60 0x7f7df0032420 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:10:13.034 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:13.032+0000 7f7e0a582700 1 --2- 172.21.15.178:0/2244335847 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7e040f51a0 0x7f7e041053b0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:10:13.034 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:13.032+0000 7f7e0a582700 1 -- 172.21.15.178:0/2244335847 >> 172.21.15.178:0/2244335847 conn(0x7f7e040f09a0 msgr2=0x7f7e040f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:10:13.034 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:13.032+0000 7f7e0a582700 1 -- 172.21.15.178:0/2244335847 shutdown_connections 2024-02-11T17:10:13.034 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:13.032+0000 7f7e0a582700 1 -- 172.21.15.178:0/2244335847 wait complete. 2024-02-11T17:10:13.035 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:10:14.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:10:12.741124+0000 mgr.smithi037.eofnaq (mgr.14162) 315 : cluster [DBG] pgmap v299: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:10:14.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:10:13.031965+0000 mon.smithi037 (mon.0) 305 : audit [DBG] from='client.? 172.21.15.178:0/2244335847' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:10:14.807 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:10:14.807 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:10:16.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:15 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:10:14.741463+0000 mgr.smithi037.eofnaq (mgr.14162) 316 : cluster [DBG] pgmap v300: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:10:16.898 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:16.894+0000 7f4b433f3700 1 -- 172.21.15.178:0/34047034 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4b3c060180 msgr2=0x7f4b3c0f8f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:10:16.898 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:16.894+0000 7f4b433f3700 1 --2- 172.21.15.178:0/34047034 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4b3c060180 0x7f4b3c0f8f80 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f4b2c009fb0 tx=0x7f4b2c009b30).stop 2024-02-11T17:10:16.898 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:16.894+0000 7f4b433f3700 1 -- 172.21.15.178:0/34047034 shutdown_connections 2024-02-11T17:10:16.898 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:16.894+0000 7f4b433f3700 1 --2- 172.21.15.178:0/34047034 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4b3c060180 0x7f4b3c0f8f80 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:10:16.898 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:16.894+0000 7f4b433f3700 1 -- 172.21.15.178:0/34047034 >> 172.21.15.178:0/34047034 conn(0x7f4b3c0f09a0 msgr2=0x7f4b3c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:10:16.898 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:16.894+0000 7f4b433f3700 1 -- 172.21.15.178:0/34047034 shutdown_connections 2024-02-11T17:10:16.898 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:16.894+0000 7f4b433f3700 1 -- 172.21.15.178:0/34047034 wait complete. 2024-02-11T17:10:16.898 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:16.895+0000 7f4b433f3700 1 Processor -- start 2024-02-11T17:10:16.898 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:16.895+0000 7f4b433f3700 1 -- start start 2024-02-11T17:10:16.898 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:16.895+0000 7f4b433f3700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4b3c060180 0x7f4b3c106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:10:16.898 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:16.895+0000 7f4b433f3700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b3c1069f0 con 0x7f4b3c060180 2024-02-11T17:10:16.898 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:16.895+0000 7f4b423f1700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4b3c060180 0x7f4b3c106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:10:16.899 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:16.895+0000 7f4b423f1700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4b3c060180 0x7f4b3c106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:50172/0 (socket says 172.21.15.178:50172) 2024-02-11T17:10:16.899 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:16.895+0000 7f4b423f1700 1 -- 172.21.15.178:0/2732643024 learned_addr learned my addr 172.21.15.178:0/2732643024 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:10:16.899 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:16.895+0000 7f4b423f1700 1 -- 172.21.15.178:0/2732643024 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b2c005040 con 0x7f4b3c060180 2024-02-11T17:10:16.899 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:16.895+0000 7f4b423f1700 1 --2- 172.21.15.178:0/2732643024 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4b3c060180 0x7f4b3c106500 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f4b2c013510 tx=0x7f4b2c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:10:16.899 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:16.896+0000 7f4b32ffd700 1 -- 172.21.15.178:0/2732643024 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b2c009320 con 0x7f4b3c060180 2024-02-11T17:10:16.899 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:16.896+0000 7f4b32ffd700 1 -- 172.21.15.178:0/2732643024 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4b2c013f00 con 0x7f4b3c060180 2024-02-11T17:10:16.899 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:16.896+0000 7f4b433f3700 1 -- 172.21.15.178:0/2732643024 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b3c106bf0 con 0x7f4b3c060180 2024-02-11T17:10:16.899 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:16.896+0000 7f4b32ffd700 1 -- 172.21.15.178:0/2732643024 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b2c014d90 con 0x7f4b3c060180 2024-02-11T17:10:16.899 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:16.896+0000 7f4b433f3700 1 -- 172.21.15.178:0/2732643024 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b3c107080 con 0x7f4b3c060180 2024-02-11T17:10:16.900 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:16.896+0000 7f4b32ffd700 1 -- 172.21.15.178:0/2732643024 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4b2c014420 con 0x7f4b3c060180 2024-02-11T17:10:16.900 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:16.897+0000 7f4b32ffd700 1 --2- 172.21.15.178:0/2732643024 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f4b2802ff60 0x7f4b28032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:10:16.900 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:16.897+0000 7f4b32ffd700 1 -- 172.21.15.178:0/2732643024 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4b2c041000 con 0x7f4b3c060180 2024-02-11T17:10:16.900 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:16.897+0000 7f4b433f3700 1 -- 172.21.15.178:0/2732643024 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b3c045ad0 con 0x7f4b3c060180 2024-02-11T17:10:16.900 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:16.897+0000 7f4b41bf0700 1 --2- 172.21.15.178:0/2732643024 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f4b2802ff60 0x7f4b28032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:10:16.900 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:16.897+0000 7f4b41bf0700 1 --2- 172.21.15.178:0/2732643024 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f4b2802ff60 0x7f4b28032420 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f4b380096a0 tx=0x7f4b38006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:10:16.903 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:16.902+0000 7f4b32ffd700 1 -- 172.21.15.178:0/2732643024 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4b2c0095f0 con 0x7f4b3c060180 2024-02-11T17:10:17.110 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:17.107+0000 7f4b433f3700 1 -- 172.21.15.178:0/2732643024 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4b3c0fbdd0 con 0x7f4b3c060180 2024-02-11T17:10:17.110 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:17.108+0000 7f4b32ffd700 1 -- 172.21.15.178:0/2732643024 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4b2c01db70 con 0x7f4b3c060180 2024-02-11T17:10:17.110 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:10:17.111 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:10:17.112 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:17.109+0000 7f4b433f3700 1 -- 172.21.15.178:0/2732643024 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f4b2802ff60 msgr2=0x7f4b28032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:10:17.112 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:17.109+0000 7f4b433f3700 1 --2- 172.21.15.178:0/2732643024 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f4b2802ff60 0x7f4b28032420 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f4b380096a0 tx=0x7f4b38006b40).stop 2024-02-11T17:10:17.112 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:17.109+0000 7f4b433f3700 1 -- 172.21.15.178:0/2732643024 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4b3c060180 msgr2=0x7f4b3c106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:10:17.112 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:17.109+0000 7f4b433f3700 1 --2- 172.21.15.178:0/2732643024 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4b3c060180 0x7f4b3c106500 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f4b2c013510 tx=0x7f4b2c00b040).stop 2024-02-11T17:10:17.113 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:17.109+0000 7f4b433f3700 1 -- 172.21.15.178:0/2732643024 shutdown_connections 2024-02-11T17:10:17.113 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:17.109+0000 7f4b433f3700 1 --2- 172.21.15.178:0/2732643024 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f4b2802ff60 0x7f4b28032420 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:10:17.113 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:17.109+0000 7f4b433f3700 1 --2- 172.21.15.178:0/2732643024 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4b3c060180 0x7f4b3c106500 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:10:17.113 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:17.109+0000 7f4b433f3700 1 -- 172.21.15.178:0/2732643024 >> 172.21.15.178:0/2732643024 conn(0x7f4b3c0f09a0 msgr2=0x7f4b3c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:10:17.113 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:17.109+0000 7f4b433f3700 1 -- 172.21.15.178:0/2732643024 shutdown_connections 2024-02-11T17:10:17.113 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:17.109+0000 7f4b433f3700 1 -- 172.21.15.178:0/2732643024 wait complete. 2024-02-11T17:10:17.113 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:10:18.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:17 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:10:16.741816+0000 mgr.smithi037.eofnaq (mgr.14162) 317 : cluster [DBG] pgmap v301: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:10:18.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:17 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:10:17.109119+0000 mon.smithi037 (mon.0) 306 : audit [DBG] from='client.? 172.21.15.178:0/2732643024' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:10:18.857 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:10:18.857 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:10:20.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:19 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:10:18.742064+0000 mgr.smithi037.eofnaq (mgr.14162) 318 : cluster [DBG] pgmap v302: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:10:20.882 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:20.879+0000 7f7a732e8700 1 -- 172.21.15.178:0/2027135935 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7a6c0f5410 msgr2=0x7f7a6c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:10:20.882 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:20.879+0000 7f7a732e8700 1 --2- 172.21.15.178:0/2027135935 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7a6c0f5410 0x7f7a6c0f5830 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f7a60009fb0 tx=0x7f7a60009b30).stop 2024-02-11T17:10:20.882 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:20.879+0000 7f7a732e8700 1 -- 172.21.15.178:0/2027135935 shutdown_connections 2024-02-11T17:10:20.882 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:20.879+0000 7f7a732e8700 1 --2- 172.21.15.178:0/2027135935 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7a6c0f5410 0x7f7a6c0f5830 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:10:20.882 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:20.879+0000 7f7a732e8700 1 -- 172.21.15.178:0/2027135935 >> 172.21.15.178:0/2027135935 conn(0x7f7a6c0f0b70 msgr2=0x7f7a6c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:10:20.882 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:20.879+0000 7f7a732e8700 1 -- 172.21.15.178:0/2027135935 shutdown_connections 2024-02-11T17:10:20.882 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:20.879+0000 7f7a732e8700 1 -- 172.21.15.178:0/2027135935 wait complete. 2024-02-11T17:10:20.882 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:20.879+0000 7f7a732e8700 1 Processor -- start 2024-02-11T17:10:20.883 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:20.879+0000 7f7a732e8700 1 -- start start 2024-02-11T17:10:20.883 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:20.880+0000 7f7a732e8700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7a6c0f5410 0x7f7a6c105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:10:20.883 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:20.880+0000 7f7a732e8700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a6c105af0 con 0x7f7a6c0f5410 2024-02-11T17:10:20.883 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:20.880+0000 7f7a71084700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7a6c0f5410 0x7f7a6c105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:10:20.883 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:20.880+0000 7f7a71084700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7a6c0f5410 0x7f7a6c105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:42336/0 (socket says 172.21.15.178:42336) 2024-02-11T17:10:20.883 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:20.880+0000 7f7a71084700 1 -- 172.21.15.178:0/2071731384 learned_addr learned my addr 172.21.15.178:0/2071731384 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:10:20.883 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:20.880+0000 7f7a71084700 1 -- 172.21.15.178:0/2071731384 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a60005040 con 0x7f7a6c0f5410 2024-02-11T17:10:20.883 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:20.880+0000 7f7a71084700 1 --2- 172.21.15.178:0/2071731384 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7a6c0f5410 0x7f7a6c105600 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f7a60014000 tx=0x7f7a6000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:10:20.883 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:20.880+0000 7f7a5dffb700 1 -- 172.21.15.178:0/2071731384 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a600095f0 con 0x7f7a6c0f5410 2024-02-11T17:10:20.883 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:20.881+0000 7f7a732e8700 1 -- 172.21.15.178:0/2071731384 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a6c105cf0 con 0x7f7a6c0f5410 2024-02-11T17:10:20.883 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:20.881+0000 7f7a5dffb700 1 -- 172.21.15.178:0/2071731384 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7a60007e70 con 0x7f7a6c0f5410 2024-02-11T17:10:20.883 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:20.881+0000 7f7a5dffb700 1 -- 172.21.15.178:0/2071731384 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a60014ea0 con 0x7f7a6c0f5410 2024-02-11T17:10:20.884 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:20.881+0000 7f7a732e8700 1 -- 172.21.15.178:0/2071731384 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a6c106180 con 0x7f7a6c0f5410 2024-02-11T17:10:20.884 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:20.882+0000 7f7a5dffb700 1 -- 172.21.15.178:0/2071731384 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7a60025070 con 0x7f7a6c0f5410 2024-02-11T17:10:20.884 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:20.882+0000 7f7a5dffb700 1 --2- 172.21.15.178:0/2071731384 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f7a5802ff60 0x7f7a58032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:10:20.884 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:20.882+0000 7f7a5dffb700 1 -- 172.21.15.178:0/2071731384 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7a60040d20 con 0x7f7a6c0f5410 2024-02-11T17:10:20.885 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:20.882+0000 7f7a732e8700 1 -- 172.21.15.178:0/2071731384 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a6c045ad0 con 0x7f7a6c0f5410 2024-02-11T17:10:20.885 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:20.882+0000 7f7a70883700 1 --2- 172.21.15.178:0/2071731384 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f7a5802ff60 0x7f7a58032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:10:20.885 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:20.882+0000 7f7a70883700 1 --2- 172.21.15.178:0/2071731384 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f7a5802ff60 0x7f7a58032420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f7a680096a0 tx=0x7f7a68006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:10:20.888 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:20.887+0000 7f7a5dffb700 1 -- 172.21.15.178:0/2071731384 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7a60026350 con 0x7f7a6c0f5410 2024-02-11T17:10:21.092 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:21.091+0000 7f7a732e8700 1 -- 172.21.15.178:0/2071731384 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7a6c0faf50 con 0x7f7a6c0f5410 2024-02-11T17:10:21.093 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:21.092+0000 7f7a5dffb700 1 -- 172.21.15.178:0/2071731384 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7a60012330 con 0x7f7a6c0f5410 2024-02-11T17:10:21.093 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:10:21.093 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:10:21.095 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:21.093+0000 7f7a732e8700 1 -- 172.21.15.178:0/2071731384 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f7a5802ff60 msgr2=0x7f7a58032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:10:21.095 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:21.093+0000 7f7a732e8700 1 --2- 172.21.15.178:0/2071731384 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f7a5802ff60 0x7f7a58032420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f7a680096a0 tx=0x7f7a68006b40).stop 2024-02-11T17:10:21.095 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:21.093+0000 7f7a732e8700 1 -- 172.21.15.178:0/2071731384 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7a6c0f5410 msgr2=0x7f7a6c105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:10:21.095 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:21.093+0000 7f7a732e8700 1 --2- 172.21.15.178:0/2071731384 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7a6c0f5410 0x7f7a6c105600 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f7a60014000 tx=0x7f7a6000b040).stop 2024-02-11T17:10:21.095 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:21.093+0000 7f7a732e8700 1 -- 172.21.15.178:0/2071731384 shutdown_connections 2024-02-11T17:10:21.095 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:21.093+0000 7f7a732e8700 1 --2- 172.21.15.178:0/2071731384 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f7a5802ff60 0x7f7a58032420 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:10:21.095 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:21.093+0000 7f7a732e8700 1 --2- 172.21.15.178:0/2071731384 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7a6c0f5410 0x7f7a6c105600 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:10:21.096 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:21.093+0000 7f7a732e8700 1 -- 172.21.15.178:0/2071731384 >> 172.21.15.178:0/2071731384 conn(0x7f7a6c0f0b70 msgr2=0x7f7a6c0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:10:21.096 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:21.093+0000 7f7a732e8700 1 -- 172.21.15.178:0/2071731384 shutdown_connections 2024-02-11T17:10:21.096 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:21.093+0000 7f7a732e8700 1 -- 172.21.15.178:0/2071731384 wait complete. 2024-02-11T17:10:21.096 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:10:21.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:21 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:10:20.742358+0000 mgr.smithi037.eofnaq (mgr.14162) 319 : cluster [DBG] pgmap v303: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:10:21.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:21 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:10:21.092836+0000 mon.smithi037 (mon.0) 307 : audit [DBG] from='client.? 172.21.15.178:0/2071731384' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:10:22.701 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:10:22.702 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:10:24.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:23 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:10:22.742612+0000 mgr.smithi037.eofnaq (mgr.14162) 320 : cluster [DBG] pgmap v304: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:10:24.598 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:24.596+0000 7fafd5d3e700 1 -- 172.21.15.178:0/1865639641 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fafd00f47c0 msgr2=0x7fafd00f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:10:24.598 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:24.596+0000 7fafd5d3e700 1 --2- 172.21.15.178:0/1865639641 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fafd00f47c0 0x7fafd00f4be0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7fafc4009fb0 tx=0x7fafc4009b30).stop 2024-02-11T17:10:24.598 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:24.596+0000 7fafd5d3e700 1 -- 172.21.15.178:0/1865639641 shutdown_connections 2024-02-11T17:10:24.599 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:24.596+0000 7fafd5d3e700 1 --2- 172.21.15.178:0/1865639641 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fafd00f47c0 0x7fafd00f4be0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:10:24.599 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:24.596+0000 7fafd5d3e700 1 -- 172.21.15.178:0/1865639641 >> 172.21.15.178:0/1865639641 conn(0x7fafd00f02e0 msgr2=0x7fafd00f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:10:24.599 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:24.596+0000 7fafd5d3e700 1 -- 172.21.15.178:0/1865639641 shutdown_connections 2024-02-11T17:10:24.599 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:24.596+0000 7fafd5d3e700 1 -- 172.21.15.178:0/1865639641 wait complete. 2024-02-11T17:10:24.599 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:24.597+0000 7fafd5d3e700 1 Processor -- start 2024-02-11T17:10:24.599 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:24.597+0000 7fafd5d3e700 1 -- start start 2024-02-11T17:10:24.600 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:24.597+0000 7fafd5d3e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fafd00f47c0 0x7fafd00f7370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:10:24.601 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:24.597+0000 7fafd5d3e700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafd00f57f0 con 0x7fafd00f47c0 2024-02-11T17:10:24.601 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:24.597+0000 7fafcf7fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fafd00f47c0 0x7fafd00f7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:10:24.601 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:24.597+0000 7fafcf7fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fafd00f47c0 0x7fafd00f7370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:42346/0 (socket says 172.21.15.178:42346) 2024-02-11T17:10:24.601 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:24.597+0000 7fafcf7fe700 1 -- 172.21.15.178:0/1023633900 learned_addr learned my addr 172.21.15.178:0/1023633900 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:10:24.601 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:24.598+0000 7fafcf7fe700 1 -- 172.21.15.178:0/1023633900 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafc4005040 con 0x7fafd00f47c0 2024-02-11T17:10:24.601 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:24.598+0000 7fafcf7fe700 1 --2- 172.21.15.178:0/1023633900 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fafd00f47c0 0x7fafd00f7370 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7fafc4013510 tx=0x7fafc400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:10:24.601 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:24.598+0000 7fafb7fff700 1 -- 172.21.15.178:0/1023633900 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafc4009320 con 0x7fafd00f47c0 2024-02-11T17:10:24.601 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:24.598+0000 7fafd5d3e700 1 -- 172.21.15.178:0/1023633900 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafd00f5990 con 0x7fafd00f47c0 2024-02-11T17:10:24.601 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:24.598+0000 7fafb7fff700 1 -- 172.21.15.178:0/1023633900 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fafc4013f00 con 0x7fafd00f47c0 2024-02-11T17:10:24.601 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:24.598+0000 7fafb7fff700 1 -- 172.21.15.178:0/1023633900 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafc4014d90 con 0x7fafd00f47c0 2024-02-11T17:10:24.601 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:24.598+0000 7fafd5d3e700 1 -- 172.21.15.178:0/1023633900 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafd00f5e20 con 0x7fafd00f47c0 2024-02-11T17:10:24.601 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:24.599+0000 7fafb7fff700 1 -- 172.21.15.178:0/1023633900 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fafc40095f0 con 0x7fafd00f47c0 2024-02-11T17:10:24.601 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:24.599+0000 7fafd5d3e700 1 -- 172.21.15.178:0/1023633900 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafd0059100 con 0x7fafd00f47c0 2024-02-11T17:10:24.601 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:24.599+0000 7fafb7fff700 1 --2- 172.21.15.178:0/1023633900 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fafb802ff60 0x7fafb8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:10:24.602 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:24.599+0000 7fafb7fff700 1 -- 172.21.15.178:0/1023633900 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fafc4040fe0 con 0x7fafd00f47c0 2024-02-11T17:10:24.602 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:24.599+0000 7fafceffd700 1 --2- 172.21.15.178:0/1023633900 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fafb802ff60 0x7fafb8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:10:24.603 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:24.600+0000 7fafceffd700 1 --2- 172.21.15.178:0/1023633900 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fafb802ff60 0x7fafb8032420 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7fafc00096a0 tx=0x7fafc0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:10:24.606 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:24.604+0000 7fafb7fff700 1 -- 172.21.15.178:0/1023633900 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fafc401d7f0 con 0x7fafd00f47c0 2024-02-11T17:10:24.811 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:24.809+0000 7fafd5d3e700 1 -- 172.21.15.178:0/1023633900 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fafd00fbf60 con 0x7fafd00f47c0 2024-02-11T17:10:24.812 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:24.810+0000 7fafb7fff700 1 -- 172.21.15.178:0/1023633900 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fafc4010230 con 0x7fafd00f47c0 2024-02-11T17:10:24.812 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:10:24.812 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:10:24.813 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:24.812+0000 7fafd5d3e700 1 -- 172.21.15.178:0/1023633900 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fafb802ff60 msgr2=0x7fafb8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:10:24.813 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:24.812+0000 7fafd5d3e700 1 --2- 172.21.15.178:0/1023633900 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fafb802ff60 0x7fafb8032420 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7fafc00096a0 tx=0x7fafc0006b40).stop 2024-02-11T17:10:24.813 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:24.812+0000 7fafd5d3e700 1 -- 172.21.15.178:0/1023633900 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fafd00f47c0 msgr2=0x7fafd00f7370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:10:24.813 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:24.812+0000 7fafd5d3e700 1 --2- 172.21.15.178:0/1023633900 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fafd00f47c0 0x7fafd00f7370 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7fafc4013510 tx=0x7fafc400b040).stop 2024-02-11T17:10:24.813 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:24.812+0000 7fafd5d3e700 1 -- 172.21.15.178:0/1023633900 shutdown_connections 2024-02-11T17:10:24.813 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:24.812+0000 7fafd5d3e700 1 --2- 172.21.15.178:0/1023633900 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fafb802ff60 0x7fafb8032420 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:10:24.814 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:24.812+0000 7fafd5d3e700 1 --2- 172.21.15.178:0/1023633900 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fafd00f47c0 0x7fafd00f7370 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:10:24.814 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:24.812+0000 7fafd5d3e700 1 -- 172.21.15.178:0/1023633900 >> 172.21.15.178:0/1023633900 conn(0x7fafd00f02e0 msgr2=0x7fafd00f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:10:24.814 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:24.812+0000 7fafd5d3e700 1 -- 172.21.15.178:0/1023633900 shutdown_connections 2024-02-11T17:10:24.814 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:24.812+0000 7fafd5d3e700 1 -- 172.21.15.178:0/1023633900 wait complete. 2024-02-11T17:10:24.814 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:10:26.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:25 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:10:24.742868+0000 mgr.smithi037.eofnaq (mgr.14162) 321 : cluster [DBG] pgmap v305: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:10:26.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:25 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:10:24.811335+0000 mon.smithi037 (mon.0) 308 : audit [DBG] from='client.? 172.21.15.178:0/1023633900' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:10:26.720 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:10:26.720 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:10:28.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:27 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:10:26.743127+0000 mgr.smithi037.eofnaq (mgr.14162) 322 : cluster [DBG] pgmap v306: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:10:28.667 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:28.666+0000 7fcf8a5ce700 1 -- 172.21.15.178:0/2249498793 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcf840f6510 msgr2=0x7fcf840f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:10:28.668 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:28.666+0000 7fcf8a5ce700 1 --2- 172.21.15.178:0/2249498793 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcf840f6510 0x7fcf840f6930 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7fcf74009fb0 tx=0x7fcf74009b30).stop 2024-02-11T17:10:28.668 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:28.666+0000 7fcf8a5ce700 1 -- 172.21.15.178:0/2249498793 shutdown_connections 2024-02-11T17:10:28.668 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:28.666+0000 7fcf8a5ce700 1 --2- 172.21.15.178:0/2249498793 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcf840f6510 0x7fcf840f6930 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:10:28.668 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:28.666+0000 7fcf8a5ce700 1 -- 172.21.15.178:0/2249498793 >> 172.21.15.178:0/2249498793 conn(0x7fcf840f1c70 msgr2=0x7fcf840f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:10:28.668 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:28.666+0000 7fcf8a5ce700 1 -- 172.21.15.178:0/2249498793 shutdown_connections 2024-02-11T17:10:28.668 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:28.666+0000 7fcf8a5ce700 1 -- 172.21.15.178:0/2249498793 wait complete. 2024-02-11T17:10:28.668 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:28.667+0000 7fcf8a5ce700 1 Processor -- start 2024-02-11T17:10:28.668 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:28.667+0000 7fcf8a5ce700 1 -- start start 2024-02-11T17:10:28.670 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:28.667+0000 7fcf8a5ce700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcf840f6510 0x7fcf84106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:10:28.670 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:28.667+0000 7fcf8a5ce700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf84106c20 con 0x7fcf840f6510 2024-02-11T17:10:28.670 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:28.668+0000 7fcf83fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcf840f6510 0x7fcf84106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:10:28.670 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:28.668+0000 7fcf83fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcf840f6510 0x7fcf84106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:42350/0 (socket says 172.21.15.178:42350) 2024-02-11T17:10:28.670 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:28.668+0000 7fcf83fff700 1 -- 172.21.15.178:0/3615565415 learned_addr learned my addr 172.21.15.178:0/3615565415 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:10:28.670 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:28.668+0000 7fcf83fff700 1 -- 172.21.15.178:0/3615565415 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf74005040 con 0x7fcf840f6510 2024-02-11T17:10:28.670 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:28.668+0000 7fcf83fff700 1 --2- 172.21.15.178:0/3615565415 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcf840f6510 0x7fcf84106730 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7fcf74013510 tx=0x7fcf7400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:10:28.670 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:28.668+0000 7fcf80ff9700 1 -- 172.21.15.178:0/3615565415 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf74009320 con 0x7fcf840f6510 2024-02-11T17:10:28.670 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:28.668+0000 7fcf80ff9700 1 -- 172.21.15.178:0/3615565415 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcf74013f00 con 0x7fcf840f6510 2024-02-11T17:10:28.670 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:28.668+0000 7fcf8a5ce700 1 -- 172.21.15.178:0/3615565415 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf84106e20 con 0x7fcf840f6510 2024-02-11T17:10:28.671 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:28.668+0000 7fcf80ff9700 1 -- 172.21.15.178:0/3615565415 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf74014d90 con 0x7fcf840f6510 2024-02-11T17:10:28.671 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:28.668+0000 7fcf8a5ce700 1 -- 172.21.15.178:0/3615565415 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf841072b0 con 0x7fcf840f6510 2024-02-11T17:10:28.671 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:28.669+0000 7fcf80ff9700 1 -- 172.21.15.178:0/3615565415 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcf740095f0 con 0x7fcf840f6510 2024-02-11T17:10:28.671 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:28.669+0000 7fcf8a5ce700 1 -- 172.21.15.178:0/3615565415 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf84045ad0 con 0x7fcf840f6510 2024-02-11T17:10:28.671 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:28.669+0000 7fcf80ff9700 1 --2- 172.21.15.178:0/3615565415 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fcf6c02ff60 0x7fcf6c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:10:28.671 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:28.669+0000 7fcf80ff9700 1 -- 172.21.15.178:0/3615565415 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcf74041000 con 0x7fcf840f6510 2024-02-11T17:10:28.671 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:28.669+0000 7fcf837fe700 1 --2- 172.21.15.178:0/3615565415 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fcf6c02ff60 0x7fcf6c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:10:28.671 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:28.670+0000 7fcf837fe700 1 --2- 172.21.15.178:0/3615565415 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fcf6c02ff60 0x7fcf6c032420 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fcf780096a0 tx=0x7fcf78006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:10:28.674 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:28.674+0000 7fcf80ff9700 1 -- 172.21.15.178:0/3615565415 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcf7401d7f0 con 0x7fcf840f6510 2024-02-11T17:10:28.878 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:28.878+0000 7fcf8a5ce700 1 -- 172.21.15.178:0/3615565415 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcf840fbf60 con 0x7fcf840f6510 2024-02-11T17:10:28.879 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:28.878+0000 7fcf80ff9700 1 -- 172.21.15.178:0/3615565415 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcf74010230 con 0x7fcf840f6510 2024-02-11T17:10:28.879 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:10:28.880 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:10:28.881 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:28.880+0000 7fcf8a5ce700 1 -- 172.21.15.178:0/3615565415 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fcf6c02ff60 msgr2=0x7fcf6c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:10:28.881 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:28.880+0000 7fcf8a5ce700 1 --2- 172.21.15.178:0/3615565415 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fcf6c02ff60 0x7fcf6c032420 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fcf780096a0 tx=0x7fcf78006b40).stop 2024-02-11T17:10:28.881 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:28.880+0000 7fcf8a5ce700 1 -- 172.21.15.178:0/3615565415 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcf840f6510 msgr2=0x7fcf84106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:10:28.881 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:28.880+0000 7fcf8a5ce700 1 --2- 172.21.15.178:0/3615565415 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcf840f6510 0x7fcf84106730 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7fcf74013510 tx=0x7fcf7400b040).stop 2024-02-11T17:10:28.881 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:28.880+0000 7fcf8a5ce700 1 -- 172.21.15.178:0/3615565415 shutdown_connections 2024-02-11T17:10:28.881 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:28.880+0000 7fcf8a5ce700 1 --2- 172.21.15.178:0/3615565415 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fcf6c02ff60 0x7fcf6c032420 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:10:28.881 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:28.880+0000 7fcf8a5ce700 1 --2- 172.21.15.178:0/3615565415 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcf840f6510 0x7fcf84106730 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:10:28.881 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:28.880+0000 7fcf8a5ce700 1 -- 172.21.15.178:0/3615565415 >> 172.21.15.178:0/3615565415 conn(0x7fcf840f1c70 msgr2=0x7fcf840f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:10:28.881 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:28.880+0000 7fcf8a5ce700 1 -- 172.21.15.178:0/3615565415 shutdown_connections 2024-02-11T17:10:28.882 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:28.880+0000 7fcf8a5ce700 1 -- 172.21.15.178:0/3615565415 wait complete. 2024-02-11T17:10:28.882 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:10:30.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:29 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:10:28.743417+0000 mgr.smithi037.eofnaq (mgr.14162) 323 : cluster [DBG] pgmap v307: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:10:30.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:29 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:10:28.878759+0000 mon.smithi037 (mon.0) 309 : audit [DBG] from='client.? 172.21.15.178:0/3615565415' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:10:30.574 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:10:30.575 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:10:31.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:31 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:10:30.743779+0000 mgr.smithi037.eofnaq (mgr.14162) 324 : cluster [DBG] pgmap v308: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:10:32.543 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:32.540+0000 7f0ef3f48700 1 -- 172.21.15.178:0/1738095000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0eec060200 msgr2=0x7f0eec0fb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:10:32.543 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:32.540+0000 7f0ef3f48700 1 --2- 172.21.15.178:0/1738095000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0eec060200 0x7f0eec0fb570 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f0edc009fb0 tx=0x7f0edc009b30).stop 2024-02-11T17:10:32.543 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:32.540+0000 7f0ef3f48700 1 -- 172.21.15.178:0/1738095000 shutdown_connections 2024-02-11T17:10:32.543 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:32.540+0000 7f0ef3f48700 1 --2- 172.21.15.178:0/1738095000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0eec060200 0x7f0eec0fb570 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:10:32.543 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:32.540+0000 7f0ef3f48700 1 -- 172.21.15.178:0/1738095000 >> 172.21.15.178:0/1738095000 conn(0x7f0eec0f0b70 msgr2=0x7f0eec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:10:32.544 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:32.541+0000 7f0ef3f48700 1 -- 172.21.15.178:0/1738095000 shutdown_connections 2024-02-11T17:10:32.544 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:32.541+0000 7f0ef3f48700 1 -- 172.21.15.178:0/1738095000 wait complete. 2024-02-11T17:10:32.544 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:32.541+0000 7f0ef3f48700 1 Processor -- start 2024-02-11T17:10:32.544 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:32.541+0000 7f0ef3f48700 1 -- start start 2024-02-11T17:10:32.544 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:32.541+0000 7f0ef3f48700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0eec060200 0x7f0eec1088d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:10:32.544 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:32.541+0000 7f0ef3f48700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eec108dc0 con 0x7f0eec060200 2024-02-11T17:10:32.544 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:32.542+0000 7f0ef1ce4700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0eec060200 0x7f0eec1088d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:10:32.544 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:32.542+0000 7f0ef1ce4700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0eec060200 0x7f0eec1088d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:34356/0 (socket says 172.21.15.178:34356) 2024-02-11T17:10:32.544 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:32.542+0000 7f0ef1ce4700 1 -- 172.21.15.178:0/167897613 learned_addr learned my addr 172.21.15.178:0/167897613 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:10:32.544 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:32.542+0000 7f0ef1ce4700 1 -- 172.21.15.178:0/167897613 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0edc005040 con 0x7f0eec060200 2024-02-11T17:10:32.544 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:32.542+0000 7f0ef1ce4700 1 --2- 172.21.15.178:0/167897613 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0eec060200 0x7f0eec1088d0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f0edc009830 tx=0x7f0edc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:10:32.544 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:32.542+0000 7f0ee27fc700 1 -- 172.21.15.178:0/167897613 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0edc009320 con 0x7f0eec060200 2024-02-11T17:10:32.544 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:32.542+0000 7f0ee27fc700 1 -- 172.21.15.178:0/167897613 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0edc0136c0 con 0x7f0eec060200 2024-02-11T17:10:32.544 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:32.542+0000 7f0ef3f48700 1 -- 172.21.15.178:0/167897613 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0eec108fc0 con 0x7f0eec060200 2024-02-11T17:10:32.545 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:32.542+0000 7f0ee27fc700 1 -- 172.21.15.178:0/167897613 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0edc01ee90 con 0x7f0eec060200 2024-02-11T17:10:32.545 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:32.542+0000 7f0ef3f48700 1 -- 172.21.15.178:0/167897613 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0eec109450 con 0x7f0eec060200 2024-02-11T17:10:32.545 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:32.543+0000 7f0ee27fc700 1 -- 172.21.15.178:0/167897613 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0edc01e420 con 0x7f0eec060200 2024-02-11T17:10:32.546 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:32.543+0000 7f0ef3f48700 1 -- 172.21.15.178:0/167897613 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0eec045ad0 con 0x7f0eec060200 2024-02-11T17:10:32.546 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:32.543+0000 7f0ee27fc700 1 --2- 172.21.15.178:0/167897613 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f0ed802ff10 0x7f0ed80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:10:32.546 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:32.543+0000 7f0ee27fc700 1 -- 172.21.15.178:0/167897613 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0edc0410e0 con 0x7f0eec060200 2024-02-11T17:10:32.546 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:32.544+0000 7f0ef14e3700 1 --2- 172.21.15.178:0/167897613 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f0ed802ff10 0x7f0ed80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:10:32.546 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:32.544+0000 7f0ef14e3700 1 --2- 172.21.15.178:0/167897613 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f0ed802ff10 0x7f0ed80323d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f0ee80096a0 tx=0x7f0ee8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:10:32.549 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:32.549+0000 7f0ee27fc700 1 -- 172.21.15.178:0/167897613 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0edc0095f0 con 0x7f0eec060200 2024-02-11T17:10:32.753 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:32.752+0000 7f0ef3f48700 1 -- 172.21.15.178:0/167897613 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0eec10a300 con 0x7f0eec060200 2024-02-11T17:10:32.754 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:32.753+0000 7f0ee27fc700 1 -- 172.21.15.178:0/167897613 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0edc01cb10 con 0x7f0eec060200 2024-02-11T17:10:32.754 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:10:32.754 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:10:32.755 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:32.755+0000 7f0ef3f48700 1 -- 172.21.15.178:0/167897613 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f0ed802ff10 msgr2=0x7f0ed80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:10:32.755 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:32.755+0000 7f0ef3f48700 1 --2- 172.21.15.178:0/167897613 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f0ed802ff10 0x7f0ed80323d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f0ee80096a0 tx=0x7f0ee8006b40).stop 2024-02-11T17:10:32.756 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:32.755+0000 7f0ef3f48700 1 -- 172.21.15.178:0/167897613 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0eec060200 msgr2=0x7f0eec1088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:10:32.756 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:32.755+0000 7f0ef3f48700 1 --2- 172.21.15.178:0/167897613 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0eec060200 0x7f0eec1088d0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f0edc009830 tx=0x7f0edc00b040).stop 2024-02-11T17:10:32.756 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:32.755+0000 7f0ef3f48700 1 -- 172.21.15.178:0/167897613 shutdown_connections 2024-02-11T17:10:32.756 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:32.755+0000 7f0ef3f48700 1 --2- 172.21.15.178:0/167897613 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f0ed802ff10 0x7f0ed80323d0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:10:32.756 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:32.755+0000 7f0ef3f48700 1 --2- 172.21.15.178:0/167897613 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0eec060200 0x7f0eec1088d0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:10:32.756 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:32.755+0000 7f0ef3f48700 1 -- 172.21.15.178:0/167897613 >> 172.21.15.178:0/167897613 conn(0x7f0eec0f0b70 msgr2=0x7f0eec0f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:10:32.756 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:32.755+0000 7f0ef3f48700 1 -- 172.21.15.178:0/167897613 shutdown_connections 2024-02-11T17:10:32.756 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:32.755+0000 7f0ef3f48700 1 -- 172.21.15.178:0/167897613 wait complete. 2024-02-11T17:10:32.756 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:10:33.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:32 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:10:32.753390+0000 mon.smithi037 (mon.0) 310 : audit [DBG] from='client.? 172.21.15.178:0/167897613' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:10:34.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:33 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:10:32.744009+0000 mgr.smithi037.eofnaq (mgr.14162) 325 : cluster [DBG] pgmap v309: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:10:34.518 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:10:34.518 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:10:36.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:35 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:10:34.744351+0000 mgr.smithi037.eofnaq (mgr.14162) 326 : cluster [DBG] pgmap v310: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:10:36.437 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:36.417+0000 7fbea1edb700 1 -- 172.21.15.178:0/611763590 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbe9c0601e0 msgr2=0x7fbe9c0f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:10:36.437 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:36.417+0000 7fbea1edb700 1 --2- 172.21.15.178:0/611763590 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbe9c0601e0 0x7fbe9c0f8f20 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7fbe90009fb0 tx=0x7fbe90009b30).stop 2024-02-11T17:10:36.437 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:36.417+0000 7fbea1edb700 1 -- 172.21.15.178:0/611763590 shutdown_connections 2024-02-11T17:10:36.437 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:36.417+0000 7fbea1edb700 1 --2- 172.21.15.178:0/611763590 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbe9c0601e0 0x7fbe9c0f8f20 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:10:36.438 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:36.417+0000 7fbea1edb700 1 -- 172.21.15.178:0/611763590 >> 172.21.15.178:0/611763590 conn(0x7fbe9c0f09a0 msgr2=0x7fbe9c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:10:36.438 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:36.417+0000 7fbea1edb700 1 -- 172.21.15.178:0/611763590 shutdown_connections 2024-02-11T17:10:36.438 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:36.417+0000 7fbea1edb700 1 -- 172.21.15.178:0/611763590 wait complete. 2024-02-11T17:10:36.438 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:36.417+0000 7fbea1edb700 1 Processor -- start 2024-02-11T17:10:36.438 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:36.417+0000 7fbea1edb700 1 -- start start 2024-02-11T17:10:36.438 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:36.418+0000 7fbea1edb700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbe9c0601e0 0x7fbe9c106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:10:36.438 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:36.418+0000 7fbea1edb700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9c1069f0 con 0x7fbe9c0601e0 2024-02-11T17:10:36.438 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:36.418+0000 7fbea0ed9700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbe9c0601e0 0x7fbe9c106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:10:36.438 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:36.418+0000 7fbea0ed9700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbe9c0601e0 0x7fbe9c106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:34374/0 (socket says 172.21.15.178:34374) 2024-02-11T17:10:36.438 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:36.418+0000 7fbea0ed9700 1 -- 172.21.15.178:0/4285321390 learned_addr learned my addr 172.21.15.178:0/4285321390 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:10:36.438 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:36.418+0000 7fbea0ed9700 1 -- 172.21.15.178:0/4285321390 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe90005040 con 0x7fbe9c0601e0 2024-02-11T17:10:36.438 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:36.418+0000 7fbea0ed9700 1 --2- 172.21.15.178:0/4285321390 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbe9c0601e0 0x7fbe9c106500 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7fbe90013510 tx=0x7fbe9000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:10:36.438 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:36.418+0000 7fbe997fa700 1 -- 172.21.15.178:0/4285321390 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe90009320 con 0x7fbe9c0601e0 2024-02-11T17:10:36.438 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:36.419+0000 7fbea1edb700 1 -- 172.21.15.178:0/4285321390 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe9c106bf0 con 0x7fbe9c0601e0 2024-02-11T17:10:36.438 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:36.419+0000 7fbe997fa700 1 -- 172.21.15.178:0/4285321390 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbe90013f00 con 0x7fbe9c0601e0 2024-02-11T17:10:36.439 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:36.419+0000 7fbe997fa700 1 -- 172.21.15.178:0/4285321390 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe90014d90 con 0x7fbe9c0601e0 2024-02-11T17:10:36.439 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:36.419+0000 7fbea1edb700 1 -- 172.21.15.178:0/4285321390 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe9c107080 con 0x7fbe9c0601e0 2024-02-11T17:10:36.439 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:36.419+0000 7fbe997fa700 1 -- 172.21.15.178:0/4285321390 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbe900095f0 con 0x7fbe9c0601e0 2024-02-11T17:10:36.439 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:36.419+0000 7fbea1edb700 1 -- 172.21.15.178:0/4285321390 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe9c045ad0 con 0x7fbe9c0601e0 2024-02-11T17:10:36.439 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:36.419+0000 7fbe997fa700 1 --2- 172.21.15.178:0/4285321390 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fbe8c02ff60 0x7fbe8c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:10:36.439 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:36.420+0000 7fbe997fa700 1 -- 172.21.15.178:0/4285321390 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbe900410a0 con 0x7fbe9c0601e0 2024-02-11T17:10:36.439 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:36.420+0000 7fbe9bfff700 1 --2- 172.21.15.178:0/4285321390 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fbe8c02ff60 0x7fbe8c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:10:36.439 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:36.420+0000 7fbe9bfff700 1 --2- 172.21.15.178:0/4285321390 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fbe8c02ff60 0x7fbe8c032420 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fbe880096a0 tx=0x7fbe88006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:10:36.439 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:36.425+0000 7fbe997fa700 1 -- 172.21.15.178:0/4285321390 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbe9001d7f0 con 0x7fbe9c0601e0 2024-02-11T17:10:36.628 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:36.626+0000 7fbea1edb700 1 -- 172.21.15.178:0/4285321390 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbe9c107f30 con 0x7fbe9c0601e0 2024-02-11T17:10:36.628 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:36.627+0000 7fbe997fa700 1 -- 172.21.15.178:0/4285321390 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbe90010230 con 0x7fbe9c0601e0 2024-02-11T17:10:36.629 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:10:36.629 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:10:36.630 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:36.628+0000 7fbea1edb700 1 -- 172.21.15.178:0/4285321390 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fbe8c02ff60 msgr2=0x7fbe8c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:10:36.630 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:36.628+0000 7fbea1edb700 1 --2- 172.21.15.178:0/4285321390 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fbe8c02ff60 0x7fbe8c032420 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fbe880096a0 tx=0x7fbe88006b40).stop 2024-02-11T17:10:36.630 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:36.628+0000 7fbea1edb700 1 -- 172.21.15.178:0/4285321390 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbe9c0601e0 msgr2=0x7fbe9c106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:10:36.630 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:36.628+0000 7fbea1edb700 1 --2- 172.21.15.178:0/4285321390 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbe9c0601e0 0x7fbe9c106500 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7fbe90013510 tx=0x7fbe9000b040).stop 2024-02-11T17:10:36.630 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:36.628+0000 7fbea1edb700 1 -- 172.21.15.178:0/4285321390 shutdown_connections 2024-02-11T17:10:36.630 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:36.628+0000 7fbea1edb700 1 --2- 172.21.15.178:0/4285321390 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fbe8c02ff60 0x7fbe8c032420 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:10:36.630 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:36.628+0000 7fbea1edb700 1 --2- 172.21.15.178:0/4285321390 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbe9c0601e0 0x7fbe9c106500 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:10:36.630 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:36.628+0000 7fbea1edb700 1 -- 172.21.15.178:0/4285321390 >> 172.21.15.178:0/4285321390 conn(0x7fbe9c0f09a0 msgr2=0x7fbe9c0f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:10:36.630 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:36.628+0000 7fbea1edb700 1 -- 172.21.15.178:0/4285321390 shutdown_connections 2024-02-11T17:10:36.630 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:36.628+0000 7fbea1edb700 1 -- 172.21.15.178:0/4285321390 wait complete. 2024-02-11T17:10:36.631 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:10:37.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:36 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:10:36.628041+0000 mon.smithi037 (mon.0) 311 : audit [DBG] from='client.? 172.21.15.178:0/4285321390' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:10:38.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:37 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:10:36.744753+0000 mgr.smithi037.eofnaq (mgr.14162) 327 : cluster [DBG] pgmap v311: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:10:38.187 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:10:38.187 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:10:40.142 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:40.139+0000 7f9930e44700 1 -- 172.21.15.178:0/256268813 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f992c0601e0 msgr2=0x7f992c0f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:10:40.142 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:40.139+0000 7f9930e44700 1 --2- 172.21.15.178:0/256268813 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f992c0601e0 0x7f992c0f8f20 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7f991c009fb0 tx=0x7f991c009b30).stop 2024-02-11T17:10:40.142 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:40.139+0000 7f9930e44700 1 -- 172.21.15.178:0/256268813 shutdown_connections 2024-02-11T17:10:40.143 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:40.139+0000 7f9930e44700 1 --2- 172.21.15.178:0/256268813 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f992c0601e0 0x7f992c0f8f20 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:10:40.143 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:40.139+0000 7f9930e44700 1 -- 172.21.15.178:0/256268813 >> 172.21.15.178:0/256268813 conn(0x7f992c0f09a0 msgr2=0x7f992c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:10:40.143 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:40.139+0000 7f9930e44700 1 -- 172.21.15.178:0/256268813 shutdown_connections 2024-02-11T17:10:40.143 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:40.139+0000 7f9930e44700 1 -- 172.21.15.178:0/256268813 wait complete. 2024-02-11T17:10:40.143 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:40.139+0000 7f9930e44700 1 Processor -- start 2024-02-11T17:10:40.143 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:40.140+0000 7f9930e44700 1 -- start start 2024-02-11T17:10:40.143 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:40.140+0000 7f9930e44700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f992c0601e0 0x7f992c0f69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:10:40.143 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:40.140+0000 7f9930e44700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f992c0f4e20 con 0x7f992c0601e0 2024-02-11T17:10:40.143 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:40.140+0000 7f992b7fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f992c0601e0 0x7f992c0f69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:10:40.143 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:40.140+0000 7f992b7fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f992c0601e0 0x7f992c0f69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:44558/0 (socket says 172.21.15.178:44558) 2024-02-11T17:10:40.143 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:40.140+0000 7f992b7fe700 1 -- 172.21.15.178:0/2878277431 learned_addr learned my addr 172.21.15.178:0/2878277431 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:10:40.143 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:40.140+0000 7f992b7fe700 1 -- 172.21.15.178:0/2878277431 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f991c005040 con 0x7f992c0601e0 2024-02-11T17:10:40.143 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:40.140+0000 7f992b7fe700 1 --2- 172.21.15.178:0/2878277431 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f992c0601e0 0x7f992c0f69a0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f991c013510 tx=0x7f991c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:10:40.144 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:40.141+0000 7f9913fff700 1 -- 172.21.15.178:0/2878277431 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f991c009320 con 0x7f992c0601e0 2024-02-11T17:10:40.144 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:40.141+0000 7f9930e44700 1 -- 172.21.15.178:0/2878277431 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f992c0f4fc0 con 0x7f992c0601e0 2024-02-11T17:10:40.144 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:40.141+0000 7f9913fff700 1 -- 172.21.15.178:0/2878277431 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f991c013f00 con 0x7f992c0601e0 2024-02-11T17:10:40.144 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:40.141+0000 7f9913fff700 1 -- 172.21.15.178:0/2878277431 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f991c014d90 con 0x7f992c0601e0 2024-02-11T17:10:40.144 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:40.141+0000 7f9930e44700 1 -- 172.21.15.178:0/2878277431 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f992c0f5450 con 0x7f992c0601e0 2024-02-11T17:10:40.145 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:40.141+0000 7f9913fff700 1 -- 172.21.15.178:0/2878277431 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f991c0095f0 con 0x7f992c0601e0 2024-02-11T17:10:40.145 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:40.141+0000 7f9930e44700 1 -- 172.21.15.178:0/2878277431 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f992c059100 con 0x7f992c0601e0 2024-02-11T17:10:40.145 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:40.142+0000 7f9913fff700 1 --2- 172.21.15.178:0/2878277431 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f991402ff60 0x7f9914032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:10:40.145 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:40.142+0000 7f9913fff700 1 -- 172.21.15.178:0/2878277431 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f991c041080 con 0x7f992c0601e0 2024-02-11T17:10:40.145 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:40.142+0000 7f992affd700 1 --2- 172.21.15.178:0/2878277431 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f991402ff60 0x7f9914032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:10:40.145 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:40.142+0000 7f992affd700 1 --2- 172.21.15.178:0/2878277431 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f991402ff60 0x7f9914032420 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f99200096a0 tx=0x7f9920006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:10:40.148 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:40.146+0000 7f9913fff700 1 -- 172.21.15.178:0/2878277431 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f991c014420 con 0x7f992c0601e0 2024-02-11T17:10:40.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:39 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:10:38.745172+0000 mgr.smithi037.eofnaq (mgr.14162) 328 : cluster [DBG] pgmap v312: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:10:40.351 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:40.350+0000 7f9930e44700 1 -- 172.21.15.178:0/2878277431 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f992c0fbcd0 con 0x7f992c0601e0 2024-02-11T17:10:40.352 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:40.351+0000 7f9913fff700 1 -- 172.21.15.178:0/2878277431 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f991c010230 con 0x7f992c0601e0 2024-02-11T17:10:40.353 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:10:40.353 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:10:40.354 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:40.352+0000 7f9930e44700 1 -- 172.21.15.178:0/2878277431 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f991402ff60 msgr2=0x7f9914032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:10:40.354 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:40.352+0000 7f9930e44700 1 --2- 172.21.15.178:0/2878277431 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f991402ff60 0x7f9914032420 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f99200096a0 tx=0x7f9920006b40).stop 2024-02-11T17:10:40.354 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:40.352+0000 7f9930e44700 1 -- 172.21.15.178:0/2878277431 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f992c0601e0 msgr2=0x7f992c0f69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:10:40.354 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:40.352+0000 7f9930e44700 1 --2- 172.21.15.178:0/2878277431 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f992c0601e0 0x7f992c0f69a0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f991c013510 tx=0x7f991c00b040).stop 2024-02-11T17:10:40.354 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:40.352+0000 7f9930e44700 1 -- 172.21.15.178:0/2878277431 shutdown_connections 2024-02-11T17:10:40.354 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:40.352+0000 7f9930e44700 1 --2- 172.21.15.178:0/2878277431 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f991402ff60 0x7f9914032420 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:10:40.354 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:40.352+0000 7f9930e44700 1 --2- 172.21.15.178:0/2878277431 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f992c0601e0 0x7f992c0f69a0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:10:40.355 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:40.352+0000 7f9930e44700 1 -- 172.21.15.178:0/2878277431 >> 172.21.15.178:0/2878277431 conn(0x7f992c0f09a0 msgr2=0x7f992c0f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:10:40.355 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:40.352+0000 7f9930e44700 1 -- 172.21.15.178:0/2878277431 shutdown_connections 2024-02-11T17:10:40.355 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:40.352+0000 7f9930e44700 1 -- 172.21.15.178:0/2878277431 wait complete. 2024-02-11T17:10:40.355 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:10:41.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:41 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:10:40.351965+0000 mon.smithi037 (mon.0) 312 : audit [DBG] from='client.? 172.21.15.178:0/2878277431' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:10:41.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:41 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:10:40.745567+0000 mgr.smithi037.eofnaq (mgr.14162) 329 : cluster [DBG] pgmap v313: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:10:42.002 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:10:42.002 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:10:43.845 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:43.842+0000 7fd064b2f700 1 -- 172.21.15.178:0/3981678397 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd060060280 msgr2=0x7fd0600f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:10:43.845 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:43.842+0000 7fd064b2f700 1 --2- 172.21.15.178:0/3981678397 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd060060280 0x7fd0600f90f0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7fd054009fb0 tx=0x7fd054009b30).stop 2024-02-11T17:10:43.845 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:43.842+0000 7fd064b2f700 1 -- 172.21.15.178:0/3981678397 shutdown_connections 2024-02-11T17:10:43.845 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:43.842+0000 7fd064b2f700 1 --2- 172.21.15.178:0/3981678397 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd060060280 0x7fd0600f90f0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:10:43.845 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:43.842+0000 7fd064b2f700 1 -- 172.21.15.178:0/3981678397 >> 172.21.15.178:0/3981678397 conn(0x7fd0600f0b70 msgr2=0x7fd0600f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:10:43.845 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:43.842+0000 7fd064b2f700 1 -- 172.21.15.178:0/3981678397 shutdown_connections 2024-02-11T17:10:43.845 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:43.842+0000 7fd064b2f700 1 -- 172.21.15.178:0/3981678397 wait complete. 2024-02-11T17:10:43.845 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:43.843+0000 7fd064b2f700 1 Processor -- start 2024-02-11T17:10:43.845 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:43.843+0000 7fd064b2f700 1 -- start start 2024-02-11T17:10:43.845 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:43.843+0000 7fd064b2f700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd060060280 0x7fd0601066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:10:43.845 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:43.843+0000 7fd064b2f700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd060106bd0 con 0x7fd060060280 2024-02-11T17:10:43.845 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:43.843+0000 7fd05e59c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd060060280 0x7fd0601066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:10:43.846 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:43.843+0000 7fd05e59c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd060060280 0x7fd0601066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:44584/0 (socket says 172.21.15.178:44584) 2024-02-11T17:10:43.846 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:43.843+0000 7fd05e59c700 1 -- 172.21.15.178:0/1520651620 learned_addr learned my addr 172.21.15.178:0/1520651620 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:10:43.847 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:43.843+0000 7fd05e59c700 1 -- 172.21.15.178:0/1520651620 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd054005040 con 0x7fd060060280 2024-02-11T17:10:43.847 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:43.844+0000 7fd05e59c700 1 --2- 172.21.15.178:0/1520651620 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd060060280 0x7fd0601066e0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7fd054013510 tx=0x7fd05400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:10:43.847 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:43.844+0000 7fd04effd700 1 -- 172.21.15.178:0/1520651620 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd054009320 con 0x7fd060060280 2024-02-11T17:10:43.847 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:43.844+0000 7fd064b2f700 1 -- 172.21.15.178:0/1520651620 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd060106dd0 con 0x7fd060060280 2024-02-11T17:10:43.847 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:43.844+0000 7fd04effd700 1 -- 172.21.15.178:0/1520651620 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd054013f00 con 0x7fd060060280 2024-02-11T17:10:43.847 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:43.844+0000 7fd064b2f700 1 -- 172.21.15.178:0/1520651620 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd060107260 con 0x7fd060060280 2024-02-11T17:10:43.847 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:43.844+0000 7fd04effd700 1 -- 172.21.15.178:0/1520651620 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd054014dc0 con 0x7fd060060280 2024-02-11T17:10:43.847 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:43.845+0000 7fd064b2f700 1 -- 172.21.15.178:0/1520651620 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd060045ad0 con 0x7fd060060280 2024-02-11T17:10:43.847 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:43.845+0000 7fd04effd700 1 -- 172.21.15.178:0/1520651620 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd0540095f0 con 0x7fd060060280 2024-02-11T17:10:43.847 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:43.845+0000 7fd04effd700 1 --2- 172.21.15.178:0/1520651620 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fd04802ff60 0x7fd048032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:10:43.847 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:43.845+0000 7fd04effd700 1 -- 172.21.15.178:0/1520651620 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd0540410f0 con 0x7fd060060280 2024-02-11T17:10:43.847 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:43.845+0000 7fd05dd9b700 1 --2- 172.21.15.178:0/1520651620 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fd04802ff60 0x7fd048032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:10:43.848 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:43.845+0000 7fd05dd9b700 1 --2- 172.21.15.178:0/1520651620 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fd04802ff60 0x7fd048032420 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fd0500096a0 tx=0x7fd050006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:10:43.851 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:43.850+0000 7fd04effd700 1 -- 172.21.15.178:0/1520651620 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd05401c790 con 0x7fd060060280 2024-02-11T17:10:44.055 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:44.054+0000 7fd064b2f700 1 -- 172.21.15.178:0/1520651620 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd0601080b0 con 0x7fd060060280 2024-02-11T17:10:44.056 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:44.054+0000 7fd04effd700 1 -- 172.21.15.178:0/1520651620 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd054010230 con 0x7fd060060280 2024-02-11T17:10:44.056 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:10:44.056 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:10:44.057 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:44.056+0000 7fd064b2f700 1 -- 172.21.15.178:0/1520651620 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fd04802ff60 msgr2=0x7fd048032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:10:44.057 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:44.056+0000 7fd064b2f700 1 --2- 172.21.15.178:0/1520651620 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fd04802ff60 0x7fd048032420 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fd0500096a0 tx=0x7fd050006b40).stop 2024-02-11T17:10:44.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:44.056+0000 7fd064b2f700 1 -- 172.21.15.178:0/1520651620 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd060060280 msgr2=0x7fd0601066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:10:44.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:44.056+0000 7fd064b2f700 1 --2- 172.21.15.178:0/1520651620 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd060060280 0x7fd0601066e0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7fd054013510 tx=0x7fd05400b040).stop 2024-02-11T17:10:44.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:44.056+0000 7fd064b2f700 1 -- 172.21.15.178:0/1520651620 shutdown_connections 2024-02-11T17:10:44.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:44.056+0000 7fd064b2f700 1 --2- 172.21.15.178:0/1520651620 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fd04802ff60 0x7fd048032420 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:10:44.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:44.056+0000 7fd064b2f700 1 --2- 172.21.15.178:0/1520651620 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd060060280 0x7fd0601066e0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:10:44.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:44.056+0000 7fd064b2f700 1 -- 172.21.15.178:0/1520651620 >> 172.21.15.178:0/1520651620 conn(0x7fd0600f0b70 msgr2=0x7fd0600f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:10:44.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:44.056+0000 7fd064b2f700 1 -- 172.21.15.178:0/1520651620 shutdown_connections 2024-02-11T17:10:44.058 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:44.056+0000 7fd064b2f700 1 -- 172.21.15.178:0/1520651620 wait complete. 2024-02-11T17:10:44.058 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:10:44.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:43 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:10:42.745808+0000 mgr.smithi037.eofnaq (mgr.14162) 330 : cluster [DBG] pgmap v314: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:10:45.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:44 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:10:44.055592+0000 mon.smithi037 (mon.0) 313 : audit [DBG] from='client.? 172.21.15.178:0/1520651620' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:10:45.769 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:10:45.770 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:10:46.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:45 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:10:44.746112+0000 mgr.smithi037.eofnaq (mgr.14162) 331 : cluster [DBG] pgmap v315: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:10:47.806 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:47.803+0000 7f1921f94700 1 -- 172.21.15.178:0/2114564005 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f191c0f6510 msgr2=0x7f191c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:10:47.807 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:47.803+0000 7f1921f94700 1 --2- 172.21.15.178:0/2114564005 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f191c0f6510 0x7f191c0f6930 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7f1910009fb0 tx=0x7f1910009b30).stop 2024-02-11T17:10:47.807 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:47.803+0000 7f1921f94700 1 -- 172.21.15.178:0/2114564005 shutdown_connections 2024-02-11T17:10:47.807 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:47.803+0000 7f1921f94700 1 --2- 172.21.15.178:0/2114564005 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f191c0f6510 0x7f191c0f6930 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:10:47.807 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:47.803+0000 7f1921f94700 1 -- 172.21.15.178:0/2114564005 >> 172.21.15.178:0/2114564005 conn(0x7f191c0f1c70 msgr2=0x7f191c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:10:47.807 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:47.803+0000 7f1921f94700 1 -- 172.21.15.178:0/2114564005 shutdown_connections 2024-02-11T17:10:47.807 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:47.803+0000 7f1921f94700 1 -- 172.21.15.178:0/2114564005 wait complete. 2024-02-11T17:10:47.807 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:47.804+0000 7f1921f94700 1 Processor -- start 2024-02-11T17:10:47.807 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:47.804+0000 7f1921f94700 1 -- start start 2024-02-11T17:10:47.807 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:47.804+0000 7f1921f94700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f191c0f6510 0x7f191c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:10:47.807 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:47.804+0000 7f1921f94700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f191c106c20 con 0x7f191c0f6510 2024-02-11T17:10:47.807 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:47.805+0000 7f191b7fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f191c0f6510 0x7f191c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:10:47.807 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:47.805+0000 7f191b7fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f191c0f6510 0x7f191c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:44592/0 (socket says 172.21.15.178:44592) 2024-02-11T17:10:47.807 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:47.805+0000 7f191b7fe700 1 -- 172.21.15.178:0/4221877744 learned_addr learned my addr 172.21.15.178:0/4221877744 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:10:47.808 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:47.805+0000 7f191b7fe700 1 -- 172.21.15.178:0/4221877744 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1910005040 con 0x7f191c0f6510 2024-02-11T17:10:47.808 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:47.805+0000 7f191b7fe700 1 --2- 172.21.15.178:0/4221877744 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f191c0f6510 0x7f191c106730 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f1910013510 tx=0x7f191000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:10:47.808 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:47.806+0000 7f1903fff700 1 -- 172.21.15.178:0/4221877744 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1910009320 con 0x7f191c0f6510 2024-02-11T17:10:47.808 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:47.807+0000 7f1903fff700 1 -- 172.21.15.178:0/4221877744 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1910013f00 con 0x7f191c0f6510 2024-02-11T17:10:47.808 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:47.807+0000 7f1921f94700 1 -- 172.21.15.178:0/4221877744 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f191c106e20 con 0x7f191c0f6510 2024-02-11T17:10:47.809 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:47.807+0000 7f1903fff700 1 -- 172.21.15.178:0/4221877744 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1910014d90 con 0x7f191c0f6510 2024-02-11T17:10:47.809 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:47.807+0000 7f1921f94700 1 -- 172.21.15.178:0/4221877744 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f191c1072b0 con 0x7f191c0f6510 2024-02-11T17:10:47.809 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:47.807+0000 7f1903fff700 1 -- 172.21.15.178:0/4221877744 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1910014420 con 0x7f191c0f6510 2024-02-11T17:10:47.809 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:47.807+0000 7f1921f94700 1 -- 172.21.15.178:0/4221877744 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f191c045ad0 con 0x7f191c0f6510 2024-02-11T17:10:47.809 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:47.808+0000 7f1903fff700 1 --2- 172.21.15.178:0/4221877744 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f190402ffb0 0x7f1904032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:10:47.810 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:47.808+0000 7f1903fff700 1 -- 172.21.15.178:0/4221877744 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1910040ff0 con 0x7f191c0f6510 2024-02-11T17:10:47.810 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:47.808+0000 7f191affd700 1 --2- 172.21.15.178:0/4221877744 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f190402ffb0 0x7f1904032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:10:47.810 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:47.808+0000 7f191affd700 1 --2- 172.21.15.178:0/4221877744 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f190402ffb0 0x7f1904032470 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f190c0096a0 tx=0x7f190c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:10:47.814 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:47.812+0000 7f1903fff700 1 -- 172.21.15.178:0/4221877744 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f191006d2b0 con 0x7f191c0f6510 2024-02-11T17:10:48.019 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:48.017+0000 7f1921f94700 1 -- 172.21.15.178:0/4221877744 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f191c0fbf60 con 0x7f191c0f6510 2024-02-11T17:10:48.020 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:48.018+0000 7f1903fff700 1 -- 172.21.15.178:0/4221877744 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f19100095f0 con 0x7f191c0f6510 2024-02-11T17:10:48.021 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:10:48.021 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:10:48.022 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:48.020+0000 7f1921f94700 1 -- 172.21.15.178:0/4221877744 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f190402ffb0 msgr2=0x7f1904032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:10:48.022 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:48.020+0000 7f1921f94700 1 --2- 172.21.15.178:0/4221877744 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f190402ffb0 0x7f1904032470 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f190c0096a0 tx=0x7f190c006b40).stop 2024-02-11T17:10:48.022 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:48.020+0000 7f1921f94700 1 -- 172.21.15.178:0/4221877744 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f191c0f6510 msgr2=0x7f191c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:10:48.022 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:48.020+0000 7f1921f94700 1 --2- 172.21.15.178:0/4221877744 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f191c0f6510 0x7f191c106730 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f1910013510 tx=0x7f191000b040).stop 2024-02-11T17:10:48.022 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:48.020+0000 7f1921f94700 1 -- 172.21.15.178:0/4221877744 shutdown_connections 2024-02-11T17:10:48.023 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:48.020+0000 7f1921f94700 1 --2- 172.21.15.178:0/4221877744 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f190402ffb0 0x7f1904032470 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:10:48.023 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:48.020+0000 7f1921f94700 1 --2- 172.21.15.178:0/4221877744 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f191c0f6510 0x7f191c106730 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:10:48.023 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:48.020+0000 7f1921f94700 1 -- 172.21.15.178:0/4221877744 >> 172.21.15.178:0/4221877744 conn(0x7f191c0f1c70 msgr2=0x7f191c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:10:48.023 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:48.020+0000 7f1921f94700 1 -- 172.21.15.178:0/4221877744 shutdown_connections 2024-02-11T17:10:48.023 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:48.020+0000 7f1921f94700 1 -- 172.21.15.178:0/4221877744 wait complete. 2024-02-11T17:10:48.023 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:10:48.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:47 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:10:46.746368+0000 mgr.smithi037.eofnaq (mgr.14162) 332 : cluster [DBG] pgmap v316: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:10:49.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:48 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:10:48.019670+0000 mon.smithi037 (mon.0) 314 : audit [DBG] from='client.? 172.21.15.178:0/4221877744' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:10:49.691 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:10:49.691 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:10:50.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:49 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:10:48.746650+0000 mgr.smithi037.eofnaq (mgr.14162) 333 : cluster [DBG] pgmap v317: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:10:51.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:51 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:10:50.746900+0000 mgr.smithi037.eofnaq (mgr.14162) 334 : cluster [DBG] pgmap v318: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:10:51.546 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:51.502+0000 7fb3b759e700 1 -- 172.21.15.178:0/2843933577 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3b80f62a0 msgr2=0x7fb3b80f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:10:51.546 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:51.502+0000 7fb3b759e700 1 --2- 172.21.15.178:0/2843933577 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3b80f62a0 0x7fb3b80f66c0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7fb3ac009fb0 tx=0x7fb3ac009b30).stop 2024-02-11T17:10:51.546 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:51.502+0000 7fb3b759e700 1 -- 172.21.15.178:0/2843933577 shutdown_connections 2024-02-11T17:10:51.546 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:51.502+0000 7fb3b759e700 1 --2- 172.21.15.178:0/2843933577 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3b80f62a0 0x7fb3b80f66c0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:10:51.546 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:51.502+0000 7fb3b759e700 1 -- 172.21.15.178:0/2843933577 >> 172.21.15.178:0/2843933577 conn(0x7fb3b80f1aa0 msgr2=0x7fb3b80f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:10:51.547 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:51.502+0000 7fb3b759e700 1 -- 172.21.15.178:0/2843933577 shutdown_connections 2024-02-11T17:10:51.547 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:51.502+0000 7fb3b759e700 1 -- 172.21.15.178:0/2843933577 wait complete. 2024-02-11T17:10:51.547 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:51.502+0000 7fb3b759e700 1 Processor -- start 2024-02-11T17:10:51.547 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:51.503+0000 7fb3b759e700 1 -- start start 2024-02-11T17:10:51.547 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:51.503+0000 7fb3b759e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3b80f62a0 0x7fb3b81064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:10:51.547 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:51.503+0000 7fb3b759e700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b81069a0 con 0x7fb3b80f62a0 2024-02-11T17:10:51.547 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:51.503+0000 7fb3b659c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3b80f62a0 0x7fb3b81064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:10:51.547 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:51.503+0000 7fb3b659c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3b80f62a0 0x7fb3b81064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:41592/0 (socket says 172.21.15.178:41592) 2024-02-11T17:10:51.547 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:51.503+0000 7fb3b659c700 1 -- 172.21.15.178:0/990249993 learned_addr learned my addr 172.21.15.178:0/990249993 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:10:51.547 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:51.503+0000 7fb3b659c700 1 -- 172.21.15.178:0/990249993 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3ac005040 con 0x7fb3b80f62a0 2024-02-11T17:10:51.547 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:51.503+0000 7fb3b659c700 1 --2- 172.21.15.178:0/990249993 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3b80f62a0 0x7fb3b81064b0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7fb3ac013510 tx=0x7fb3ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:10:51.547 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:51.504+0000 7fb3a6ffd700 1 -- 172.21.15.178:0/990249993 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3ac009320 con 0x7fb3b80f62a0 2024-02-11T17:10:51.547 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:51.504+0000 7fb3b759e700 1 -- 172.21.15.178:0/990249993 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3b8106ba0 con 0x7fb3b80f62a0 2024-02-11T17:10:51.547 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:51.504+0000 7fb3a6ffd700 1 -- 172.21.15.178:0/990249993 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb3ac013f00 con 0x7fb3b80f62a0 2024-02-11T17:10:51.547 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:51.504+0000 7fb3a6ffd700 1 -- 172.21.15.178:0/990249993 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3ac014dc0 con 0x7fb3b80f62a0 2024-02-11T17:10:51.548 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:51.504+0000 7fb3b759e700 1 -- 172.21.15.178:0/990249993 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3b8107030 con 0x7fb3b80f62a0 2024-02-11T17:10:51.548 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:51.504+0000 7fb3a6ffd700 1 -- 172.21.15.178:0/990249993 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb3ac0095f0 con 0x7fb3b80f62a0 2024-02-11T17:10:51.548 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:51.504+0000 7fb3b759e700 1 -- 172.21.15.178:0/990249993 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3b8045ad0 con 0x7fb3b80f62a0 2024-02-11T17:10:51.548 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:51.505+0000 7fb3a6ffd700 1 --2- 172.21.15.178:0/990249993 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb3a002ff60 0x7fb3a0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:10:51.548 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:51.505+0000 7fb3a6ffd700 1 -- 172.21.15.178:0/990249993 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb3ac041030 con 0x7fb3b80f62a0 2024-02-11T17:10:51.548 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:51.505+0000 7fb3b5d9b700 1 --2- 172.21.15.178:0/990249993 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb3a002ff60 0x7fb3a0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:10:51.548 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:51.505+0000 7fb3b5d9b700 1 --2- 172.21.15.178:0/990249993 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb3a002ff60 0x7fb3a0032420 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fb3a80096a0 tx=0x7fb3a8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:10:51.548 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:51.509+0000 7fb3a6ffd700 1 -- 172.21.15.178:0/990249993 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb3ac014420 con 0x7fb3b80f62a0 2024-02-11T17:10:51.717 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:51.715+0000 7fb3b759e700 1 -- 172.21.15.178:0/990249993 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb3b8107ee0 con 0x7fb3b80f62a0 2024-02-11T17:10:51.718 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:51.716+0000 7fb3a6ffd700 1 -- 172.21.15.178:0/990249993 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb3ac010230 con 0x7fb3b80f62a0 2024-02-11T17:10:51.718 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:10:51.718 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:10:51.720 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:51.718+0000 7fb3b759e700 1 -- 172.21.15.178:0/990249993 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb3a002ff60 msgr2=0x7fb3a0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:10:51.720 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:51.718+0000 7fb3b759e700 1 --2- 172.21.15.178:0/990249993 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb3a002ff60 0x7fb3a0032420 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fb3a80096a0 tx=0x7fb3a8006b40).stop 2024-02-11T17:10:51.720 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:51.718+0000 7fb3b759e700 1 -- 172.21.15.178:0/990249993 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3b80f62a0 msgr2=0x7fb3b81064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:10:51.720 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:51.718+0000 7fb3b759e700 1 --2- 172.21.15.178:0/990249993 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3b80f62a0 0x7fb3b81064b0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7fb3ac013510 tx=0x7fb3ac00b040).stop 2024-02-11T17:10:51.720 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:51.718+0000 7fb3b759e700 1 -- 172.21.15.178:0/990249993 shutdown_connections 2024-02-11T17:10:51.720 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:51.718+0000 7fb3b759e700 1 --2- 172.21.15.178:0/990249993 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fb3a002ff60 0x7fb3a0032420 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:10:51.720 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:51.718+0000 7fb3b759e700 1 --2- 172.21.15.178:0/990249993 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb3b80f62a0 0x7fb3b81064b0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:10:51.720 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:51.718+0000 7fb3b759e700 1 -- 172.21.15.178:0/990249993 >> 172.21.15.178:0/990249993 conn(0x7fb3b80f1aa0 msgr2=0x7fb3b80f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:10:51.721 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:51.718+0000 7fb3b759e700 1 -- 172.21.15.178:0/990249993 shutdown_connections 2024-02-11T17:10:51.721 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:51.718+0000 7fb3b759e700 1 -- 172.21.15.178:0/990249993 wait complete. 2024-02-11T17:10:51.721 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:10:52.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:52 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:10:51.717632+0000 mon.smithi037 (mon.0) 315 : audit [DBG] from='client.? 172.21.15.178:0/990249993' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:10:53.394 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:10:53.394 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:10:53.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:53 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:10:52.747158+0000 mgr.smithi037.eofnaq (mgr.14162) 335 : cluster [DBG] pgmap v319: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:10:55.509 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:55.506+0000 7f54fb86f700 1 -- 172.21.15.178:0/1461858909 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f54f40f5410 msgr2=0x7f54f40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:10:55.510 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:55.506+0000 7f54fb86f700 1 --2- 172.21.15.178:0/1461858909 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f54f40f5410 0x7f54f40f5830 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f54f0009fb0 tx=0x7f54f0009b30).stop 2024-02-11T17:10:55.510 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:55.506+0000 7f54fb86f700 1 -- 172.21.15.178:0/1461858909 shutdown_connections 2024-02-11T17:10:55.510 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:55.506+0000 7f54fb86f700 1 --2- 172.21.15.178:0/1461858909 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f54f40f5410 0x7f54f40f5830 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:10:55.510 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:55.506+0000 7f54fb86f700 1 -- 172.21.15.178:0/1461858909 >> 172.21.15.178:0/1461858909 conn(0x7f54f40f0b70 msgr2=0x7f54f40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:10:55.510 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:55.506+0000 7f54fb86f700 1 -- 172.21.15.178:0/1461858909 shutdown_connections 2024-02-11T17:10:55.510 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:55.506+0000 7f54fb86f700 1 -- 172.21.15.178:0/1461858909 wait complete. 2024-02-11T17:10:55.510 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:55.507+0000 7f54fb86f700 1 Processor -- start 2024-02-11T17:10:55.510 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:55.507+0000 7f54fb86f700 1 -- start start 2024-02-11T17:10:55.510 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:55.507+0000 7f54fb86f700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f54f40f5410 0x7f54f4108920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:10:55.510 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:55.507+0000 7f54fb86f700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54f4108e10 con 0x7f54f40f5410 2024-02-11T17:10:55.510 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:55.507+0000 7f54f960b700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f54f40f5410 0x7f54f4108920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:10:55.511 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:55.507+0000 7f54f960b700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f54f40f5410 0x7f54f4108920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:41618/0 (socket says 172.21.15.178:41618) 2024-02-11T17:10:55.511 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:55.507+0000 7f54f960b700 1 -- 172.21.15.178:0/917864823 learned_addr learned my addr 172.21.15.178:0/917864823 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:10:55.511 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:55.507+0000 7f54f960b700 1 -- 172.21.15.178:0/917864823 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54f0005040 con 0x7f54f40f5410 2024-02-11T17:10:55.511 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:55.507+0000 7f54f960b700 1 --2- 172.21.15.178:0/917864823 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f54f40f5410 0x7f54f4108920 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f54f0014000 tx=0x7f54f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:10:55.511 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:55.508+0000 7f54e9ffb700 1 -- 172.21.15.178:0/917864823 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f54f00095f0 con 0x7f54f40f5410 2024-02-11T17:10:55.511 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:55.508+0000 7f54fb86f700 1 -- 172.21.15.178:0/917864823 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54f4109010 con 0x7f54f40f5410 2024-02-11T17:10:55.511 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:55.508+0000 7f54e9ffb700 1 -- 172.21.15.178:0/917864823 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f54f0007e70 con 0x7f54f40f5410 2024-02-11T17:10:55.511 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:55.508+0000 7f54e9ffb700 1 -- 172.21.15.178:0/917864823 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f54f0014ea0 con 0x7f54f40f5410 2024-02-11T17:10:55.511 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:55.508+0000 7f54fb86f700 1 -- 172.21.15.178:0/917864823 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54f41094a0 con 0x7f54f40f5410 2024-02-11T17:10:55.512 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:55.509+0000 7f54fb86f700 1 -- 172.21.15.178:0/917864823 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54f4045ad0 con 0x7f54f40f5410 2024-02-11T17:10:55.512 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:55.509+0000 7f54e9ffb700 1 -- 172.21.15.178:0/917864823 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f54f0025070 con 0x7f54f40f5410 2024-02-11T17:10:55.512 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:55.509+0000 7f54e9ffb700 1 --2- 172.21.15.178:0/917864823 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f54e002ff60 0x7f54e0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:10:55.512 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:55.509+0000 7f54e9ffb700 1 -- 172.21.15.178:0/917864823 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f54f00409c0 con 0x7f54f40f5410 2024-02-11T17:10:55.512 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:55.509+0000 7f54f8e0a700 1 --2- 172.21.15.178:0/917864823 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f54e002ff60 0x7f54e0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:10:55.512 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:55.509+0000 7f54f8e0a700 1 --2- 172.21.15.178:0/917864823 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f54e002ff60 0x7f54e0032420 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f54e40096a0 tx=0x7f54e4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:10:55.515 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:55.513+0000 7f54e9ffb700 1 -- 172.21.15.178:0/917864823 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f54f00079c0 con 0x7f54f40f5410 2024-02-11T17:10:55.722 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:55.721+0000 7f54fb86f700 1 -- 172.21.15.178:0/917864823 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f54f410a2f0 con 0x7f54f40f5410 2024-02-11T17:10:55.723 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:55.721+0000 7f54e9ffb700 1 -- 172.21.15.178:0/917864823 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f54f000f300 con 0x7f54f40f5410 2024-02-11T17:10:55.723 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:10:55.723 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:10:55.725 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:55.723+0000 7f54fb86f700 1 -- 172.21.15.178:0/917864823 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f54e002ff60 msgr2=0x7f54e0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:10:55.725 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:55.723+0000 7f54fb86f700 1 --2- 172.21.15.178:0/917864823 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f54e002ff60 0x7f54e0032420 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f54e40096a0 tx=0x7f54e4006b40).stop 2024-02-11T17:10:55.725 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:55.723+0000 7f54fb86f700 1 -- 172.21.15.178:0/917864823 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f54f40f5410 msgr2=0x7f54f4108920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:10:55.725 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:55.723+0000 7f54fb86f700 1 --2- 172.21.15.178:0/917864823 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f54f40f5410 0x7f54f4108920 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f54f0014000 tx=0x7f54f000b040).stop 2024-02-11T17:10:55.725 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:55.723+0000 7f54fb86f700 1 -- 172.21.15.178:0/917864823 shutdown_connections 2024-02-11T17:10:55.725 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:55.723+0000 7f54fb86f700 1 --2- 172.21.15.178:0/917864823 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f54e002ff60 0x7f54e0032420 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:10:55.725 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:55.723+0000 7f54fb86f700 1 --2- 172.21.15.178:0/917864823 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f54f40f5410 0x7f54f4108920 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:10:55.725 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:55.723+0000 7f54fb86f700 1 -- 172.21.15.178:0/917864823 >> 172.21.15.178:0/917864823 conn(0x7f54f40f0b70 msgr2=0x7f54f405fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:10:55.725 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:55.723+0000 7f54fb86f700 1 -- 172.21.15.178:0/917864823 shutdown_connections 2024-02-11T17:10:55.725 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:55.723+0000 7f54fb86f700 1 -- 172.21.15.178:0/917864823 wait complete. 2024-02-11T17:10:55.726 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:10:56.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:55 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:10:54.747512+0000 mgr.smithi037.eofnaq (mgr.14162) 336 : cluster [DBG] pgmap v320: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:10:56.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:55 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:10:55.722764+0000 mon.smithi037 (mon.0) 316 : audit [DBG] from='client.? 172.21.15.178:0/917864823' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:10:57.564 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:10:57.564 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:10:58.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:57 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:10:56.747924+0000 mgr.smithi037.eofnaq (mgr.14162) 337 : cluster [DBG] pgmap v321: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:10:59.564 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:59.561+0000 7f65b3141700 1 -- 172.21.15.178:0/2322663854 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f65ac0f6510 msgr2=0x7f65ac0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:10:59.564 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:59.561+0000 7f65b3141700 1 --2- 172.21.15.178:0/2322663854 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f65ac0f6510 0x7f65ac0f6930 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7f65a0009fb0 tx=0x7f65a0009b30).stop 2024-02-11T17:10:59.564 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:59.561+0000 7f65b3141700 1 -- 172.21.15.178:0/2322663854 shutdown_connections 2024-02-11T17:10:59.564 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:59.561+0000 7f65b3141700 1 --2- 172.21.15.178:0/2322663854 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f65ac0f6510 0x7f65ac0f6930 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:10:59.564 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:59.561+0000 7f65b3141700 1 -- 172.21.15.178:0/2322663854 >> 172.21.15.178:0/2322663854 conn(0x7f65ac0f1c70 msgr2=0x7f65ac0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:10:59.564 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:59.561+0000 7f65b3141700 1 -- 172.21.15.178:0/2322663854 shutdown_connections 2024-02-11T17:10:59.564 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:59.561+0000 7f65b3141700 1 -- 172.21.15.178:0/2322663854 wait complete. 2024-02-11T17:10:59.565 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:59.562+0000 7f65b3141700 1 Processor -- start 2024-02-11T17:10:59.565 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:59.562+0000 7f65b3141700 1 -- start start 2024-02-11T17:10:59.565 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:59.562+0000 7f65b3141700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f65ac0f6510 0x7f65ac106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:10:59.565 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:59.562+0000 7f65b3141700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65ac106c20 con 0x7f65ac0f6510 2024-02-11T17:10:59.566 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:59.563+0000 7f65b0edd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f65ac0f6510 0x7f65ac106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:10:59.566 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:59.563+0000 7f65b0edd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f65ac0f6510 0x7f65ac106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:58176/0 (socket says 172.21.15.178:58176) 2024-02-11T17:10:59.567 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:59.563+0000 7f65b0edd700 1 -- 172.21.15.178:0/518009510 learned_addr learned my addr 172.21.15.178:0/518009510 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:10:59.567 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:59.563+0000 7f65b0edd700 1 -- 172.21.15.178:0/518009510 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65a0005040 con 0x7f65ac0f6510 2024-02-11T17:10:59.567 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:59.563+0000 7f65b0edd700 1 --2- 172.21.15.178:0/518009510 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f65ac0f6510 0x7f65ac106730 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f65a0013510 tx=0x7f65a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:10:59.567 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:59.563+0000 7f65a97fa700 1 -- 172.21.15.178:0/518009510 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f65a0009320 con 0x7f65ac0f6510 2024-02-11T17:10:59.567 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:59.563+0000 7f65b3141700 1 -- 172.21.15.178:0/518009510 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65ac106e20 con 0x7f65ac0f6510 2024-02-11T17:10:59.567 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:59.563+0000 7f65a97fa700 1 -- 172.21.15.178:0/518009510 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f65a0013f00 con 0x7f65ac0f6510 2024-02-11T17:10:59.567 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:59.563+0000 7f65a97fa700 1 -- 172.21.15.178:0/518009510 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f65a0014d90 con 0x7f65ac0f6510 2024-02-11T17:10:59.567 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:59.563+0000 7f65b3141700 1 -- 172.21.15.178:0/518009510 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65ac1072b0 con 0x7f65ac0f6510 2024-02-11T17:10:59.567 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:59.564+0000 7f65a97fa700 1 -- 172.21.15.178:0/518009510 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f65a0014420 con 0x7f65ac0f6510 2024-02-11T17:10:59.567 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:59.564+0000 7f65b3141700 1 -- 172.21.15.178:0/518009510 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65ac045ad0 con 0x7f65ac0f6510 2024-02-11T17:10:59.567 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:59.564+0000 7f65a97fa700 1 --2- 172.21.15.178:0/518009510 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f659c02ff60 0x7f659c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:10:59.567 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:59.564+0000 7f65a97fa700 1 -- 172.21.15.178:0/518009510 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f65a00410c0 con 0x7f65ac0f6510 2024-02-11T17:10:59.567 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:59.565+0000 7f65abfff700 1 --2- 172.21.15.178:0/518009510 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f659c02ff60 0x7f659c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:10:59.567 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:59.565+0000 7f65abfff700 1 --2- 172.21.15.178:0/518009510 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f659c02ff60 0x7f659c032420 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f65980096a0 tx=0x7f6598006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:10:59.570 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:59.569+0000 7f65a97fa700 1 -- 172.21.15.178:0/518009510 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f65a001db70 con 0x7f65ac0f6510 2024-02-11T17:10:59.778 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:59.776+0000 7f65b3141700 1 -- 172.21.15.178:0/518009510 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f65ac0fbf60 con 0x7f65ac0f6510 2024-02-11T17:10:59.779 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:59.777+0000 7f65a97fa700 1 -- 172.21.15.178:0/518009510 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f65a001db70 con 0x7f65ac0f6510 2024-02-11T17:10:59.779 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:10:59.779 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:10:59.780 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:59.779+0000 7f65b3141700 1 -- 172.21.15.178:0/518009510 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f659c02ff60 msgr2=0x7f659c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:10:59.780 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:59.779+0000 7f65b3141700 1 --2- 172.21.15.178:0/518009510 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f659c02ff60 0x7f659c032420 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f65980096a0 tx=0x7f6598006b40).stop 2024-02-11T17:10:59.781 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:59.779+0000 7f65b3141700 1 -- 172.21.15.178:0/518009510 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f65ac0f6510 msgr2=0x7f65ac106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:10:59.781 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:59.779+0000 7f65b3141700 1 --2- 172.21.15.178:0/518009510 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f65ac0f6510 0x7f65ac106730 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f65a0013510 tx=0x7f65a000b040).stop 2024-02-11T17:10:59.781 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:59.779+0000 7f65b3141700 1 -- 172.21.15.178:0/518009510 shutdown_connections 2024-02-11T17:10:59.781 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:59.779+0000 7f65b3141700 1 --2- 172.21.15.178:0/518009510 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f659c02ff60 0x7f659c032420 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:10:59.781 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:59.779+0000 7f65b3141700 1 --2- 172.21.15.178:0/518009510 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f65ac0f6510 0x7f65ac106730 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:10:59.781 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:59.779+0000 7f65b3141700 1 -- 172.21.15.178:0/518009510 >> 172.21.15.178:0/518009510 conn(0x7f65ac0f1c70 msgr2=0x7f65ac0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:10:59.781 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:59.779+0000 7f65b3141700 1 -- 172.21.15.178:0/518009510 shutdown_connections 2024-02-11T17:10:59.781 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:10:59.779+0000 7f65b3141700 1 -- 172.21.15.178:0/518009510 wait complete. 2024-02-11T17:10:59.781 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:11:00.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:59 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:10:58.748268+0000 mgr.smithi037.eofnaq (mgr.14162) 338 : cluster [DBG] pgmap v322: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:11:00.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:10:59 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:10:59.778463+0000 mon.smithi037 (mon.0) 317 : audit [DBG] from='client.? 172.21.15.178:0/518009510' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:11:01.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:11:01 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:11:00.748654+0000 mgr.smithi037.eofnaq (mgr.14162) 339 : cluster [DBG] pgmap v323: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:11:01.557 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:11:01.558 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:11:03.550 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:03.547+0000 7f9dc8a0e700 1 -- 172.21.15.178:0/259115113 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9dc40f62a0 msgr2=0x7f9dc40f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:11:03.550 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:03.547+0000 7f9dc8a0e700 1 --2- 172.21.15.178:0/259115113 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9dc40f62a0 0x7f9dc40f66c0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7f9db8009fb0 tx=0x7f9db8009b30).stop 2024-02-11T17:11:03.550 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:03.547+0000 7f9dc8a0e700 1 -- 172.21.15.178:0/259115113 shutdown_connections 2024-02-11T17:11:03.550 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:03.547+0000 7f9dc8a0e700 1 --2- 172.21.15.178:0/259115113 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9dc40f62a0 0x7f9dc40f66c0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:11:03.550 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:03.547+0000 7f9dc8a0e700 1 -- 172.21.15.178:0/259115113 >> 172.21.15.178:0/259115113 conn(0x7f9dc40f1ac0 msgr2=0x7f9dc40f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:11:03.550 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:03.547+0000 7f9dc8a0e700 1 -- 172.21.15.178:0/259115113 shutdown_connections 2024-02-11T17:11:03.550 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:03.547+0000 7f9dc8a0e700 1 -- 172.21.15.178:0/259115113 wait complete. 2024-02-11T17:11:03.550 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:03.548+0000 7f9dc8a0e700 1 Processor -- start 2024-02-11T17:11:03.550 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:03.548+0000 7f9dc8a0e700 1 -- start start 2024-02-11T17:11:03.550 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:03.548+0000 7f9dc8a0e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9dc40f62a0 0x7f9dc4106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:11:03.550 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:03.548+0000 7f9dc8a0e700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc41069f0 con 0x7f9dc40f62a0 2024-02-11T17:11:03.552 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:03.548+0000 7f9dc2d9d700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9dc40f62a0 0x7f9dc4106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:11:03.552 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:03.548+0000 7f9dc2d9d700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9dc40f62a0 0x7f9dc4106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:58202/0 (socket says 172.21.15.178:58202) 2024-02-11T17:11:03.552 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:03.548+0000 7f9dc2d9d700 1 -- 172.21.15.178:0/2003079981 learned_addr learned my addr 172.21.15.178:0/2003079981 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:11:03.552 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:03.549+0000 7f9dc2d9d700 1 -- 172.21.15.178:0/2003079981 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9db8005040 con 0x7f9dc40f62a0 2024-02-11T17:11:03.553 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:03.549+0000 7f9dc2d9d700 1 --2- 172.21.15.178:0/2003079981 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9dc40f62a0 0x7f9dc4106500 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f9db8013510 tx=0x7f9db800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:11:03.553 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:03.549+0000 7f9dab7fe700 1 -- 172.21.15.178:0/2003079981 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9db8009320 con 0x7f9dc40f62a0 2024-02-11T17:11:03.553 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:03.549+0000 7f9dab7fe700 1 -- 172.21.15.178:0/2003079981 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9db8013f00 con 0x7f9dc40f62a0 2024-02-11T17:11:03.553 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:03.549+0000 7f9dc8a0e700 1 -- 172.21.15.178:0/2003079981 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dc4106bf0 con 0x7f9dc40f62a0 2024-02-11T17:11:03.553 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:03.549+0000 7f9dab7fe700 1 -- 172.21.15.178:0/2003079981 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9db8014dc0 con 0x7f9dc40f62a0 2024-02-11T17:11:03.553 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:03.549+0000 7f9dc8a0e700 1 -- 172.21.15.178:0/2003079981 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dc4107080 con 0x7f9dc40f62a0 2024-02-11T17:11:03.553 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:03.550+0000 7f9dab7fe700 1 -- 172.21.15.178:0/2003079981 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9db8014420 con 0x7f9dc40f62a0 2024-02-11T17:11:03.553 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:03.550+0000 7f9dc8a0e700 1 -- 172.21.15.178:0/2003079981 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dc4045ad0 con 0x7f9dc40f62a0 2024-02-11T17:11:03.553 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:03.550+0000 7f9dab7fe700 1 --2- 172.21.15.178:0/2003079981 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f9dac02ffb0 0x7f9dac032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:11:03.553 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:03.550+0000 7f9dab7fe700 1 -- 172.21.15.178:0/2003079981 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9db80410b0 con 0x7f9dc40f62a0 2024-02-11T17:11:03.553 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:03.550+0000 7f9dc259c700 1 --2- 172.21.15.178:0/2003079981 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f9dac02ffb0 0x7f9dac032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:11:03.553 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:03.551+0000 7f9dc259c700 1 --2- 172.21.15.178:0/2003079981 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f9dac02ffb0 0x7f9dac032470 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f9db40096a0 tx=0x7f9db4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:11:03.556 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:03.555+0000 7f9dab7fe700 1 -- 172.21.15.178:0/2003079981 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9db806d2b0 con 0x7f9dc40f62a0 2024-02-11T17:11:03.764 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:03.761+0000 7f9dc8a0e700 1 -- 172.21.15.178:0/2003079981 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9dc4107f30 con 0x7f9dc40f62a0 2024-02-11T17:11:03.764 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:03.762+0000 7f9dab7fe700 1 -- 172.21.15.178:0/2003079981 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9db80095f0 con 0x7f9dc40f62a0 2024-02-11T17:11:03.764 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:11:03.764 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:11:03.765 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:03.764+0000 7f9dc8a0e700 1 -- 172.21.15.178:0/2003079981 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f9dac02ffb0 msgr2=0x7f9dac032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:11:03.765 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:03.764+0000 7f9dc8a0e700 1 --2- 172.21.15.178:0/2003079981 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f9dac02ffb0 0x7f9dac032470 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f9db40096a0 tx=0x7f9db4006b40).stop 2024-02-11T17:11:03.766 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:03.764+0000 7f9dc8a0e700 1 -- 172.21.15.178:0/2003079981 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9dc40f62a0 msgr2=0x7f9dc4106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:11:03.766 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:03.764+0000 7f9dc8a0e700 1 --2- 172.21.15.178:0/2003079981 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9dc40f62a0 0x7f9dc4106500 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f9db8013510 tx=0x7f9db800b040).stop 2024-02-11T17:11:03.766 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:03.764+0000 7f9dc8a0e700 1 -- 172.21.15.178:0/2003079981 shutdown_connections 2024-02-11T17:11:03.766 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:03.764+0000 7f9dc8a0e700 1 --2- 172.21.15.178:0/2003079981 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f9dac02ffb0 0x7f9dac032470 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:11:03.766 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:03.764+0000 7f9dc8a0e700 1 --2- 172.21.15.178:0/2003079981 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9dc40f62a0 0x7f9dc4106500 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:11:03.766 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:03.764+0000 7f9dc8a0e700 1 -- 172.21.15.178:0/2003079981 >> 172.21.15.178:0/2003079981 conn(0x7f9dc40f1ac0 msgr2=0x7f9dc40f2530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:11:03.766 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:03.764+0000 7f9dc8a0e700 1 -- 172.21.15.178:0/2003079981 shutdown_connections 2024-02-11T17:11:03.766 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:03.764+0000 7f9dc8a0e700 1 -- 172.21.15.178:0/2003079981 wait complete. 2024-02-11T17:11:03.766 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:11:04.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:11:03 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:11:02.749062+0000 mgr.smithi037.eofnaq (mgr.14162) 340 : cluster [DBG] pgmap v324: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:11:04.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:11:03 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:11:03.281265+0000 mon.smithi037 (mon.0) 318 : audit [INF] from='mgr.14162 172.21.15.37:0/2745465012' entity='mgr.smithi037.eofnaq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi037.eofnaq/mirror_snapshot_schedule"}]: dispatch 2024-02-11T17:11:04.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:11:03 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:11:03.283444+0000 mon.smithi037 (mon.0) 319 : audit [INF] from='mgr.14162 172.21.15.37:0/2745465012' entity='mgr.smithi037.eofnaq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi037.eofnaq/trash_purge_schedule"}]: dispatch 2024-02-11T17:11:04.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:11:03 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:11:03.763253+0000 mon.smithi037 (mon.0) 320 : audit [DBG] from='client.? 172.21.15.178:0/2003079981' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:11:05.590 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:11:05.591 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:11:06.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:11:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:11:04.749448+0000 mgr.smithi037.eofnaq (mgr.14162) 341 : cluster [DBG] pgmap v325: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:11:07.538 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:07.536+0000 7f406bfaf700 1 -- 172.21.15.178:0/1256801077 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4064060280 msgr2=0x7f40640f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:11:07.538 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:07.536+0000 7f406bfaf700 1 --2- 172.21.15.178:0/1256801077 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4064060280 0x7f40640f90f0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7f4054009fb0 tx=0x7f4054009b30).stop 2024-02-11T17:11:07.538 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:07.536+0000 7f406bfaf700 1 -- 172.21.15.178:0/1256801077 shutdown_connections 2024-02-11T17:11:07.538 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:07.536+0000 7f406bfaf700 1 --2- 172.21.15.178:0/1256801077 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4064060280 0x7f40640f90f0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:11:07.538 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:07.536+0000 7f406bfaf700 1 -- 172.21.15.178:0/1256801077 >> 172.21.15.178:0/1256801077 conn(0x7f40640f0b70 msgr2=0x7f40640f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:11:07.539 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:07.536+0000 7f406bfaf700 1 -- 172.21.15.178:0/1256801077 shutdown_connections 2024-02-11T17:11:07.539 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:07.536+0000 7f406bfaf700 1 -- 172.21.15.178:0/1256801077 wait complete. 2024-02-11T17:11:07.540 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:07.537+0000 7f406bfaf700 1 Processor -- start 2024-02-11T17:11:07.540 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:07.537+0000 7f406bfaf700 1 -- start start 2024-02-11T17:11:07.540 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:07.537+0000 7f406bfaf700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4064060280 0x7f40641066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:11:07.540 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:07.537+0000 7f406bfaf700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4064106bd0 con 0x7f4064060280 2024-02-11T17:11:07.540 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:07.538+0000 7f4069d4b700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4064060280 0x7f40641066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:11:07.540 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:07.538+0000 7f4069d4b700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4064060280 0x7f40641066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:58216/0 (socket says 172.21.15.178:58216) 2024-02-11T17:11:07.541 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:07.538+0000 7f4069d4b700 1 -- 172.21.15.178:0/928598792 learned_addr learned my addr 172.21.15.178:0/928598792 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:11:07.541 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:07.538+0000 7f4069d4b700 1 -- 172.21.15.178:0/928598792 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4054005040 con 0x7f4064060280 2024-02-11T17:11:07.541 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:07.538+0000 7f4069d4b700 1 --2- 172.21.15.178:0/928598792 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4064060280 0x7f40641066e0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f4054013510 tx=0x7f405400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:11:07.541 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:07.538+0000 7f405a7fc700 1 -- 172.21.15.178:0/928598792 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4054009320 con 0x7f4064060280 2024-02-11T17:11:07.541 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:07.538+0000 7f405a7fc700 1 -- 172.21.15.178:0/928598792 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4054013f00 con 0x7f4064060280 2024-02-11T17:11:07.541 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:07.538+0000 7f406bfaf700 1 -- 172.21.15.178:0/928598792 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4064106dd0 con 0x7f4064060280 2024-02-11T17:11:07.541 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:07.538+0000 7f405a7fc700 1 -- 172.21.15.178:0/928598792 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4054014dc0 con 0x7f4064060280 2024-02-11T17:11:07.541 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:07.538+0000 7f406bfaf700 1 -- 172.21.15.178:0/928598792 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4064107260 con 0x7f4064060280 2024-02-11T17:11:07.542 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:07.539+0000 7f405a7fc700 1 -- 172.21.15.178:0/928598792 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4054014420 con 0x7f4064060280 2024-02-11T17:11:07.542 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:07.539+0000 7f406bfaf700 1 -- 172.21.15.178:0/928598792 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4064045ad0 con 0x7f4064060280 2024-02-11T17:11:07.542 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:07.539+0000 7f405a7fc700 1 --2- 172.21.15.178:0/928598792 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f405002ff60 0x7f4050032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:11:07.542 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:07.539+0000 7f405a7fc700 1 -- 172.21.15.178:0/928598792 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f40540410f0 con 0x7f4064060280 2024-02-11T17:11:07.542 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:07.539+0000 7f406954a700 1 --2- 172.21.15.178:0/928598792 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f405002ff60 0x7f4050032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:11:07.542 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:07.540+0000 7f406954a700 1 --2- 172.21.15.178:0/928598792 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f405002ff60 0x7f4050032420 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f40600096a0 tx=0x7f4060006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:11:07.546 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:07.544+0000 7f405a7fc700 1 -- 172.21.15.178:0/928598792 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f40540095f0 con 0x7f4064060280 2024-02-11T17:11:07.749 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:07.748+0000 7f406bfaf700 1 -- 172.21.15.178:0/928598792 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4064108110 con 0x7f4064060280 2024-02-11T17:11:07.750 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:07.748+0000 7f405a7fc700 1 -- 172.21.15.178:0/928598792 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f405401cb10 con 0x7f4064060280 2024-02-11T17:11:07.750 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:11:07.751 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:11:07.752 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:07.750+0000 7f406bfaf700 1 -- 172.21.15.178:0/928598792 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f405002ff60 msgr2=0x7f4050032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:11:07.752 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:07.750+0000 7f406bfaf700 1 --2- 172.21.15.178:0/928598792 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f405002ff60 0x7f4050032420 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f40600096a0 tx=0x7f4060006b40).stop 2024-02-11T17:11:07.752 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:07.750+0000 7f406bfaf700 1 -- 172.21.15.178:0/928598792 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4064060280 msgr2=0x7f40641066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:11:07.752 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:07.750+0000 7f406bfaf700 1 --2- 172.21.15.178:0/928598792 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4064060280 0x7f40641066e0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f4054013510 tx=0x7f405400b040).stop 2024-02-11T17:11:07.752 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:07.750+0000 7f406bfaf700 1 -- 172.21.15.178:0/928598792 shutdown_connections 2024-02-11T17:11:07.752 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:07.750+0000 7f406bfaf700 1 --2- 172.21.15.178:0/928598792 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f405002ff60 0x7f4050032420 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:11:07.752 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:07.750+0000 7f406bfaf700 1 --2- 172.21.15.178:0/928598792 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4064060280 0x7f40641066e0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:11:07.752 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:07.750+0000 7f406bfaf700 1 -- 172.21.15.178:0/928598792 >> 172.21.15.178:0/928598792 conn(0x7f40640f0b70 msgr2=0x7f40640f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:11:07.752 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:07.750+0000 7f406bfaf700 1 -- 172.21.15.178:0/928598792 shutdown_connections 2024-02-11T17:11:07.752 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:07.750+0000 7f406bfaf700 1 -- 172.21.15.178:0/928598792 wait complete. 2024-02-11T17:11:07.753 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:11:08.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:11:07 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:11:06.749792+0000 mgr.smithi037.eofnaq (mgr.14162) 342 : cluster [DBG] pgmap v326: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:11:08.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:11:07 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:11:07.749807+0000 mon.smithi037 (mon.0) 321 : audit [DBG] from='client.? 172.21.15.178:0/928598792' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:11:09.406 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:11:09.406 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:11:10.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:11:09 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:11:08.750102+0000 mgr.smithi037.eofnaq (mgr.14162) 343 : cluster [DBG] pgmap v327: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:11:11.381 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:11.378+0000 7f72e7538700 1 -- 172.21.15.178:0/1297704985 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f72e00f6510 msgr2=0x7f72e00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:11:11.381 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:11.378+0000 7f72e7538700 1 --2- 172.21.15.178:0/1297704985 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f72e00f6510 0x7f72e00f6930 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7f72d0009fb0 tx=0x7f72d0009b30).stop 2024-02-11T17:11:11.381 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:11.378+0000 7f72e7538700 1 -- 172.21.15.178:0/1297704985 shutdown_connections 2024-02-11T17:11:11.381 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:11.378+0000 7f72e7538700 1 --2- 172.21.15.178:0/1297704985 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f72e00f6510 0x7f72e00f6930 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:11:11.381 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:11.378+0000 7f72e7538700 1 -- 172.21.15.178:0/1297704985 >> 172.21.15.178:0/1297704985 conn(0x7f72e00f1c70 msgr2=0x7f72e00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:11:11.381 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:11.378+0000 7f72e7538700 1 -- 172.21.15.178:0/1297704985 shutdown_connections 2024-02-11T17:11:11.381 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:11.378+0000 7f72e7538700 1 -- 172.21.15.178:0/1297704985 wait complete. 2024-02-11T17:11:11.382 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:11.379+0000 7f72e7538700 1 Processor -- start 2024-02-11T17:11:11.382 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:11.379+0000 7f72e7538700 1 -- start start 2024-02-11T17:11:11.382 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:11.379+0000 7f72e7538700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f72e00f6510 0x7f72e006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:11:11.382 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:11.379+0000 7f72e7538700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72e0069e20 con 0x7f72e00f6510 2024-02-11T17:11:11.382 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:11.379+0000 7f72e52d4700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f72e00f6510 0x7f72e006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:11:11.382 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:11.379+0000 7f72e52d4700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f72e00f6510 0x7f72e006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:33134/0 (socket says 172.21.15.178:33134) 2024-02-11T17:11:11.382 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:11.379+0000 7f72e52d4700 1 -- 172.21.15.178:0/1656555644 learned_addr learned my addr 172.21.15.178:0/1656555644 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:11:11.384 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:11.379+0000 7f72e52d4700 1 -- 172.21.15.178:0/1656555644 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72d0005040 con 0x7f72e00f6510 2024-02-11T17:11:11.384 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:11.380+0000 7f72e52d4700 1 --2- 172.21.15.178:0/1656555644 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f72e00f6510 0x7f72e006b9a0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f72d0013510 tx=0x7f72d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:11:11.384 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:11.380+0000 7f72d5ffb700 1 -- 172.21.15.178:0/1656555644 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f72d0009320 con 0x7f72e00f6510 2024-02-11T17:11:11.384 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:11.380+0000 7f72e7538700 1 -- 172.21.15.178:0/1656555644 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72e0069fc0 con 0x7f72e00f6510 2024-02-11T17:11:11.384 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:11.380+0000 7f72d5ffb700 1 -- 172.21.15.178:0/1656555644 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f72d0013f00 con 0x7f72e00f6510 2024-02-11T17:11:11.384 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:11.380+0000 7f72d5ffb700 1 -- 172.21.15.178:0/1656555644 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f72d0014d90 con 0x7f72e00f6510 2024-02-11T17:11:11.384 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:11.380+0000 7f72e7538700 1 -- 172.21.15.178:0/1656555644 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72e006a450 con 0x7f72e00f6510 2024-02-11T17:11:11.384 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:11.381+0000 7f72d5ffb700 1 -- 172.21.15.178:0/1656555644 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f72d00095f0 con 0x7f72e00f6510 2024-02-11T17:11:11.384 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:11.381+0000 7f72e7538700 1 -- 172.21.15.178:0/1656555644 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72e0059100 con 0x7f72e00f6510 2024-02-11T17:11:11.384 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:11.381+0000 7f72d5ffb700 1 --2- 172.21.15.178:0/1656555644 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f72cc02ff60 0x7f72cc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:11:11.384 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:11.381+0000 7f72d5ffb700 1 -- 172.21.15.178:0/1656555644 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f72d00410c0 con 0x7f72e00f6510 2024-02-11T17:11:11.385 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:11.381+0000 7f72e4ad3700 1 --2- 172.21.15.178:0/1656555644 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f72cc02ff60 0x7f72cc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:11:11.385 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:11.382+0000 7f72e4ad3700 1 --2- 172.21.15.178:0/1656555644 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f72cc02ff60 0x7f72cc032420 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f72dc0096a0 tx=0x7f72dc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:11:11.387 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:11.386+0000 7f72d5ffb700 1 -- 172.21.15.178:0/1656555644 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f72d001d7f0 con 0x7f72e00f6510 2024-02-11T17:11:11.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:11:11 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:11:10.750466+0000 mgr.smithi037.eofnaq (mgr.14162) 344 : cluster [DBG] pgmap v328: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:11:11.591 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:11.590+0000 7f72e7538700 1 -- 172.21.15.178:0/1656555644 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f72e00fbf60 con 0x7f72e00f6510 2024-02-11T17:11:11.592 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:11.590+0000 7f72d5ffb700 1 -- 172.21.15.178:0/1656555644 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f72d0010230 con 0x7f72e00f6510 2024-02-11T17:11:11.592 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:11:11.592 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:11:11.594 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:11.592+0000 7f72e7538700 1 -- 172.21.15.178:0/1656555644 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f72cc02ff60 msgr2=0x7f72cc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:11:11.594 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:11.592+0000 7f72e7538700 1 --2- 172.21.15.178:0/1656555644 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f72cc02ff60 0x7f72cc032420 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f72dc0096a0 tx=0x7f72dc006b40).stop 2024-02-11T17:11:11.594 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:11.592+0000 7f72e7538700 1 -- 172.21.15.178:0/1656555644 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f72e00f6510 msgr2=0x7f72e006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:11:11.594 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:11.592+0000 7f72e7538700 1 --2- 172.21.15.178:0/1656555644 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f72e00f6510 0x7f72e006b9a0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f72d0013510 tx=0x7f72d000b040).stop 2024-02-11T17:11:11.594 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:11.592+0000 7f72e7538700 1 -- 172.21.15.178:0/1656555644 shutdown_connections 2024-02-11T17:11:11.594 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:11.592+0000 7f72e7538700 1 --2- 172.21.15.178:0/1656555644 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f72cc02ff60 0x7f72cc032420 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:11:11.595 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:11.592+0000 7f72e7538700 1 --2- 172.21.15.178:0/1656555644 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f72e00f6510 0x7f72e006b9a0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:11:11.595 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:11.592+0000 7f72e7538700 1 -- 172.21.15.178:0/1656555644 >> 172.21.15.178:0/1656555644 conn(0x7f72e00f1c70 msgr2=0x7f72e00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:11:11.595 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:11.592+0000 7f72e7538700 1 -- 172.21.15.178:0/1656555644 shutdown_connections 2024-02-11T17:11:11.595 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:11.592+0000 7f72e7538700 1 -- 172.21.15.178:0/1656555644 wait complete. 2024-02-11T17:11:11.595 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:11:12.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:11:12 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:11:11.591811+0000 mon.smithi037 (mon.0) 322 : audit [DBG] from='client.? 172.21.15.178:0/1656555644' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:11:13.254 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:11:13.254 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:11:13.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:11:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:11:12.750782+0000 mgr.smithi037.eofnaq (mgr.14162) 345 : cluster [DBG] pgmap v329: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:11:15.298 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:15.288+0000 7f5976c26700 1 -- 172.21.15.178:0/1933154787 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f59700f5410 msgr2=0x7f59700f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:11:15.298 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:15.288+0000 7f5976c26700 1 --2- 172.21.15.178:0/1933154787 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f59700f5410 0x7f59700f5830 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7f5964009fb0 tx=0x7f5964009b30).stop 2024-02-11T17:11:15.298 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:15.288+0000 7f5976c26700 1 -- 172.21.15.178:0/1933154787 shutdown_connections 2024-02-11T17:11:15.298 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:15.288+0000 7f5976c26700 1 --2- 172.21.15.178:0/1933154787 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f59700f5410 0x7f59700f5830 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:11:15.298 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:15.288+0000 7f5976c26700 1 -- 172.21.15.178:0/1933154787 >> 172.21.15.178:0/1933154787 conn(0x7f59700f0b70 msgr2=0x7f59700f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:11:15.298 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:15.288+0000 7f5976c26700 1 -- 172.21.15.178:0/1933154787 shutdown_connections 2024-02-11T17:11:15.299 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:15.288+0000 7f5976c26700 1 -- 172.21.15.178:0/1933154787 wait complete. 2024-02-11T17:11:15.299 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:15.289+0000 7f5976c26700 1 Processor -- start 2024-02-11T17:11:15.299 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:15.289+0000 7f5976c26700 1 -- start start 2024-02-11T17:11:15.299 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:15.289+0000 7f5976c26700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f59700f5410 0x7f5970105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:11:15.299 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:15.289+0000 7f5976c26700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5970105af0 con 0x7f59700f5410 2024-02-11T17:11:15.299 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:15.289+0000 7f59749c2700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f59700f5410 0x7f5970105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:11:15.299 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:15.289+0000 7f59749c2700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f59700f5410 0x7f5970105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:33150/0 (socket says 172.21.15.178:33150) 2024-02-11T17:11:15.299 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:15.289+0000 7f59749c2700 1 -- 172.21.15.178:0/371074847 learned_addr learned my addr 172.21.15.178:0/371074847 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:11:15.299 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:15.290+0000 7f59749c2700 1 -- 172.21.15.178:0/371074847 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5964005040 con 0x7f59700f5410 2024-02-11T17:11:15.299 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:15.290+0000 7f59749c2700 1 --2- 172.21.15.178:0/371074847 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f59700f5410 0x7f5970105600 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f5964013510 tx=0x7f596400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:11:15.299 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:15.290+0000 7f596d7fa700 1 -- 172.21.15.178:0/371074847 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5964009320 con 0x7f59700f5410 2024-02-11T17:11:15.299 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:15.290+0000 7f596d7fa700 1 -- 172.21.15.178:0/371074847 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5964013f00 con 0x7f59700f5410 2024-02-11T17:11:15.299 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:15.290+0000 7f5976c26700 1 -- 172.21.15.178:0/371074847 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5970105cf0 con 0x7f59700f5410 2024-02-11T17:11:15.299 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:15.290+0000 7f596d7fa700 1 -- 172.21.15.178:0/371074847 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5964014d90 con 0x7f59700f5410 2024-02-11T17:11:15.299 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:15.290+0000 7f5976c26700 1 -- 172.21.15.178:0/371074847 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5970106180 con 0x7f59700f5410 2024-02-11T17:11:15.299 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:15.291+0000 7f596d7fa700 1 -- 172.21.15.178:0/371074847 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f59640095f0 con 0x7f59700f5410 2024-02-11T17:11:15.300 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:15.291+0000 7f5976c26700 1 -- 172.21.15.178:0/371074847 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5970045ad0 con 0x7f59700f5410 2024-02-11T17:11:15.300 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:15.291+0000 7f596d7fa700 1 --2- 172.21.15.178:0/371074847 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f596002ff60 0x7f5960032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:11:15.300 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:15.291+0000 7f596d7fa700 1 -- 172.21.15.178:0/371074847 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5964040fe0 con 0x7f59700f5410 2024-02-11T17:11:15.300 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:15.292+0000 7f596ffff700 1 --2- 172.21.15.178:0/371074847 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f596002ff60 0x7f5960032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:11:15.300 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:15.292+0000 7f596ffff700 1 --2- 172.21.15.178:0/371074847 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f596002ff60 0x7f5960032420 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f595c0096a0 tx=0x7f595c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:11:15.300 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:15.296+0000 7f596d7fa700 1 -- 172.21.15.178:0/371074847 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f596401d7f0 con 0x7f59700f5410 2024-02-11T17:11:15.506 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:15.505+0000 7f5976c26700 1 -- 172.21.15.178:0/371074847 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f59700fae60 con 0x7f59700f5410 2024-02-11T17:11:15.507 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:15.505+0000 7f596d7fa700 1 -- 172.21.15.178:0/371074847 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5964010230 con 0x7f59700f5410 2024-02-11T17:11:15.507 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:11:15.507 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:11:15.509 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:15.507+0000 7f5976c26700 1 -- 172.21.15.178:0/371074847 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f596002ff60 msgr2=0x7f5960032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:11:15.509 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:15.507+0000 7f5976c26700 1 --2- 172.21.15.178:0/371074847 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f596002ff60 0x7f5960032420 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f595c0096a0 tx=0x7f595c006b40).stop 2024-02-11T17:11:15.509 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:15.507+0000 7f5976c26700 1 -- 172.21.15.178:0/371074847 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f59700f5410 msgr2=0x7f5970105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:11:15.509 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:15.507+0000 7f5976c26700 1 --2- 172.21.15.178:0/371074847 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f59700f5410 0x7f5970105600 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f5964013510 tx=0x7f596400b040).stop 2024-02-11T17:11:15.509 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:15.507+0000 7f5976c26700 1 -- 172.21.15.178:0/371074847 shutdown_connections 2024-02-11T17:11:15.509 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:15.507+0000 7f5976c26700 1 --2- 172.21.15.178:0/371074847 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f596002ff60 0x7f5960032420 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:11:15.509 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:15.507+0000 7f5976c26700 1 --2- 172.21.15.178:0/371074847 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f59700f5410 0x7f5970105600 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:11:15.509 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:15.507+0000 7f5976c26700 1 -- 172.21.15.178:0/371074847 >> 172.21.15.178:0/371074847 conn(0x7f59700f0b70 msgr2=0x7f59700f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:11:15.509 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:15.507+0000 7f5976c26700 1 -- 172.21.15.178:0/371074847 shutdown_connections 2024-02-11T17:11:15.510 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:15.507+0000 7f5976c26700 1 -- 172.21.15.178:0/371074847 wait complete. 2024-02-11T17:11:15.510 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:11:16.184 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:11:15 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:11:14.751148+0000 mgr.smithi037.eofnaq (mgr.14162) 346 : cluster [DBG] pgmap v330: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:11:16.184 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:11:15 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:11:15.506774+0000 mon.smithi037 (mon.0) 323 : audit [DBG] from='client.? 172.21.15.178:0/371074847' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:11:17.302 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:11:17.302 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:11:18.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:11:17 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:11:16.751470+0000 mgr.smithi037.eofnaq (mgr.14162) 347 : cluster [DBG] pgmap v331: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:11:19.242 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:19.239+0000 7f83e304f700 1 -- 172.21.15.178:0/2863425569 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f83dc0f51a0 msgr2=0x7f83dc0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:11:19.243 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:19.239+0000 7f83e304f700 1 --2- 172.21.15.178:0/2863425569 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f83dc0f51a0 0x7f83dc0f55c0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7f83d8009fb0 tx=0x7f83d8009b30).stop 2024-02-11T17:11:19.243 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:19.239+0000 7f83e304f700 1 -- 172.21.15.178:0/2863425569 shutdown_connections 2024-02-11T17:11:19.243 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:19.239+0000 7f83e304f700 1 --2- 172.21.15.178:0/2863425569 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f83dc0f51a0 0x7f83dc0f55c0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:11:19.243 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:19.239+0000 7f83e304f700 1 -- 172.21.15.178:0/2863425569 >> 172.21.15.178:0/2863425569 conn(0x7f83dc0f09a0 msgr2=0x7f83dc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:11:19.243 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:19.239+0000 7f83e304f700 1 -- 172.21.15.178:0/2863425569 shutdown_connections 2024-02-11T17:11:19.243 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:19.239+0000 7f83e304f700 1 -- 172.21.15.178:0/2863425569 wait complete. 2024-02-11T17:11:19.243 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:19.239+0000 7f83e304f700 1 Processor -- start 2024-02-11T17:11:19.243 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:19.239+0000 7f83e304f700 1 -- start start 2024-02-11T17:11:19.243 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:19.240+0000 7f83e304f700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f83dc0f51a0 0x7f83dc1091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:11:19.243 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:19.240+0000 7f83e304f700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83dc060240 con 0x7f83dc0f51a0 2024-02-11T17:11:19.243 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:19.240+0000 7f83e204d700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f83dc0f51a0 0x7f83dc1091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:11:19.244 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:19.240+0000 7f83e204d700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f83dc0f51a0 0x7f83dc1091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:33164/0 (socket says 172.21.15.178:33164) 2024-02-11T17:11:19.244 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:19.240+0000 7f83e204d700 1 -- 172.21.15.178:0/1437441469 learned_addr learned my addr 172.21.15.178:0/1437441469 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:11:19.244 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:19.240+0000 7f83e204d700 1 -- 172.21.15.178:0/1437441469 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83d8005040 con 0x7f83dc0f51a0 2024-02-11T17:11:19.244 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:19.240+0000 7f83e204d700 1 --2- 172.21.15.178:0/1437441469 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f83dc0f51a0 0x7f83dc1091b0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f83d8013510 tx=0x7f83d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:11:19.244 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:19.241+0000 7f83d2ffd700 1 -- 172.21.15.178:0/1437441469 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f83d8009320 con 0x7f83dc0f51a0 2024-02-11T17:11:19.244 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:19.241+0000 7f83e304f700 1 -- 172.21.15.178:0/1437441469 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83dc060440 con 0x7f83dc0f51a0 2024-02-11T17:11:19.244 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:19.241+0000 7f83d2ffd700 1 -- 172.21.15.178:0/1437441469 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f83d8013f00 con 0x7f83dc0f51a0 2024-02-11T17:11:19.244 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:19.241+0000 7f83d2ffd700 1 -- 172.21.15.178:0/1437441469 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f83d8014dc0 con 0x7f83dc0f51a0 2024-02-11T17:11:19.244 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:19.241+0000 7f83e304f700 1 -- 172.21.15.178:0/1437441469 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83dc109860 con 0x7f83dc0f51a0 2024-02-11T17:11:19.245 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:19.241+0000 7f83d2ffd700 1 -- 172.21.15.178:0/1437441469 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f83d801d3d0 con 0x7f83dc0f51a0 2024-02-11T17:11:19.245 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:19.242+0000 7f83d2ffd700 1 --2- 172.21.15.178:0/1437441469 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f83c802ff60 0x7f83c8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:11:19.245 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:19.242+0000 7f83d2ffd700 1 -- 172.21.15.178:0/1437441469 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f83d80410d0 con 0x7f83dc0f51a0 2024-02-11T17:11:19.245 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:19.242+0000 7f83e304f700 1 -- 172.21.15.178:0/1437441469 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83dc045ad0 con 0x7f83dc0f51a0 2024-02-11T17:11:19.245 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:19.242+0000 7f83e184c700 1 --2- 172.21.15.178:0/1437441469 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f83c802ff60 0x7f83c8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:11:19.245 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:19.242+0000 7f83e184c700 1 --2- 172.21.15.178:0/1437441469 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f83c802ff60 0x7f83c8032420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f83cc0096a0 tx=0x7f83cc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:11:19.248 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:19.247+0000 7f83d2ffd700 1 -- 172.21.15.178:0/1437441469 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f83d8012920 con 0x7f83dc0f51a0 2024-02-11T17:11:19.450 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:19.450+0000 7f83e304f700 1 -- 172.21.15.178:0/1437441469 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f83dc109f40 con 0x7f83dc0f51a0 2024-02-11T17:11:19.451 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:19.450+0000 7f83d2ffd700 1 -- 172.21.15.178:0/1437441469 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f83d802b020 con 0x7f83dc0f51a0 2024-02-11T17:11:19.451 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:11:19.451 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:11:19.453 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:19.452+0000 7f83e304f700 1 -- 172.21.15.178:0/1437441469 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f83c802ff60 msgr2=0x7f83c8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:11:19.453 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:19.452+0000 7f83e304f700 1 --2- 172.21.15.178:0/1437441469 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f83c802ff60 0x7f83c8032420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f83cc0096a0 tx=0x7f83cc006b40).stop 2024-02-11T17:11:19.453 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:19.452+0000 7f83e304f700 1 -- 172.21.15.178:0/1437441469 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f83dc0f51a0 msgr2=0x7f83dc1091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:11:19.453 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:19.452+0000 7f83e304f700 1 --2- 172.21.15.178:0/1437441469 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f83dc0f51a0 0x7f83dc1091b0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f83d8013510 tx=0x7f83d800b040).stop 2024-02-11T17:11:19.453 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:19.452+0000 7f83e304f700 1 -- 172.21.15.178:0/1437441469 shutdown_connections 2024-02-11T17:11:19.453 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:19.452+0000 7f83e304f700 1 --2- 172.21.15.178:0/1437441469 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f83c802ff60 0x7f83c8032420 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:11:19.453 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:19.452+0000 7f83e304f700 1 --2- 172.21.15.178:0/1437441469 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f83dc0f51a0 0x7f83dc1091b0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:11:19.453 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:19.452+0000 7f83e304f700 1 -- 172.21.15.178:0/1437441469 >> 172.21.15.178:0/1437441469 conn(0x7f83dc0f09a0 msgr2=0x7f83dc0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:11:19.454 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:19.452+0000 7f83e304f700 1 -- 172.21.15.178:0/1437441469 shutdown_connections 2024-02-11T17:11:19.454 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:19.452+0000 7f83e304f700 1 -- 172.21.15.178:0/1437441469 wait complete. 2024-02-11T17:11:19.454 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:11:20.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:11:19 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:11:18.751771+0000 mgr.smithi037.eofnaq (mgr.14162) 348 : cluster [DBG] pgmap v332: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:11:20.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:11:19 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:11:19.450724+0000 mon.smithi037 (mon.0) 324 : audit [DBG] from='client.? 172.21.15.178:0/1437441469' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:11:21.143 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:11:21.143 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:11:21.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:11:21 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:11:20.752183+0000 mgr.smithi037.eofnaq (mgr.14162) 349 : cluster [DBG] pgmap v333: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:11:23.074 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:23.074+0000 7fead6b01700 1 -- 172.21.15.178:0/3911914729 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fead00f51a0 msgr2=0x7fead00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:11:23.075 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:23.074+0000 7fead6b01700 1 --2- 172.21.15.178:0/3911914729 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fead00f51a0 0x7fead00f55c0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7feacc009fb0 tx=0x7feacc009b30).stop 2024-02-11T17:11:23.075 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:23.074+0000 7fead6b01700 1 -- 172.21.15.178:0/3911914729 shutdown_connections 2024-02-11T17:11:23.075 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:23.074+0000 7fead6b01700 1 --2- 172.21.15.178:0/3911914729 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fead00f51a0 0x7fead00f55c0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:11:23.075 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:23.074+0000 7fead6b01700 1 -- 172.21.15.178:0/3911914729 >> 172.21.15.178:0/3911914729 conn(0x7fead00f09a0 msgr2=0x7fead00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:11:23.075 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:23.074+0000 7fead6b01700 1 -- 172.21.15.178:0/3911914729 shutdown_connections 2024-02-11T17:11:23.075 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:23.074+0000 7fead6b01700 1 -- 172.21.15.178:0/3911914729 wait complete. 2024-02-11T17:11:23.077 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:23.074+0000 7fead6b01700 1 Processor -- start 2024-02-11T17:11:23.077 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:23.074+0000 7fead6b01700 1 -- start start 2024-02-11T17:11:23.077 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:23.074+0000 7fead6b01700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fead00f51a0 0x7fead01053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:11:23.077 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:23.075+0000 7fead6b01700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fead01058a0 con 0x7fead00f51a0 2024-02-11T17:11:23.077 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:23.075+0000 7fead5aff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fead00f51a0 0x7fead01053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:11:23.077 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:23.075+0000 7fead5aff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fead00f51a0 0x7fead01053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:36912/0 (socket says 172.21.15.178:36912) 2024-02-11T17:11:23.077 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:23.075+0000 7fead5aff700 1 -- 172.21.15.178:0/2071680040 learned_addr learned my addr 172.21.15.178:0/2071680040 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:11:23.077 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:23.075+0000 7fead5aff700 1 -- 172.21.15.178:0/2071680040 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feacc005040 con 0x7fead00f51a0 2024-02-11T17:11:23.077 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:23.075+0000 7fead5aff700 1 --2- 172.21.15.178:0/2071680040 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fead00f51a0 0x7fead01053b0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7feacc013510 tx=0x7feacc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:11:23.077 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:23.075+0000 7feac67fc700 1 -- 172.21.15.178:0/2071680040 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feacc009320 con 0x7fead00f51a0 2024-02-11T17:11:23.077 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:23.075+0000 7fead6b01700 1 -- 172.21.15.178:0/2071680040 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fead0105aa0 con 0x7fead00f51a0 2024-02-11T17:11:23.078 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:23.075+0000 7feac67fc700 1 -- 172.21.15.178:0/2071680040 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feacc013f00 con 0x7fead00f51a0 2024-02-11T17:11:23.078 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:23.075+0000 7feac67fc700 1 -- 172.21.15.178:0/2071680040 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feacc014dc0 con 0x7fead00f51a0 2024-02-11T17:11:23.078 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:23.075+0000 7fead6b01700 1 -- 172.21.15.178:0/2071680040 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fead0105f30 con 0x7fead00f51a0 2024-02-11T17:11:23.078 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:23.076+0000 7feac67fc700 1 -- 172.21.15.178:0/2071680040 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7feacc0095f0 con 0x7fead00f51a0 2024-02-11T17:11:23.078 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:23.076+0000 7fead6b01700 1 -- 172.21.15.178:0/2071680040 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fead0045ad0 con 0x7fead00f51a0 2024-02-11T17:11:23.078 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:23.076+0000 7feac67fc700 1 --2- 172.21.15.178:0/2071680040 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7feabc02ff60 0x7feabc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:11:23.078 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:23.076+0000 7feac67fc700 1 -- 172.21.15.178:0/2071680040 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7feacc041000 con 0x7fead00f51a0 2024-02-11T17:11:23.078 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:23.077+0000 7fead52fe700 1 --2- 172.21.15.178:0/2071680040 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7feabc02ff60 0x7feabc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:11:23.079 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:23.077+0000 7fead52fe700 1 --2- 172.21.15.178:0/2071680040 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7feabc02ff60 0x7feabc032420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7feac00096a0 tx=0x7feac0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:11:23.082 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:23.081+0000 7feac67fc700 1 -- 172.21.15.178:0/2071680040 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7feacc01c790 con 0x7fead00f51a0 2024-02-11T17:11:23.283 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:23.283+0000 7fead6b01700 1 -- 172.21.15.178:0/2071680040 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fead0106de0 con 0x7fead00f51a0 2024-02-11T17:11:23.284 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:23.283+0000 7feac67fc700 1 -- 172.21.15.178:0/2071680040 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7feacc010230 con 0x7fead00f51a0 2024-02-11T17:11:23.284 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:11:23.284 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:11:23.286 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:23.285+0000 7fead6b01700 1 -- 172.21.15.178:0/2071680040 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7feabc02ff60 msgr2=0x7feabc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:11:23.286 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:23.285+0000 7fead6b01700 1 --2- 172.21.15.178:0/2071680040 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7feabc02ff60 0x7feabc032420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7feac00096a0 tx=0x7feac0006b40).stop 2024-02-11T17:11:23.286 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:23.285+0000 7fead6b01700 1 -- 172.21.15.178:0/2071680040 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fead00f51a0 msgr2=0x7fead01053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:11:23.286 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:23.285+0000 7fead6b01700 1 --2- 172.21.15.178:0/2071680040 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fead00f51a0 0x7fead01053b0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7feacc013510 tx=0x7feacc00b040).stop 2024-02-11T17:11:23.286 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:23.285+0000 7fead6b01700 1 -- 172.21.15.178:0/2071680040 shutdown_connections 2024-02-11T17:11:23.286 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:23.285+0000 7fead6b01700 1 --2- 172.21.15.178:0/2071680040 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7feabc02ff60 0x7feabc032420 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:11:23.286 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:23.285+0000 7fead6b01700 1 --2- 172.21.15.178:0/2071680040 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fead00f51a0 0x7fead01053b0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:11:23.286 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:23.285+0000 7fead6b01700 1 -- 172.21.15.178:0/2071680040 >> 172.21.15.178:0/2071680040 conn(0x7fead00f09a0 msgr2=0x7fead00f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:11:23.286 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:23.285+0000 7fead6b01700 1 -- 172.21.15.178:0/2071680040 shutdown_connections 2024-02-11T17:11:23.286 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:23.285+0000 7fead6b01700 1 -- 172.21.15.178:0/2071680040 wait complete. 2024-02-11T17:11:23.287 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:11:24.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:11:23 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:11:22.752511+0000 mgr.smithi037.eofnaq (mgr.14162) 350 : cluster [DBG] pgmap v334: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:11:24.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:11:23 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:11:23.283856+0000 mon.smithi037 (mon.0) 325 : audit [DBG] from='client.? 172.21.15.178:0/2071680040' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:11:25.006 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:11:25.006 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:11:26.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:11:25 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:11:24.752917+0000 mgr.smithi037.eofnaq (mgr.14162) 351 : cluster [DBG] pgmap v335: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:11:26.945 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:26.933+0000 7fac4dcf2700 1 -- 172.21.15.178:0/430298944 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fac480f51a0 msgr2=0x7fac480f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:11:26.945 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:26.933+0000 7fac4dcf2700 1 --2- 172.21.15.178:0/430298944 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fac480f51a0 0x7fac480f55c0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7fac3c009fb0 tx=0x7fac3c009b30).stop 2024-02-11T17:11:26.945 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:26.933+0000 7fac4dcf2700 1 -- 172.21.15.178:0/430298944 shutdown_connections 2024-02-11T17:11:26.945 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:26.933+0000 7fac4dcf2700 1 --2- 172.21.15.178:0/430298944 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fac480f51a0 0x7fac480f55c0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:11:26.945 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:26.933+0000 7fac4dcf2700 1 -- 172.21.15.178:0/430298944 >> 172.21.15.178:0/430298944 conn(0x7fac480f09a0 msgr2=0x7fac480f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:11:26.945 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:26.933+0000 7fac4dcf2700 1 -- 172.21.15.178:0/430298944 shutdown_connections 2024-02-11T17:11:26.945 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:26.933+0000 7fac4dcf2700 1 -- 172.21.15.178:0/430298944 wait complete. 2024-02-11T17:11:26.946 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:26.933+0000 7fac4dcf2700 1 Processor -- start 2024-02-11T17:11:26.946 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:26.933+0000 7fac4dcf2700 1 -- start start 2024-02-11T17:11:26.946 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:26.934+0000 7fac4dcf2700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fac480f51a0 0x7fac481053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:11:26.946 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:26.934+0000 7fac4dcf2700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac481058a0 con 0x7fac480f51a0 2024-02-11T17:11:26.946 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:26.934+0000 7fac4ccf0700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fac480f51a0 0x7fac481053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:11:26.946 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:26.934+0000 7fac4ccf0700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fac480f51a0 0x7fac481053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:36926/0 (socket says 172.21.15.178:36926) 2024-02-11T17:11:26.946 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:26.934+0000 7fac4ccf0700 1 -- 172.21.15.178:0/3003112800 learned_addr learned my addr 172.21.15.178:0/3003112800 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:11:26.946 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:26.934+0000 7fac4ccf0700 1 -- 172.21.15.178:0/3003112800 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac3c005040 con 0x7fac480f51a0 2024-02-11T17:11:26.946 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:26.935+0000 7fac4ccf0700 1 --2- 172.21.15.178:0/3003112800 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fac480f51a0 0x7fac481053b0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7fac3c013510 tx=0x7fac3c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:11:26.946 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:26.935+0000 7fac457fa700 1 -- 172.21.15.178:0/3003112800 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac3c009320 con 0x7fac480f51a0 2024-02-11T17:11:26.946 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:26.935+0000 7fac457fa700 1 -- 172.21.15.178:0/3003112800 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fac3c013f00 con 0x7fac480f51a0 2024-02-11T17:11:26.946 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:26.935+0000 7fac4dcf2700 1 -- 172.21.15.178:0/3003112800 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac48105aa0 con 0x7fac480f51a0 2024-02-11T17:11:26.946 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:26.935+0000 7fac457fa700 1 -- 172.21.15.178:0/3003112800 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac3c014dc0 con 0x7fac480f51a0 2024-02-11T17:11:26.946 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:26.935+0000 7fac4dcf2700 1 -- 172.21.15.178:0/3003112800 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac48105f30 con 0x7fac480f51a0 2024-02-11T17:11:26.947 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:26.936+0000 7fac457fa700 1 -- 172.21.15.178:0/3003112800 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fac3c0095f0 con 0x7fac480f51a0 2024-02-11T17:11:26.947 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:26.936+0000 7fac4dcf2700 1 -- 172.21.15.178:0/3003112800 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac48045ad0 con 0x7fac480f51a0 2024-02-11T17:11:26.947 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:26.936+0000 7fac457fa700 1 --2- 172.21.15.178:0/3003112800 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fac3802ffb0 0x7fac38032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:11:26.947 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:26.936+0000 7fac457fa700 1 -- 172.21.15.178:0/3003112800 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fac3c0410b0 con 0x7fac480f51a0 2024-02-11T17:11:26.947 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:26.936+0000 7fac47fff700 1 --2- 172.21.15.178:0/3003112800 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fac3802ffb0 0x7fac38032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:11:26.947 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:26.937+0000 7fac47fff700 1 --2- 172.21.15.178:0/3003112800 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fac3802ffb0 0x7fac38032470 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fac340096a0 tx=0x7fac34006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:11:26.947 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:26.941+0000 7fac457fa700 1 -- 172.21.15.178:0/3003112800 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fac3c06d2b0 con 0x7fac480f51a0 2024-02-11T17:11:27.150 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:27.148+0000 7fac4dcf2700 1 -- 172.21.15.178:0/3003112800 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fac48106de0 con 0x7fac480f51a0 2024-02-11T17:11:27.150 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:27.149+0000 7fac457fa700 1 -- 172.21.15.178:0/3003112800 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fac3c014420 con 0x7fac480f51a0 2024-02-11T17:11:27.151 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:11:27.151 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:11:27.152 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:27.150+0000 7fac4dcf2700 1 -- 172.21.15.178:0/3003112800 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fac3802ffb0 msgr2=0x7fac38032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:11:27.153 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:27.150+0000 7fac4dcf2700 1 --2- 172.21.15.178:0/3003112800 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fac3802ffb0 0x7fac38032470 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fac340096a0 tx=0x7fac34006b40).stop 2024-02-11T17:11:27.153 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:27.150+0000 7fac4dcf2700 1 -- 172.21.15.178:0/3003112800 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fac480f51a0 msgr2=0x7fac481053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:11:27.153 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:27.150+0000 7fac4dcf2700 1 --2- 172.21.15.178:0/3003112800 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fac480f51a0 0x7fac481053b0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7fac3c013510 tx=0x7fac3c00b040).stop 2024-02-11T17:11:27.153 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:27.151+0000 7fac4dcf2700 1 -- 172.21.15.178:0/3003112800 shutdown_connections 2024-02-11T17:11:27.153 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:27.151+0000 7fac4dcf2700 1 --2- 172.21.15.178:0/3003112800 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fac3802ffb0 0x7fac38032470 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:11:27.153 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:27.151+0000 7fac4dcf2700 1 --2- 172.21.15.178:0/3003112800 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fac480f51a0 0x7fac481053b0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:11:27.153 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:27.151+0000 7fac4dcf2700 1 -- 172.21.15.178:0/3003112800 >> 172.21.15.178:0/3003112800 conn(0x7fac480f09a0 msgr2=0x7fac480f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:11:27.153 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:27.151+0000 7fac4dcf2700 1 -- 172.21.15.178:0/3003112800 shutdown_connections 2024-02-11T17:11:27.153 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:27.151+0000 7fac4dcf2700 1 -- 172.21.15.178:0/3003112800 wait complete. 2024-02-11T17:11:27.153 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:11:28.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:11:27 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:11:26.753375+0000 mgr.smithi037.eofnaq (mgr.14162) 352 : cluster [DBG] pgmap v336: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:11:28.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:11:27 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:11:27.150347+0000 mon.smithi037 (mon.0) 326 : audit [DBG] from='client.? 172.21.15.178:0/3003112800' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:11:28.825 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:11:28.825 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:11:30.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:11:29 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:11:28.753799+0000 mgr.smithi037.eofnaq (mgr.14162) 353 : cluster [DBG] pgmap v337: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:11:30.909 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:30.907+0000 7f3ff3fff700 1 -- 172.21.15.178:0/2870064195 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3ff40f51a0 msgr2=0x7f3ff40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:11:30.909 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:30.907+0000 7f3ff3fff700 1 --2- 172.21.15.178:0/2870064195 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3ff40f51a0 0x7f3ff40f55c0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7f3fe8009fb0 tx=0x7f3fe8009b30).stop 2024-02-11T17:11:30.909 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:30.907+0000 7f3ff3fff700 1 -- 172.21.15.178:0/2870064195 shutdown_connections 2024-02-11T17:11:30.909 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:30.907+0000 7f3ff3fff700 1 --2- 172.21.15.178:0/2870064195 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3ff40f51a0 0x7f3ff40f55c0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:11:30.909 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:30.907+0000 7f3ff3fff700 1 -- 172.21.15.178:0/2870064195 >> 172.21.15.178:0/2870064195 conn(0x7f3ff40f09a0 msgr2=0x7f3ff40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:11:30.909 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:30.907+0000 7f3ff3fff700 1 -- 172.21.15.178:0/2870064195 shutdown_connections 2024-02-11T17:11:30.910 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:30.907+0000 7f3ff3fff700 1 -- 172.21.15.178:0/2870064195 wait complete. 2024-02-11T17:11:30.911 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:30.908+0000 7f3ff3fff700 1 Processor -- start 2024-02-11T17:11:30.911 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:30.908+0000 7f3ff3fff700 1 -- start start 2024-02-11T17:11:30.911 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:30.908+0000 7f3ff3fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3ff40f51a0 0x7f3ff41091e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:11:30.912 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:30.908+0000 7f3ff3fff700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ff4060240 con 0x7f3ff40f51a0 2024-02-11T17:11:30.912 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:30.909+0000 7f3ff2ffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3ff40f51a0 0x7f3ff41091e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:11:30.912 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:30.909+0000 7f3ff2ffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3ff40f51a0 0x7f3ff41091e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:50038/0 (socket says 172.21.15.178:50038) 2024-02-11T17:11:30.912 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:30.909+0000 7f3ff2ffd700 1 -- 172.21.15.178:0/1085105720 learned_addr learned my addr 172.21.15.178:0/1085105720 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:11:30.912 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:30.909+0000 7f3ff2ffd700 1 -- 172.21.15.178:0/1085105720 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe8005040 con 0x7f3ff40f51a0 2024-02-11T17:11:30.912 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:30.909+0000 7f3ff2ffd700 1 --2- 172.21.15.178:0/1085105720 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3ff40f51a0 0x7f3ff41091e0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f3fe8009830 tx=0x7f3fe800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:11:30.912 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:30.909+0000 7f3fdb7fe700 1 -- 172.21.15.178:0/1085105720 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3fe8009320 con 0x7f3ff40f51a0 2024-02-11T17:11:30.912 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:30.909+0000 7f3ff3fff700 1 -- 172.21.15.178:0/1085105720 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ff4060440 con 0x7f3ff40f51a0 2024-02-11T17:11:30.912 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:30.909+0000 7f3fdb7fe700 1 -- 172.21.15.178:0/1085105720 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3fe80136c0 con 0x7f3ff40f51a0 2024-02-11T17:11:30.912 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:30.909+0000 7f3fdb7fe700 1 -- 172.21.15.178:0/1085105720 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3fe801ee90 con 0x7f3ff40f51a0 2024-02-11T17:11:30.912 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:30.909+0000 7f3ff3fff700 1 -- 172.21.15.178:0/1085105720 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ff4109890 con 0x7f3ff40f51a0 2024-02-11T17:11:30.913 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:30.910+0000 7f3fdb7fe700 1 -- 172.21.15.178:0/1085105720 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3fe801e420 con 0x7f3ff40f51a0 2024-02-11T17:11:30.913 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:30.910+0000 7f3ff3fff700 1 -- 172.21.15.178:0/1085105720 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ff4045ad0 con 0x7f3ff40f51a0 2024-02-11T17:11:30.914 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:30.910+0000 7f3fdb7fe700 1 --2- 172.21.15.178:0/1085105720 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f3fdc02ff10 0x7f3fdc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:11:30.914 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:30.910+0000 7f3fdb7fe700 1 -- 172.21.15.178:0/1085105720 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3fe8041120 con 0x7f3ff40f51a0 2024-02-11T17:11:30.914 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:30.911+0000 7f3ff27fc700 1 --2- 172.21.15.178:0/1085105720 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f3fdc02ff10 0x7f3fdc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:11:30.914 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:30.911+0000 7f3ff27fc700 1 --2- 172.21.15.178:0/1085105720 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f3fdc02ff10 0x7f3fdc0323d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f3fe40096a0 tx=0x7f3fe4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:11:30.916 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:30.915+0000 7f3fdb7fe700 1 -- 172.21.15.178:0/1085105720 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3fe80095f0 con 0x7f3ff40f51a0 2024-02-11T17:11:31.123 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:31.121+0000 7f3ff3fff700 1 -- 172.21.15.178:0/1085105720 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3ff410a0b0 con 0x7f3ff40f51a0 2024-02-11T17:11:31.124 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:31.122+0000 7f3fdb7fe700 1 -- 172.21.15.178:0/1085105720 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3fe801cb10 con 0x7f3ff40f51a0 2024-02-11T17:11:31.124 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:11:31.124 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:11:31.125 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:31.123+0000 7f3ff3fff700 1 -- 172.21.15.178:0/1085105720 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f3fdc02ff10 msgr2=0x7f3fdc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:11:31.125 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:31.123+0000 7f3ff3fff700 1 --2- 172.21.15.178:0/1085105720 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f3fdc02ff10 0x7f3fdc0323d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f3fe40096a0 tx=0x7f3fe4006b40).stop 2024-02-11T17:11:31.126 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:31.123+0000 7f3ff3fff700 1 -- 172.21.15.178:0/1085105720 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3ff40f51a0 msgr2=0x7f3ff41091e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:11:31.126 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:31.123+0000 7f3ff3fff700 1 --2- 172.21.15.178:0/1085105720 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3ff40f51a0 0x7f3ff41091e0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f3fe8009830 tx=0x7f3fe800b040).stop 2024-02-11T17:11:31.126 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:31.123+0000 7f3ff3fff700 1 -- 172.21.15.178:0/1085105720 shutdown_connections 2024-02-11T17:11:31.126 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:31.123+0000 7f3ff3fff700 1 --2- 172.21.15.178:0/1085105720 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f3fdc02ff10 0x7f3fdc0323d0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:11:31.126 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:31.123+0000 7f3ff3fff700 1 --2- 172.21.15.178:0/1085105720 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3ff40f51a0 0x7f3ff41091e0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:11:31.126 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:31.123+0000 7f3ff3fff700 1 -- 172.21.15.178:0/1085105720 >> 172.21.15.178:0/1085105720 conn(0x7f3ff40f09a0 msgr2=0x7f3ff40f13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:11:31.126 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:31.123+0000 7f3ff3fff700 1 -- 172.21.15.178:0/1085105720 shutdown_connections 2024-02-11T17:11:31.126 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:31.123+0000 7f3ff3fff700 1 -- 172.21.15.178:0/1085105720 wait complete. 2024-02-11T17:11:31.126 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:11:31.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:11:31 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:11:30.754240+0000 mgr.smithi037.eofnaq (mgr.14162) 354 : cluster [DBG] pgmap v338: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:11:31.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:11:31 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:11:31.123223+0000 mon.smithi037 (mon.0) 327 : audit [DBG] from='client.? 172.21.15.178:0/1085105720' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:11:32.822 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:11:32.823 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:11:34.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:11:33 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:11:32.754528+0000 mgr.smithi037.eofnaq (mgr.14162) 355 : cluster [DBG] pgmap v339: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:11:34.688 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:34.685+0000 7fce372ef700 1 -- 172.21.15.178:0/4161829583 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce300f6510 msgr2=0x7fce300f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:11:34.688 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:34.685+0000 7fce372ef700 1 --2- 172.21.15.178:0/4161829583 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce300f6510 0x7fce300f6930 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7fce20009fb0 tx=0x7fce20009b30).stop 2024-02-11T17:11:34.688 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:34.685+0000 7fce372ef700 1 -- 172.21.15.178:0/4161829583 shutdown_connections 2024-02-11T17:11:34.689 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:34.685+0000 7fce372ef700 1 --2- 172.21.15.178:0/4161829583 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce300f6510 0x7fce300f6930 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:11:34.689 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:34.685+0000 7fce372ef700 1 -- 172.21.15.178:0/4161829583 >> 172.21.15.178:0/4161829583 conn(0x7fce300f1c70 msgr2=0x7fce300f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:11:34.689 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:34.685+0000 7fce372ef700 1 -- 172.21.15.178:0/4161829583 shutdown_connections 2024-02-11T17:11:34.689 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:34.685+0000 7fce372ef700 1 -- 172.21.15.178:0/4161829583 wait complete. 2024-02-11T17:11:34.689 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:34.685+0000 7fce372ef700 1 Processor -- start 2024-02-11T17:11:34.689 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:34.685+0000 7fce372ef700 1 -- start start 2024-02-11T17:11:34.689 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:34.685+0000 7fce372ef700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce300f6510 0x7fce30106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:11:34.689 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:34.685+0000 7fce372ef700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce30106c20 con 0x7fce300f6510 2024-02-11T17:11:34.689 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:34.686+0000 7fce3508b700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce300f6510 0x7fce30106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:11:34.689 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:34.686+0000 7fce3508b700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce300f6510 0x7fce30106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:50056/0 (socket says 172.21.15.178:50056) 2024-02-11T17:11:34.689 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:34.686+0000 7fce3508b700 1 -- 172.21.15.178:0/3638478379 learned_addr learned my addr 172.21.15.178:0/3638478379 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:11:34.690 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:34.686+0000 7fce3508b700 1 -- 172.21.15.178:0/3638478379 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce20005040 con 0x7fce300f6510 2024-02-11T17:11:34.690 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:34.686+0000 7fce3508b700 1 --2- 172.21.15.178:0/3638478379 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce300f6510 0x7fce30106730 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7fce20013510 tx=0x7fce2000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:11:34.690 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:34.686+0000 7fce25ffb700 1 -- 172.21.15.178:0/3638478379 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce20009320 con 0x7fce300f6510 2024-02-11T17:11:34.690 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:34.686+0000 7fce372ef700 1 -- 172.21.15.178:0/3638478379 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce30106e20 con 0x7fce300f6510 2024-02-11T17:11:34.690 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:34.686+0000 7fce25ffb700 1 -- 172.21.15.178:0/3638478379 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fce20013f00 con 0x7fce300f6510 2024-02-11T17:11:34.690 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:34.686+0000 7fce25ffb700 1 -- 172.21.15.178:0/3638478379 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce20014d90 con 0x7fce300f6510 2024-02-11T17:11:34.690 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:34.687+0000 7fce372ef700 1 -- 172.21.15.178:0/3638478379 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce301072b0 con 0x7fce300f6510 2024-02-11T17:11:34.692 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:34.687+0000 7fce372ef700 1 -- 172.21.15.178:0/3638478379 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce30045ad0 con 0x7fce300f6510 2024-02-11T17:11:34.692 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:34.687+0000 7fce25ffb700 1 -- 172.21.15.178:0/3638478379 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fce200095f0 con 0x7fce300f6510 2024-02-11T17:11:34.692 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:34.688+0000 7fce25ffb700 1 --2- 172.21.15.178:0/3638478379 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fce1c02ff60 0x7fce1c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:11:34.692 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:34.688+0000 7fce25ffb700 1 -- 172.21.15.178:0/3638478379 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fce20040fc0 con 0x7fce300f6510 2024-02-11T17:11:34.692 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:34.688+0000 7fce3488a700 1 --2- 172.21.15.178:0/3638478379 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fce1c02ff60 0x7fce1c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:11:34.692 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:34.688+0000 7fce3488a700 1 --2- 172.21.15.178:0/3638478379 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fce1c02ff60 0x7fce1c032420 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fce2c0096a0 tx=0x7fce2c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:11:34.694 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:34.692+0000 7fce25ffb700 1 -- 172.21.15.178:0/3638478379 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fce20014420 con 0x7fce300f6510 2024-02-11T17:11:34.898 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:34.897+0000 7fce372ef700 1 -- 172.21.15.178:0/3638478379 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fce300fbf60 con 0x7fce300f6510 2024-02-11T17:11:34.899 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:34.897+0000 7fce25ffb700 1 -- 172.21.15.178:0/3638478379 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fce20010230 con 0x7fce300f6510 2024-02-11T17:11:34.899 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:11:34.900 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:11:34.901 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:34.899+0000 7fce372ef700 1 -- 172.21.15.178:0/3638478379 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fce1c02ff60 msgr2=0x7fce1c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:11:34.901 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:34.899+0000 7fce372ef700 1 --2- 172.21.15.178:0/3638478379 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fce1c02ff60 0x7fce1c032420 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fce2c0096a0 tx=0x7fce2c006b40).stop 2024-02-11T17:11:34.901 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:34.899+0000 7fce372ef700 1 -- 172.21.15.178:0/3638478379 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce300f6510 msgr2=0x7fce30106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:11:34.902 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:34.899+0000 7fce372ef700 1 --2- 172.21.15.178:0/3638478379 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce300f6510 0x7fce30106730 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7fce20013510 tx=0x7fce2000b040).stop 2024-02-11T17:11:34.902 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:34.899+0000 7fce372ef700 1 -- 172.21.15.178:0/3638478379 shutdown_connections 2024-02-11T17:11:34.902 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:34.899+0000 7fce372ef700 1 --2- 172.21.15.178:0/3638478379 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7fce1c02ff60 0x7fce1c032420 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:11:34.902 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:34.899+0000 7fce372ef700 1 --2- 172.21.15.178:0/3638478379 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce300f6510 0x7fce30106730 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:11:34.902 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:34.899+0000 7fce372ef700 1 -- 172.21.15.178:0/3638478379 >> 172.21.15.178:0/3638478379 conn(0x7fce300f1c70 msgr2=0x7fce300f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:11:34.902 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:34.899+0000 7fce372ef700 1 -- 172.21.15.178:0/3638478379 shutdown_connections 2024-02-11T17:11:34.902 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:34.899+0000 7fce372ef700 1 -- 172.21.15.178:0/3638478379 wait complete. 2024-02-11T17:11:34.902 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:11:36.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:11:35 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:11:34.754831+0000 mgr.smithi037.eofnaq (mgr.14162) 356 : cluster [DBG] pgmap v340: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:11:36.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:11:35 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:11:34.898720+0000 mon.smithi037 (mon.0) 328 : audit [DBG] from='client.? 172.21.15.178:0/3638478379' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:11:36.689 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:11:36.689 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:11:38.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:11:37 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:11:36.755253+0000 mgr.smithi037.eofnaq (mgr.14162) 357 : cluster [DBG] pgmap v341: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:11:38.626 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:38.623+0000 7f805fe92700 1 -- 172.21.15.178:0/2962084874 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f80580f6510 msgr2=0x7f80580f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:11:38.628 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:38.623+0000 7f805fe92700 1 --2- 172.21.15.178:0/2962084874 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f80580f6510 0x7f80580f6930 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7f8048009fb0 tx=0x7f8048009b30).stop 2024-02-11T17:11:38.628 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:38.623+0000 7f805fe92700 1 -- 172.21.15.178:0/2962084874 shutdown_connections 2024-02-11T17:11:38.628 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:38.623+0000 7f805fe92700 1 --2- 172.21.15.178:0/2962084874 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f80580f6510 0x7f80580f6930 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:11:38.628 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:38.623+0000 7f805fe92700 1 -- 172.21.15.178:0/2962084874 >> 172.21.15.178:0/2962084874 conn(0x7f80580f1c70 msgr2=0x7f80580f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:11:38.628 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:38.623+0000 7f805fe92700 1 -- 172.21.15.178:0/2962084874 shutdown_connections 2024-02-11T17:11:38.628 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:38.623+0000 7f805fe92700 1 -- 172.21.15.178:0/2962084874 wait complete. 2024-02-11T17:11:38.628 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:38.624+0000 7f805fe92700 1 Processor -- start 2024-02-11T17:11:38.628 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:38.624+0000 7f805fe92700 1 -- start start 2024-02-11T17:11:38.629 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:38.624+0000 7f805fe92700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f80580f6510 0x7f80581066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:11:38.629 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:38.624+0000 7f805fe92700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8058106bc0 con 0x7f80580f6510 2024-02-11T17:11:38.629 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:38.624+0000 7f805dc2e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f80580f6510 0x7f80581066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:11:38.629 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:38.624+0000 7f805dc2e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f80580f6510 0x7f80581066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:50076/0 (socket says 172.21.15.178:50076) 2024-02-11T17:11:38.629 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:38.624+0000 7f805dc2e700 1 -- 172.21.15.178:0/68185150 learned_addr learned my addr 172.21.15.178:0/68185150 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:11:38.629 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:38.624+0000 7f805dc2e700 1 -- 172.21.15.178:0/68185150 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8048005040 con 0x7f80580f6510 2024-02-11T17:11:38.629 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:38.624+0000 7f805dc2e700 1 --2- 172.21.15.178:0/68185150 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f80580f6510 0x7f80581066d0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f8048013510 tx=0x7f804800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:11:38.629 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:38.625+0000 7f804e7fc700 1 -- 172.21.15.178:0/68185150 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8048009320 con 0x7f80580f6510 2024-02-11T17:11:38.629 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:38.625+0000 7f805fe92700 1 -- 172.21.15.178:0/68185150 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8058106dc0 con 0x7f80580f6510 2024-02-11T17:11:38.629 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:38.625+0000 7f804e7fc700 1 -- 172.21.15.178:0/68185150 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8048013f00 con 0x7f80580f6510 2024-02-11T17:11:38.629 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:38.625+0000 7f804e7fc700 1 -- 172.21.15.178:0/68185150 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8048014d90 con 0x7f80580f6510 2024-02-11T17:11:38.629 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:38.625+0000 7f805fe92700 1 -- 172.21.15.178:0/68185150 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8058107250 con 0x7f80580f6510 2024-02-11T17:11:38.630 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:38.626+0000 7f804e7fc700 1 -- 172.21.15.178:0/68185150 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f80480263a0 con 0x7f80580f6510 2024-02-11T17:11:38.630 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:38.626+0000 7f804e7fc700 1 --2- 172.21.15.178:0/68185150 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f804402ff60 0x7f8044032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:11:38.630 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:38.626+0000 7f804e7fc700 1 -- 172.21.15.178:0/68185150 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8048041080 con 0x7f80580f6510 2024-02-11T17:11:38.630 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:38.626+0000 7f805fe92700 1 -- 172.21.15.178:0/68185150 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8058045ad0 con 0x7f80580f6510 2024-02-11T17:11:38.630 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:38.626+0000 7f805d42d700 1 --2- 172.21.15.178:0/68185150 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f804402ff60 0x7f8044032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:11:38.630 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:38.626+0000 7f805d42d700 1 --2- 172.21.15.178:0/68185150 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f804402ff60 0x7f8044032420 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f80540096a0 tx=0x7f8054006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:11:38.632 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:38.631+0000 7f804e7fc700 1 -- 172.21.15.178:0/68185150 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8048012920 con 0x7f80580f6510 2024-02-11T17:11:38.837 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:38.836+0000 7f805fe92700 1 -- 172.21.15.178:0/68185150 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f80580fc050 con 0x7f80580f6510 2024-02-11T17:11:38.838 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:38.837+0000 7f804e7fc700 1 -- 172.21.15.178:0/68185150 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f804802b020 con 0x7f80580f6510 2024-02-11T17:11:38.838 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:11:38.838 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:11:38.840 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:38.838+0000 7f805fe92700 1 -- 172.21.15.178:0/68185150 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f804402ff60 msgr2=0x7f8044032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:11:38.840 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:38.838+0000 7f805fe92700 1 --2- 172.21.15.178:0/68185150 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f804402ff60 0x7f8044032420 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f80540096a0 tx=0x7f8054006b40).stop 2024-02-11T17:11:38.840 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:38.838+0000 7f805fe92700 1 -- 172.21.15.178:0/68185150 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f80580f6510 msgr2=0x7f80581066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:11:38.840 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:38.838+0000 7f805fe92700 1 --2- 172.21.15.178:0/68185150 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f80580f6510 0x7f80581066d0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f8048013510 tx=0x7f804800b040).stop 2024-02-11T17:11:38.841 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:38.838+0000 7f805fe92700 1 -- 172.21.15.178:0/68185150 shutdown_connections 2024-02-11T17:11:38.841 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:38.838+0000 7f805fe92700 1 --2- 172.21.15.178:0/68185150 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f804402ff60 0x7f8044032420 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:11:38.841 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:38.838+0000 7f805fe92700 1 --2- 172.21.15.178:0/68185150 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f80580f6510 0x7f80581066d0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:11:38.841 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:38.838+0000 7f805fe92700 1 -- 172.21.15.178:0/68185150 >> 172.21.15.178:0/68185150 conn(0x7f80580f1c70 msgr2=0x7f80580f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:11:38.841 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:38.839+0000 7f805fe92700 1 -- 172.21.15.178:0/68185150 shutdown_connections 2024-02-11T17:11:38.841 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:38.839+0000 7f805fe92700 1 -- 172.21.15.178:0/68185150 wait complete. 2024-02-11T17:11:38.841 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:11:40.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:11:39 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:11:38.755742+0000 mgr.smithi037.eofnaq (mgr.14162) 358 : cluster [DBG] pgmap v342: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:11:40.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:11:39 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:11:38.837934+0000 mon.smithi037 (mon.0) 329 : audit [DBG] from='client.? 172.21.15.178:0/68185150' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:11:40.531 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:11:40.532 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:11:41.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:11:41 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:11:40.756232+0000 mgr.smithi037.eofnaq (mgr.14162) 359 : cluster [DBG] pgmap v343: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:11:42.489 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:42.472+0000 7f604e8bf700 1 -- 172.21.15.178:0/3499420682 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f60480f6510 msgr2=0x7f60480f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:11:42.489 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:42.472+0000 7f604e8bf700 1 --2- 172.21.15.178:0/3499420682 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f60480f6510 0x7f60480f6930 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7f6038009fb0 tx=0x7f6038009b30).stop 2024-02-11T17:11:42.489 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:42.472+0000 7f604e8bf700 1 -- 172.21.15.178:0/3499420682 shutdown_connections 2024-02-11T17:11:42.489 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:42.472+0000 7f604e8bf700 1 --2- 172.21.15.178:0/3499420682 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f60480f6510 0x7f60480f6930 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:11:42.489 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:42.472+0000 7f604e8bf700 1 -- 172.21.15.178:0/3499420682 >> 172.21.15.178:0/3499420682 conn(0x7f60480f1c70 msgr2=0x7f60480f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:11:42.489 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:42.472+0000 7f604e8bf700 1 -- 172.21.15.178:0/3499420682 shutdown_connections 2024-02-11T17:11:42.489 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:42.473+0000 7f604e8bf700 1 -- 172.21.15.178:0/3499420682 wait complete. 2024-02-11T17:11:42.489 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:42.473+0000 7f604e8bf700 1 Processor -- start 2024-02-11T17:11:42.489 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:42.473+0000 7f604e8bf700 1 -- start start 2024-02-11T17:11:42.489 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:42.473+0000 7f604e8bf700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f60480f6510 0x7f6048106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:11:42.489 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:42.473+0000 7f604e8bf700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6048106c20 con 0x7f60480f6510 2024-02-11T17:11:42.490 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:42.474+0000 7f6047fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f60480f6510 0x7f6048106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:11:42.490 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:42.474+0000 7f6047fff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f60480f6510 0x7f6048106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:54926/0 (socket says 172.21.15.178:54926) 2024-02-11T17:11:42.490 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:42.474+0000 7f6047fff700 1 -- 172.21.15.178:0/1399637395 learned_addr learned my addr 172.21.15.178:0/1399637395 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:11:42.490 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:42.474+0000 7f6047fff700 1 -- 172.21.15.178:0/1399637395 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6038005040 con 0x7f60480f6510 2024-02-11T17:11:42.490 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:42.474+0000 7f6047fff700 1 --2- 172.21.15.178:0/1399637395 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f60480f6510 0x7f6048106730 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f6038013510 tx=0x7f603800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:11:42.490 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:42.474+0000 7f6044ff9700 1 -- 172.21.15.178:0/1399637395 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6038009320 con 0x7f60480f6510 2024-02-11T17:11:42.490 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:42.474+0000 7f604e8bf700 1 -- 172.21.15.178:0/1399637395 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6048106e20 con 0x7f60480f6510 2024-02-11T17:11:42.490 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:42.474+0000 7f6044ff9700 1 -- 172.21.15.178:0/1399637395 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6038013f00 con 0x7f60480f6510 2024-02-11T17:11:42.490 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:42.474+0000 7f6044ff9700 1 -- 172.21.15.178:0/1399637395 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6038014d90 con 0x7f60480f6510 2024-02-11T17:11:42.490 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:42.474+0000 7f604e8bf700 1 -- 172.21.15.178:0/1399637395 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60481072b0 con 0x7f60480f6510 2024-02-11T17:11:42.490 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:42.475+0000 7f6044ff9700 1 -- 172.21.15.178:0/1399637395 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f60380095f0 con 0x7f60480f6510 2024-02-11T17:11:42.490 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:42.475+0000 7f604e8bf700 1 -- 172.21.15.178:0/1399637395 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6048045ad0 con 0x7f60480f6510 2024-02-11T17:11:42.490 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:42.475+0000 7f6044ff9700 1 --2- 172.21.15.178:0/1399637395 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f603002ff60 0x7f6030032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:11:42.490 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:42.475+0000 7f6044ff9700 1 -- 172.21.15.178:0/1399637395 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6038040fc0 con 0x7f60480f6510 2024-02-11T17:11:42.491 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:42.475+0000 7f60477fe700 1 --2- 172.21.15.178:0/1399637395 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f603002ff60 0x7f6030032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:11:42.491 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:42.476+0000 7f60477fe700 1 --2- 172.21.15.178:0/1399637395 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f603002ff60 0x7f6030032420 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f603c0096a0 tx=0x7f603c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:11:42.491 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:42.480+0000 7f6044ff9700 1 -- 172.21.15.178:0/1399637395 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6038014420 con 0x7f60480f6510 2024-02-11T17:11:42.686 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:42.684+0000 7f604e8bf700 1 -- 172.21.15.178:0/1399637395 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f60480fbf60 con 0x7f60480f6510 2024-02-11T17:11:42.686 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:42.685+0000 7f6044ff9700 1 -- 172.21.15.178:0/1399637395 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6038010230 con 0x7f60480f6510 2024-02-11T17:11:42.687 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:11:42.687 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:11:42.689 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:42.686+0000 7f604e8bf700 1 -- 172.21.15.178:0/1399637395 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f603002ff60 msgr2=0x7f6030032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:11:42.689 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:42.686+0000 7f604e8bf700 1 --2- 172.21.15.178:0/1399637395 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f603002ff60 0x7f6030032420 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f603c0096a0 tx=0x7f603c006b40).stop 2024-02-11T17:11:42.689 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:42.686+0000 7f604e8bf700 1 -- 172.21.15.178:0/1399637395 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f60480f6510 msgr2=0x7f6048106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:11:42.689 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:42.686+0000 7f604e8bf700 1 --2- 172.21.15.178:0/1399637395 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f60480f6510 0x7f6048106730 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f6038013510 tx=0x7f603800b040).stop 2024-02-11T17:11:42.689 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:42.687+0000 7f604e8bf700 1 -- 172.21.15.178:0/1399637395 shutdown_connections 2024-02-11T17:11:42.689 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:42.687+0000 7f604e8bf700 1 --2- 172.21.15.178:0/1399637395 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f603002ff60 0x7f6030032420 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:11:42.689 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:42.687+0000 7f604e8bf700 1 --2- 172.21.15.178:0/1399637395 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f60480f6510 0x7f6048106730 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:11:42.689 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:42.687+0000 7f604e8bf700 1 -- 172.21.15.178:0/1399637395 >> 172.21.15.178:0/1399637395 conn(0x7f60480f1c70 msgr2=0x7f60480f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:11:42.689 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:42.687+0000 7f604e8bf700 1 -- 172.21.15.178:0/1399637395 shutdown_connections 2024-02-11T17:11:42.690 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:42.687+0000 7f604e8bf700 1 -- 172.21.15.178:0/1399637395 wait complete. 2024-02-11T17:11:42.690 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:11:43.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:11:42 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:11:42.685989+0000 mon.smithi037 (mon.0) 330 : audit [DBG] from='client.? 172.21.15.178:0/1399637395' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:11:44.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:11:43 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:11:42.756753+0000 mgr.smithi037.eofnaq (mgr.14162) 360 : cluster [DBG] pgmap v344: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:11:44.470 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:11:44.471 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:11:46.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:11:45 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:11:44.757234+0000 mgr.smithi037.eofnaq (mgr.14162) 361 : cluster [DBG] pgmap v345: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:11:46.480 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:46.478+0000 7f7041dc7700 1 -- 172.21.15.178:0/400257123 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f703c0f62a0 msgr2=0x7f703c0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:11:46.481 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:46.478+0000 7f7041dc7700 1 --2- 172.21.15.178:0/400257123 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f703c0f62a0 0x7f703c0f66c0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7f7030009fb0 tx=0x7f7030009b30).stop 2024-02-11T17:11:46.481 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:46.478+0000 7f7041dc7700 1 -- 172.21.15.178:0/400257123 shutdown_connections 2024-02-11T17:11:46.481 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:46.478+0000 7f7041dc7700 1 --2- 172.21.15.178:0/400257123 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f703c0f62a0 0x7f703c0f66c0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:11:46.481 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:46.478+0000 7f7041dc7700 1 -- 172.21.15.178:0/400257123 >> 172.21.15.178:0/400257123 conn(0x7f703c0f1aa0 msgr2=0x7f703c0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:11:46.481 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:46.478+0000 7f7041dc7700 1 -- 172.21.15.178:0/400257123 shutdown_connections 2024-02-11T17:11:46.481 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:46.478+0000 7f7041dc7700 1 -- 172.21.15.178:0/400257123 wait complete. 2024-02-11T17:11:46.481 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:46.478+0000 7f7041dc7700 1 Processor -- start 2024-02-11T17:11:46.481 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:46.478+0000 7f7041dc7700 1 -- start start 2024-02-11T17:11:46.481 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:46.478+0000 7f7041dc7700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f703c0f62a0 0x7f703c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:11:46.481 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:46.478+0000 7f7041dc7700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f703c069e20 con 0x7f703c0f62a0 2024-02-11T17:11:46.482 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:46.479+0000 7f7040dc5700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f703c0f62a0 0x7f703c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:11:46.482 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:46.479+0000 7f7040dc5700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f703c0f62a0 0x7f703c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:54942/0 (socket says 172.21.15.178:54942) 2024-02-11T17:11:46.482 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:46.479+0000 7f7040dc5700 1 -- 172.21.15.178:0/1890532263 learned_addr learned my addr 172.21.15.178:0/1890532263 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:11:46.482 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:46.479+0000 7f7040dc5700 1 -- 172.21.15.178:0/1890532263 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7030005040 con 0x7f703c0f62a0 2024-02-11T17:11:46.482 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:46.479+0000 7f7040dc5700 1 --2- 172.21.15.178:0/1890532263 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f703c0f62a0 0x7f703c06b9a0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f7030013510 tx=0x7f703000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:11:46.483 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:46.479+0000 7f70397fa700 1 -- 172.21.15.178:0/1890532263 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7030009320 con 0x7f703c0f62a0 2024-02-11T17:11:46.483 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:46.479+0000 7f7041dc7700 1 -- 172.21.15.178:0/1890532263 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f703c069fc0 con 0x7f703c0f62a0 2024-02-11T17:11:46.483 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:46.479+0000 7f70397fa700 1 -- 172.21.15.178:0/1890532263 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7030013f00 con 0x7f703c0f62a0 2024-02-11T17:11:46.483 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:46.479+0000 7f70397fa700 1 -- 172.21.15.178:0/1890532263 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7030014dc0 con 0x7f703c0f62a0 2024-02-11T17:11:46.483 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:46.480+0000 7f7041dc7700 1 -- 172.21.15.178:0/1890532263 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f703c06a450 con 0x7f703c0f62a0 2024-02-11T17:11:46.483 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:46.480+0000 7f70397fa700 1 -- 172.21.15.178:0/1890532263 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f70300095f0 con 0x7f703c0f62a0 2024-02-11T17:11:46.483 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:46.480+0000 7f7041dc7700 1 -- 172.21.15.178:0/1890532263 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f703c059100 con 0x7f703c0f62a0 2024-02-11T17:11:46.484 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:46.481+0000 7f70397fa700 1 --2- 172.21.15.178:0/1890532263 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f702c02ff60 0x7f702c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:11:46.484 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:46.481+0000 7f70397fa700 1 -- 172.21.15.178:0/1890532263 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7030041100 con 0x7f703c0f62a0 2024-02-11T17:11:46.484 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:46.481+0000 7f703bfff700 1 --2- 172.21.15.178:0/1890532263 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f702c02ff60 0x7f702c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:11:46.484 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:46.481+0000 7f703bfff700 1 --2- 172.21.15.178:0/1890532263 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f702c02ff60 0x7f702c032420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f70280096a0 tx=0x7f7028006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:11:46.487 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:46.485+0000 7f70397fa700 1 -- 172.21.15.178:0/1890532263 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7030014420 con 0x7f703c0f62a0 2024-02-11T17:11:46.693 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:46.691+0000 7f7041dc7700 1 -- 172.21.15.178:0/1890532263 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f703c0fbce0 con 0x7f703c0f62a0 2024-02-11T17:11:46.694 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:46.692+0000 7f70397fa700 1 -- 172.21.15.178:0/1890532263 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7030010230 con 0x7f703c0f62a0 2024-02-11T17:11:46.694 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:11:46.694 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:11:46.695 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:46.693+0000 7f7041dc7700 1 -- 172.21.15.178:0/1890532263 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f702c02ff60 msgr2=0x7f702c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:11:46.695 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:46.693+0000 7f7041dc7700 1 --2- 172.21.15.178:0/1890532263 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f702c02ff60 0x7f702c032420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f70280096a0 tx=0x7f7028006b40).stop 2024-02-11T17:11:46.695 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:46.693+0000 7f7041dc7700 1 -- 172.21.15.178:0/1890532263 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f703c0f62a0 msgr2=0x7f703c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:11:46.696 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:46.693+0000 7f7041dc7700 1 --2- 172.21.15.178:0/1890532263 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f703c0f62a0 0x7f703c06b9a0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f7030013510 tx=0x7f703000b040).stop 2024-02-11T17:11:46.696 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:46.693+0000 7f7041dc7700 1 -- 172.21.15.178:0/1890532263 shutdown_connections 2024-02-11T17:11:46.696 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:46.693+0000 7f7041dc7700 1 --2- 172.21.15.178:0/1890532263 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f702c02ff60 0x7f702c032420 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:11:46.696 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:46.693+0000 7f7041dc7700 1 --2- 172.21.15.178:0/1890532263 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f703c0f62a0 0x7f703c06b9a0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:11:46.696 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:46.693+0000 7f7041dc7700 1 -- 172.21.15.178:0/1890532263 >> 172.21.15.178:0/1890532263 conn(0x7f703c0f1aa0 msgr2=0x7f703c0f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:11:46.696 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:46.694+0000 7f7041dc7700 1 -- 172.21.15.178:0/1890532263 shutdown_connections 2024-02-11T17:11:46.696 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:46.694+0000 7f7041dc7700 1 -- 172.21.15.178:0/1890532263 wait complete. 2024-02-11T17:11:46.696 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:11:47.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:11:46 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:11:46.693186+0000 mon.smithi037 (mon.0) 331 : audit [DBG] from='client.? 172.21.15.178:0/1890532263' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:11:48.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:11:47 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:11:46.757610+0000 mgr.smithi037.eofnaq (mgr.14162) 362 : cluster [DBG] pgmap v346: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:11:48.546 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:11:48.546 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:11:50.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:11:49 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:11:48.758083+0000 mgr.smithi037.eofnaq (mgr.14162) 363 : cluster [DBG] pgmap v347: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:11:50.371 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:50.369+0000 7f96096a9700 1 -- 172.21.15.178:0/1692706672 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f96040f5410 msgr2=0x7f96040f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:11:50.371 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:50.369+0000 7f96096a9700 1 --2- 172.21.15.178:0/1692706672 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f96040f5410 0x7f96040f5830 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7f95f8009fb0 tx=0x7f95f8009b30).stop 2024-02-11T17:11:50.371 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:50.369+0000 7f96096a9700 1 -- 172.21.15.178:0/1692706672 shutdown_connections 2024-02-11T17:11:50.372 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:50.369+0000 7f96096a9700 1 --2- 172.21.15.178:0/1692706672 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f96040f5410 0x7f96040f5830 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:11:50.372 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:50.369+0000 7f96096a9700 1 -- 172.21.15.178:0/1692706672 >> 172.21.15.178:0/1692706672 conn(0x7f96040f0b70 msgr2=0x7f96040f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:11:50.372 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:50.369+0000 7f96096a9700 1 -- 172.21.15.178:0/1692706672 shutdown_connections 2024-02-11T17:11:50.372 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:50.369+0000 7f96096a9700 1 -- 172.21.15.178:0/1692706672 wait complete. 2024-02-11T17:11:50.373 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:50.370+0000 7f96096a9700 1 Processor -- start 2024-02-11T17:11:50.373 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:50.370+0000 7f96096a9700 1 -- start start 2024-02-11T17:11:50.373 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:50.370+0000 7f96096a9700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f96040f5410 0x7f9604109420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:11:50.373 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:50.370+0000 7f96096a9700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9604060220 con 0x7f96040f5410 2024-02-11T17:11:50.373 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:50.371+0000 7f9602ffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f96040f5410 0x7f9604109420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:11:50.374 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:50.371+0000 7f9602ffd700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f96040f5410 0x7f9604109420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:47888/0 (socket says 172.21.15.178:47888) 2024-02-11T17:11:50.374 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:50.371+0000 7f9602ffd700 1 -- 172.21.15.178:0/4271356099 learned_addr learned my addr 172.21.15.178:0/4271356099 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:11:50.374 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:50.371+0000 7f9602ffd700 1 -- 172.21.15.178:0/4271356099 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95f8005040 con 0x7f96040f5410 2024-02-11T17:11:50.374 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:50.371+0000 7f9602ffd700 1 --2- 172.21.15.178:0/4271356099 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f96040f5410 0x7f9604109420 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f95f8013510 tx=0x7f95f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:11:50.374 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:50.371+0000 7f95eb7fe700 1 -- 172.21.15.178:0/4271356099 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f95f8009320 con 0x7f96040f5410 2024-02-11T17:11:50.374 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:50.371+0000 7f96096a9700 1 -- 172.21.15.178:0/4271356099 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9604060420 con 0x7f96040f5410 2024-02-11T17:11:50.374 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:50.371+0000 7f95eb7fe700 1 -- 172.21.15.178:0/4271356099 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f95f8013f00 con 0x7f96040f5410 2024-02-11T17:11:50.374 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:50.371+0000 7f95eb7fe700 1 -- 172.21.15.178:0/4271356099 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f95f8014d90 con 0x7f96040f5410 2024-02-11T17:11:50.374 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:50.371+0000 7f96096a9700 1 -- 172.21.15.178:0/4271356099 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9604109ad0 con 0x7f96040f5410 2024-02-11T17:11:50.375 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:50.372+0000 7f95eb7fe700 1 -- 172.21.15.178:0/4271356099 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f95f80095f0 con 0x7f96040f5410 2024-02-11T17:11:50.375 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:50.372+0000 7f96096a9700 1 -- 172.21.15.178:0/4271356099 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9604059100 con 0x7f96040f5410 2024-02-11T17:11:50.375 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:50.372+0000 7f95eb7fe700 1 --2- 172.21.15.178:0/4271356099 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f95ec02ff60 0x7f95ec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:11:50.375 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:50.372+0000 7f95eb7fe700 1 -- 172.21.15.178:0/4271356099 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f95f80410a0 con 0x7f96040f5410 2024-02-11T17:11:50.375 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:50.373+0000 7f96027fc700 1 --2- 172.21.15.178:0/4271356099 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f95ec02ff60 0x7f95ec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:11:50.375 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:50.373+0000 7f96027fc700 1 --2- 172.21.15.178:0/4271356099 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f95ec02ff60 0x7f95ec032420 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f95f40096a0 tx=0x7f95f4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:11:50.379 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:50.377+0000 7f95eb7fe700 1 -- 172.21.15.178:0/4271356099 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f95f801d7f0 con 0x7f96040f5410 2024-02-11T17:11:50.584 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:50.583+0000 7f96096a9700 1 -- 172.21.15.178:0/4271356099 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9604045ad0 con 0x7f96040f5410 2024-02-11T17:11:50.585 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:50.583+0000 7f95eb7fe700 1 -- 172.21.15.178:0/4271356099 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f95f8010230 con 0x7f96040f5410 2024-02-11T17:11:50.585 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:11:50.585 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:11:50.586 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:50.585+0000 7f96096a9700 1 -- 172.21.15.178:0/4271356099 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f95ec02ff60 msgr2=0x7f95ec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:11:50.586 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:50.585+0000 7f96096a9700 1 --2- 172.21.15.178:0/4271356099 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f95ec02ff60 0x7f95ec032420 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f95f40096a0 tx=0x7f95f4006b40).stop 2024-02-11T17:11:50.587 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:50.585+0000 7f96096a9700 1 -- 172.21.15.178:0/4271356099 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f96040f5410 msgr2=0x7f9604109420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:11:50.587 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:50.585+0000 7f96096a9700 1 --2- 172.21.15.178:0/4271356099 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f96040f5410 0x7f9604109420 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f95f8013510 tx=0x7f95f800b040).stop 2024-02-11T17:11:50.587 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:50.585+0000 7f96096a9700 1 -- 172.21.15.178:0/4271356099 shutdown_connections 2024-02-11T17:11:50.587 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:50.585+0000 7f96096a9700 1 --2- 172.21.15.178:0/4271356099 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f95ec02ff60 0x7f95ec032420 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:11:50.587 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:50.585+0000 7f96096a9700 1 --2- 172.21.15.178:0/4271356099 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f96040f5410 0x7f9604109420 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:11:50.587 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:50.585+0000 7f96096a9700 1 -- 172.21.15.178:0/4271356099 >> 172.21.15.178:0/4271356099 conn(0x7f96040f0b70 msgr2=0x7f96040f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:11:50.587 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:50.585+0000 7f96096a9700 1 -- 172.21.15.178:0/4271356099 shutdown_connections 2024-02-11T17:11:50.587 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:50.585+0000 7f96096a9700 1 -- 172.21.15.178:0/4271356099 wait complete. 2024-02-11T17:11:50.587 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:11:51.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:11:51 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:11:50.584521+0000 mon.smithi037 (mon.0) 332 : audit [DBG] from='client.? 172.21.15.178:0/4271356099' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:11:51.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:11:51 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:11:50.758610+0000 mgr.smithi037.eofnaq (mgr.14162) 364 : cluster [DBG] pgmap v348: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:11:52.181 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:11:52.181 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:11:54.086 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:54.065+0000 7f6b41bd2700 1 -- 172.21.15.178:0/1579125302 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6b3c0f51a0 msgr2=0x7f6b3c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:11:54.086 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:54.065+0000 7f6b41bd2700 1 --2- 172.21.15.178:0/1579125302 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6b3c0f51a0 0x7f6b3c0f55c0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7f6b30009fb0 tx=0x7f6b30009b30).stop 2024-02-11T17:11:54.087 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:54.065+0000 7f6b41bd2700 1 -- 172.21.15.178:0/1579125302 shutdown_connections 2024-02-11T17:11:54.087 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:54.065+0000 7f6b41bd2700 1 --2- 172.21.15.178:0/1579125302 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6b3c0f51a0 0x7f6b3c0f55c0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:11:54.087 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:54.065+0000 7f6b41bd2700 1 -- 172.21.15.178:0/1579125302 >> 172.21.15.178:0/1579125302 conn(0x7f6b3c0f09a0 msgr2=0x7f6b3c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:11:54.087 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:54.065+0000 7f6b41bd2700 1 -- 172.21.15.178:0/1579125302 shutdown_connections 2024-02-11T17:11:54.087 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:54.065+0000 7f6b41bd2700 1 -- 172.21.15.178:0/1579125302 wait complete. 2024-02-11T17:11:54.087 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:54.065+0000 7f6b41bd2700 1 Processor -- start 2024-02-11T17:11:54.087 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:54.065+0000 7f6b41bd2700 1 -- start start 2024-02-11T17:11:54.087 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:54.066+0000 7f6b41bd2700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6b3c0f51a0 0x7f6b3c1091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:11:54.087 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:54.066+0000 7f6b41bd2700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b3c060240 con 0x7f6b3c0f51a0 2024-02-11T17:11:54.087 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:54.066+0000 7f6b40bd0700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6b3c0f51a0 0x7f6b3c1091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:11:54.087 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:54.066+0000 7f6b40bd0700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6b3c0f51a0 0x7f6b3c1091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:47906/0 (socket says 172.21.15.178:47906) 2024-02-11T17:11:54.087 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:54.066+0000 7f6b40bd0700 1 -- 172.21.15.178:0/627882579 learned_addr learned my addr 172.21.15.178:0/627882579 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:11:54.087 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:54.066+0000 7f6b40bd0700 1 -- 172.21.15.178:0/627882579 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b30005040 con 0x7f6b3c0f51a0 2024-02-11T17:11:54.087 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:54.066+0000 7f6b40bd0700 1 --2- 172.21.15.178:0/627882579 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6b3c0f51a0 0x7f6b3c1091b0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f6b30013510 tx=0x7f6b3000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:11:54.088 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:54.066+0000 7f6b397fa700 1 -- 172.21.15.178:0/627882579 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b30009320 con 0x7f6b3c0f51a0 2024-02-11T17:11:54.088 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:54.066+0000 7f6b397fa700 1 -- 172.21.15.178:0/627882579 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6b30013f00 con 0x7f6b3c0f51a0 2024-02-11T17:11:54.088 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:54.066+0000 7f6b41bd2700 1 -- 172.21.15.178:0/627882579 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b3c060440 con 0x7f6b3c0f51a0 2024-02-11T17:11:54.088 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:54.067+0000 7f6b397fa700 1 -- 172.21.15.178:0/627882579 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b30014dc0 con 0x7f6b3c0f51a0 2024-02-11T17:11:54.088 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:54.067+0000 7f6b41bd2700 1 -- 172.21.15.178:0/627882579 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b3c109860 con 0x7f6b3c0f51a0 2024-02-11T17:11:54.088 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:54.067+0000 7f6b397fa700 1 -- 172.21.15.178:0/627882579 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6b300095f0 con 0x7f6b3c0f51a0 2024-02-11T17:11:54.088 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:54.067+0000 7f6b41bd2700 1 -- 172.21.15.178:0/627882579 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b3c045ad0 con 0x7f6b3c0f51a0 2024-02-11T17:11:54.088 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:54.068+0000 7f6b397fa700 1 --2- 172.21.15.178:0/627882579 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f6b2c02ff60 0x7f6b2c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:11:54.088 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:54.068+0000 7f6b397fa700 1 -- 172.21.15.178:0/627882579 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6b30041100 con 0x7f6b3c0f51a0 2024-02-11T17:11:54.088 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:54.068+0000 7f6b3bfff700 1 --2- 172.21.15.178:0/627882579 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f6b2c02ff60 0x7f6b2c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:11:54.088 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:54.068+0000 7f6b3bfff700 1 --2- 172.21.15.178:0/627882579 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f6b2c02ff60 0x7f6b2c032420 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f6b280096a0 tx=0x7f6b28006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:11:54.088 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:54.072+0000 7f6b397fa700 1 -- 172.21.15.178:0/627882579 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6b30014420 con 0x7f6b3c0f51a0 2024-02-11T17:11:54.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:11:53 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:11:52.758933+0000 mgr.smithi037.eofnaq (mgr.14162) 365 : cluster [DBG] pgmap v349: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:11:54.277 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:54.275+0000 7f6b41bd2700 1 -- 172.21.15.178:0/627882579 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6b3c10a080 con 0x7f6b3c0f51a0 2024-02-11T17:11:54.278 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:54.276+0000 7f6b397fa700 1 -- 172.21.15.178:0/627882579 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6b30010230 con 0x7f6b3c0f51a0 2024-02-11T17:11:54.278 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:11:54.278 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:11:54.279 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:54.278+0000 7f6b41bd2700 1 -- 172.21.15.178:0/627882579 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f6b2c02ff60 msgr2=0x7f6b2c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:11:54.279 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:54.278+0000 7f6b41bd2700 1 --2- 172.21.15.178:0/627882579 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f6b2c02ff60 0x7f6b2c032420 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f6b280096a0 tx=0x7f6b28006b40).stop 2024-02-11T17:11:54.280 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:54.278+0000 7f6b41bd2700 1 -- 172.21.15.178:0/627882579 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6b3c0f51a0 msgr2=0x7f6b3c1091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:11:54.280 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:54.278+0000 7f6b41bd2700 1 --2- 172.21.15.178:0/627882579 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6b3c0f51a0 0x7f6b3c1091b0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f6b30013510 tx=0x7f6b3000b040).stop 2024-02-11T17:11:54.280 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:54.278+0000 7f6b41bd2700 1 -- 172.21.15.178:0/627882579 shutdown_connections 2024-02-11T17:11:54.280 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:54.278+0000 7f6b41bd2700 1 --2- 172.21.15.178:0/627882579 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f6b2c02ff60 0x7f6b2c032420 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:11:54.280 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:54.278+0000 7f6b41bd2700 1 --2- 172.21.15.178:0/627882579 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6b3c0f51a0 0x7f6b3c1091b0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:11:54.280 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:54.278+0000 7f6b41bd2700 1 -- 172.21.15.178:0/627882579 >> 172.21.15.178:0/627882579 conn(0x7f6b3c0f09a0 msgr2=0x7f6b3c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:11:54.280 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:54.278+0000 7f6b41bd2700 1 -- 172.21.15.178:0/627882579 shutdown_connections 2024-02-11T17:11:54.280 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:54.278+0000 7f6b41bd2700 1 -- 172.21.15.178:0/627882579 wait complete. 2024-02-11T17:11:54.280 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:11:55.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:11:54 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:11:54.277531+0000 mon.smithi037 (mon.0) 333 : audit [DBG] from='client.? 172.21.15.178:0/627882579' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:11:55.914 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:11:55.915 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:11:56.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:11:55 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:11:54.759394+0000 mgr.smithi037.eofnaq (mgr.14162) 366 : cluster [DBG] pgmap v350: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:11:58.002 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:58.001+0000 7f0be53b1700 1 -- 172.21.15.178:0/4232764270 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0be00f51a0 msgr2=0x7f0be00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:11:58.003 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:58.001+0000 7f0be53b1700 1 --2- 172.21.15.178:0/4232764270 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0be00f51a0 0x7f0be00f55c0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7f0bd0009fb0 tx=0x7f0bd0009b30).stop 2024-02-11T17:11:58.003 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:58.001+0000 7f0be53b1700 1 -- 172.21.15.178:0/4232764270 shutdown_connections 2024-02-11T17:11:58.003 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:58.001+0000 7f0be53b1700 1 --2- 172.21.15.178:0/4232764270 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0be00f51a0 0x7f0be00f55c0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:11:58.003 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:58.001+0000 7f0be53b1700 1 -- 172.21.15.178:0/4232764270 >> 172.21.15.178:0/4232764270 conn(0x7f0be00f09a0 msgr2=0x7f0be00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:11:58.003 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:58.001+0000 7f0be53b1700 1 -- 172.21.15.178:0/4232764270 shutdown_connections 2024-02-11T17:11:58.003 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:58.001+0000 7f0be53b1700 1 -- 172.21.15.178:0/4232764270 wait complete. 2024-02-11T17:11:58.005 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:58.001+0000 7f0be53b1700 1 Processor -- start 2024-02-11T17:11:58.005 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:58.002+0000 7f0be53b1700 1 -- start start 2024-02-11T17:11:58.005 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:58.002+0000 7f0be53b1700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0be00f51a0 0x7f0be01053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:11:58.005 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:58.002+0000 7f0be53b1700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0be01058a0 con 0x7f0be00f51a0 2024-02-11T17:11:58.005 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:58.002+0000 7f0bdffff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0be00f51a0 0x7f0be01053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:11:58.006 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:58.002+0000 7f0bdffff700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0be00f51a0 0x7f0be01053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:47924/0 (socket says 172.21.15.178:47924) 2024-02-11T17:11:58.006 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:58.002+0000 7f0bdffff700 1 -- 172.21.15.178:0/3537863028 learned_addr learned my addr 172.21.15.178:0/3537863028 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:11:58.006 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:58.002+0000 7f0bdffff700 1 -- 172.21.15.178:0/3537863028 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bd0005040 con 0x7f0be00f51a0 2024-02-11T17:11:58.006 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:58.002+0000 7f0bdffff700 1 --2- 172.21.15.178:0/3537863028 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0be00f51a0 0x7f0be01053b0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f0bd0013510 tx=0x7f0bd000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:11:58.006 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:58.003+0000 7f0bdcff9700 1 -- 172.21.15.178:0/3537863028 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0bd0009320 con 0x7f0be00f51a0 2024-02-11T17:11:58.006 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:58.003+0000 7f0bdcff9700 1 -- 172.21.15.178:0/3537863028 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0bd0013f00 con 0x7f0be00f51a0 2024-02-11T17:11:58.006 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:58.003+0000 7f0be53b1700 1 -- 172.21.15.178:0/3537863028 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0be0105aa0 con 0x7f0be00f51a0 2024-02-11T17:11:58.006 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:58.003+0000 7f0bdcff9700 1 -- 172.21.15.178:0/3537863028 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0bd0014dc0 con 0x7f0be00f51a0 2024-02-11T17:11:58.006 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:58.003+0000 7f0be53b1700 1 -- 172.21.15.178:0/3537863028 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0be0105f30 con 0x7f0be00f51a0 2024-02-11T17:11:58.006 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:58.003+0000 7f0bdcff9700 1 -- 172.21.15.178:0/3537863028 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0bd0014420 con 0x7f0be00f51a0 2024-02-11T17:11:58.006 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:58.003+0000 7f0be53b1700 1 -- 172.21.15.178:0/3537863028 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0be0045ad0 con 0x7f0be00f51a0 2024-02-11T17:11:58.006 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:58.004+0000 7f0bdcff9700 1 --2- 172.21.15.178:0/3537863028 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f0bc802ff60 0x7f0bc8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:11:58.006 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:58.004+0000 7f0bdcff9700 1 -- 172.21.15.178:0/3537863028 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0bd0041110 con 0x7f0be00f51a0 2024-02-11T17:11:58.006 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:58.004+0000 7f0bdf7fe700 1 --2- 172.21.15.178:0/3537863028 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f0bc802ff60 0x7f0bc8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:11:58.007 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:58.004+0000 7f0bdf7fe700 1 --2- 172.21.15.178:0/3537863028 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f0bc802ff60 0x7f0bc8032420 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f0bd40096a0 tx=0x7f0bd4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:11:58.010 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:58.008+0000 7f0bdcff9700 1 -- 172.21.15.178:0/3537863028 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0bd00095f0 con 0x7f0be00f51a0 2024-02-11T17:11:58.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:11:57 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:11:56.759826+0000 mgr.smithi037.eofnaq (mgr.14162) 367 : cluster [DBG] pgmap v351: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:11:58.218 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:58.216+0000 7f0be53b1700 1 -- 172.21.15.178:0/3537863028 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0be0106de0 con 0x7f0be00f51a0 2024-02-11T17:11:58.218 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:58.217+0000 7f0bdcff9700 1 -- 172.21.15.178:0/3537863028 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0bd001cb10 con 0x7f0be00f51a0 2024-02-11T17:11:58.219 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:11:58.219 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:11:58.221 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:58.218+0000 7f0be53b1700 1 -- 172.21.15.178:0/3537863028 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f0bc802ff60 msgr2=0x7f0bc8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:11:58.221 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:58.218+0000 7f0be53b1700 1 --2- 172.21.15.178:0/3537863028 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f0bc802ff60 0x7f0bc8032420 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f0bd40096a0 tx=0x7f0bd4006b40).stop 2024-02-11T17:11:58.221 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:58.218+0000 7f0be53b1700 1 -- 172.21.15.178:0/3537863028 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0be00f51a0 msgr2=0x7f0be01053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:11:58.221 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:58.218+0000 7f0be53b1700 1 --2- 172.21.15.178:0/3537863028 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0be00f51a0 0x7f0be01053b0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f0bd0013510 tx=0x7f0bd000b040).stop 2024-02-11T17:11:58.221 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:58.218+0000 7f0be53b1700 1 -- 172.21.15.178:0/3537863028 shutdown_connections 2024-02-11T17:11:58.221 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:58.218+0000 7f0be53b1700 1 --2- 172.21.15.178:0/3537863028 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f0bc802ff60 0x7f0bc8032420 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:11:58.221 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:58.219+0000 7f0be53b1700 1 --2- 172.21.15.178:0/3537863028 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0be00f51a0 0x7f0be01053b0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:11:58.221 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:58.219+0000 7f0be53b1700 1 -- 172.21.15.178:0/3537863028 >> 172.21.15.178:0/3537863028 conn(0x7f0be00f09a0 msgr2=0x7f0be00f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:11:58.221 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:58.219+0000 7f0be53b1700 1 -- 172.21.15.178:0/3537863028 shutdown_connections 2024-02-11T17:11:58.221 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:11:58.219+0000 7f0be53b1700 1 -- 172.21.15.178:0/3537863028 wait complete. 2024-02-11T17:11:58.221 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:11:59.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:11:58 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:11:58.218253+0000 mon.smithi037 (mon.0) 334 : audit [DBG] from='client.? 172.21.15.178:0/3537863028' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:11:59.864 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:11:59.865 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:12:00.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:11:59 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:11:58.760280+0000 mgr.smithi037.eofnaq (mgr.14162) 368 : cluster [DBG] pgmap v352: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:12:01.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:12:01 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:12:00.760687+0000 mgr.smithi037.eofnaq (mgr.14162) 369 : cluster [DBG] pgmap v353: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:12:01.766 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:01.763+0000 7f1f76b70700 1 -- 172.21.15.178:0/2518202905 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1f700f51a0 msgr2=0x7f1f700f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:12:01.766 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:01.763+0000 7f1f76b70700 1 --2- 172.21.15.178:0/2518202905 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1f700f51a0 0x7f1f700f55c0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7f1f6c009fb0 tx=0x7f1f6c009b30).stop 2024-02-11T17:12:01.766 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:01.763+0000 7f1f76b70700 1 -- 172.21.15.178:0/2518202905 shutdown_connections 2024-02-11T17:12:01.766 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:01.763+0000 7f1f76b70700 1 --2- 172.21.15.178:0/2518202905 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1f700f51a0 0x7f1f700f55c0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:12:01.766 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:01.763+0000 7f1f76b70700 1 -- 172.21.15.178:0/2518202905 >> 172.21.15.178:0/2518202905 conn(0x7f1f700f09a0 msgr2=0x7f1f700f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:12:01.766 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:01.763+0000 7f1f76b70700 1 -- 172.21.15.178:0/2518202905 shutdown_connections 2024-02-11T17:12:01.767 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:01.763+0000 7f1f76b70700 1 -- 172.21.15.178:0/2518202905 wait complete. 2024-02-11T17:12:01.767 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:01.763+0000 7f1f76b70700 1 Processor -- start 2024-02-11T17:12:01.767 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:01.763+0000 7f1f76b70700 1 -- start start 2024-02-11T17:12:01.767 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:01.763+0000 7f1f76b70700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1f700f51a0 0x7f1f700fd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:12:01.767 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:01.763+0000 7f1f76b70700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f700fb5a0 con 0x7f1f700f51a0 2024-02-11T17:12:01.767 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:01.764+0000 7f1f75b6e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1f700f51a0 0x7f1f700fd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:12:01.767 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:01.764+0000 7f1f75b6e700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1f700f51a0 0x7f1f700fd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:42674/0 (socket says 172.21.15.178:42674) 2024-02-11T17:12:01.767 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:01.764+0000 7f1f75b6e700 1 -- 172.21.15.178:0/205635304 learned_addr learned my addr 172.21.15.178:0/205635304 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:12:01.767 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:01.764+0000 7f1f75b6e700 1 -- 172.21.15.178:0/205635304 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f6c005040 con 0x7f1f700f51a0 2024-02-11T17:12:01.767 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:01.764+0000 7f1f75b6e700 1 --2- 172.21.15.178:0/205635304 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1f700f51a0 0x7f1f700fd120 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f1f6c013510 tx=0x7f1f6c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:12:01.767 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:01.764+0000 7f1f667fc700 1 -- 172.21.15.178:0/205635304 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f6c009320 con 0x7f1f700f51a0 2024-02-11T17:12:01.767 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:01.764+0000 7f1f76b70700 1 -- 172.21.15.178:0/205635304 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f700fb740 con 0x7f1f700f51a0 2024-02-11T17:12:01.767 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:01.764+0000 7f1f667fc700 1 -- 172.21.15.178:0/205635304 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1f6c013f00 con 0x7f1f700f51a0 2024-02-11T17:12:01.768 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:01.764+0000 7f1f667fc700 1 -- 172.21.15.178:0/205635304 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f6c014dc0 con 0x7f1f700f51a0 2024-02-11T17:12:01.768 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:01.764+0000 7f1f76b70700 1 -- 172.21.15.178:0/205635304 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f700fbbd0 con 0x7f1f700f51a0 2024-02-11T17:12:01.769 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:01.765+0000 7f1f667fc700 1 -- 172.21.15.178:0/205635304 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1f6c0095f0 con 0x7f1f700f51a0 2024-02-11T17:12:01.769 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:01.765+0000 7f1f76b70700 1 -- 172.21.15.178:0/205635304 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f70059100 con 0x7f1f700f51a0 2024-02-11T17:12:01.769 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:01.765+0000 7f1f667fc700 1 --2- 172.21.15.178:0/205635304 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f1f5c02ff60 0x7f1f5c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:12:01.769 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:01.765+0000 7f1f667fc700 1 -- 172.21.15.178:0/205635304 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1f6c041100 con 0x7f1f700f51a0 2024-02-11T17:12:01.769 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:01.765+0000 7f1f7536d700 1 --2- 172.21.15.178:0/205635304 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f1f5c02ff60 0x7f1f5c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:12:01.769 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:01.766+0000 7f1f7536d700 1 --2- 172.21.15.178:0/205635304 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f1f5c02ff60 0x7f1f5c032420 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f1f600096a0 tx=0x7f1f60006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:12:01.772 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:01.770+0000 7f1f667fc700 1 -- 172.21.15.178:0/205635304 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1f6c014420 con 0x7f1f700f51a0 2024-02-11T17:12:01.975 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:01.973+0000 7f1f76b70700 1 -- 172.21.15.178:0/205635304 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1f700fabe0 con 0x7f1f700f51a0 2024-02-11T17:12:01.976 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:01.974+0000 7f1f667fc700 1 -- 172.21.15.178:0/205635304 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1f6c010230 con 0x7f1f700f51a0 2024-02-11T17:12:01.976 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:12:01.976 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:12:01.978 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:01.976+0000 7f1f76b70700 1 -- 172.21.15.178:0/205635304 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f1f5c02ff60 msgr2=0x7f1f5c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:12:01.978 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:01.976+0000 7f1f76b70700 1 --2- 172.21.15.178:0/205635304 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f1f5c02ff60 0x7f1f5c032420 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f1f600096a0 tx=0x7f1f60006b40).stop 2024-02-11T17:12:01.978 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:01.976+0000 7f1f76b70700 1 -- 172.21.15.178:0/205635304 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1f700f51a0 msgr2=0x7f1f700fd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:12:01.978 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:01.976+0000 7f1f76b70700 1 --2- 172.21.15.178:0/205635304 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1f700f51a0 0x7f1f700fd120 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f1f6c013510 tx=0x7f1f6c00b040).stop 2024-02-11T17:12:01.978 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:01.976+0000 7f1f76b70700 1 -- 172.21.15.178:0/205635304 shutdown_connections 2024-02-11T17:12:01.978 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:01.976+0000 7f1f76b70700 1 --2- 172.21.15.178:0/205635304 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f1f5c02ff60 0x7f1f5c032420 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:12:01.978 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:01.976+0000 7f1f76b70700 1 --2- 172.21.15.178:0/205635304 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1f700f51a0 0x7f1f700fd120 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:12:01.978 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:01.976+0000 7f1f76b70700 1 -- 172.21.15.178:0/205635304 >> 172.21.15.178:0/205635304 conn(0x7f1f700f09a0 msgr2=0x7f1f7005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:12:01.978 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:01.976+0000 7f1f76b70700 1 -- 172.21.15.178:0/205635304 shutdown_connections 2024-02-11T17:12:01.978 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:01.976+0000 7f1f76b70700 1 -- 172.21.15.178:0/205635304 wait complete. 2024-02-11T17:12:01.979 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:12:02.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:12:02 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:12:01.975514+0000 mon.smithi037 (mon.0) 335 : audit [DBG] from='client.? 172.21.15.178:0/205635304' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:12:03.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:12:03 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:12:02.761012+0000 mgr.smithi037.eofnaq (mgr.14162) 370 : cluster [DBG] pgmap v354: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:12:03.718 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:12:03.719 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:12:04.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:12:04 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:12:03.282432+0000 mon.smithi037 (mon.0) 336 : audit [INF] from='mgr.14162 172.21.15.37:0/2745465012' entity='mgr.smithi037.eofnaq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi037.eofnaq/mirror_snapshot_schedule"}]: dispatch 2024-02-11T17:12:04.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:12:04 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:12:03.284287+0000 mon.smithi037 (mon.0) 337 : audit [INF] from='mgr.14162 172.21.15.37:0/2745465012' entity='mgr.smithi037.eofnaq' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi037.eofnaq/trash_purge_schedule"}]: dispatch 2024-02-11T17:12:05.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:12:05 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:12:04.761486+0000 mgr.smithi037.eofnaq (mgr.14162) 371 : cluster [DBG] pgmap v355: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:12:05.599 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:05.596+0000 7f9258290700 1 -- 172.21.15.178:0/3815147488 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f92500f5410 msgr2=0x7f92500f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:12:05.599 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:05.596+0000 7f9258290700 1 --2- 172.21.15.178:0/3815147488 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f92500f5410 0x7f92500f5830 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7f924c009fb0 tx=0x7f924c009b30).stop 2024-02-11T17:12:05.599 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:05.596+0000 7f9258290700 1 -- 172.21.15.178:0/3815147488 shutdown_connections 2024-02-11T17:12:05.599 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:05.596+0000 7f9258290700 1 --2- 172.21.15.178:0/3815147488 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f92500f5410 0x7f92500f5830 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:12:05.599 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:05.596+0000 7f9258290700 1 -- 172.21.15.178:0/3815147488 >> 172.21.15.178:0/3815147488 conn(0x7f92500f0b70 msgr2=0x7f92500f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:12:05.599 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:05.596+0000 7f9258290700 1 -- 172.21.15.178:0/3815147488 shutdown_connections 2024-02-11T17:12:05.599 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:05.596+0000 7f9258290700 1 -- 172.21.15.178:0/3815147488 wait complete. 2024-02-11T17:12:05.599 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:05.597+0000 7f9258290700 1 Processor -- start 2024-02-11T17:12:05.599 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:05.597+0000 7f9258290700 1 -- start start 2024-02-11T17:12:05.599 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:05.597+0000 7f9258290700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f92500f5410 0x7f9250105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:12:05.599 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:05.597+0000 7f9258290700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9250105b20 con 0x7f92500f5410 2024-02-11T17:12:05.601 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:05.597+0000 7f925602c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f92500f5410 0x7f9250105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:12:05.601 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:05.597+0000 7f925602c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f92500f5410 0x7f9250105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:42692/0 (socket says 172.21.15.178:42692) 2024-02-11T17:12:05.601 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:05.597+0000 7f925602c700 1 -- 172.21.15.178:0/1342168042 learned_addr learned my addr 172.21.15.178:0/1342168042 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:12:05.601 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:05.598+0000 7f925602c700 1 -- 172.21.15.178:0/1342168042 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f924c005040 con 0x7f92500f5410 2024-02-11T17:12:05.601 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:05.598+0000 7f925602c700 1 --2- 172.21.15.178:0/1342168042 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f92500f5410 0x7f9250105630 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f924c013510 tx=0x7f924c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:12:05.602 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:05.598+0000 7f9246ffd700 1 -- 172.21.15.178:0/1342168042 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f924c009320 con 0x7f92500f5410 2024-02-11T17:12:05.602 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:05.598+0000 7f9258290700 1 -- 172.21.15.178:0/1342168042 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9250105d20 con 0x7f92500f5410 2024-02-11T17:12:05.602 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:05.598+0000 7f9246ffd700 1 -- 172.21.15.178:0/1342168042 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f924c013f00 con 0x7f92500f5410 2024-02-11T17:12:05.602 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:05.598+0000 7f9246ffd700 1 -- 172.21.15.178:0/1342168042 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f924c014d90 con 0x7f92500f5410 2024-02-11T17:12:05.602 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:05.598+0000 7f9258290700 1 -- 172.21.15.178:0/1342168042 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92501061b0 con 0x7f92500f5410 2024-02-11T17:12:05.602 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:05.599+0000 7f9246ffd700 1 -- 172.21.15.178:0/1342168042 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f924c0095f0 con 0x7f92500f5410 2024-02-11T17:12:05.602 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:05.599+0000 7f9258290700 1 -- 172.21.15.178:0/1342168042 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9250045ad0 con 0x7f92500f5410 2024-02-11T17:12:05.602 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:05.599+0000 7f9246ffd700 1 --2- 172.21.15.178:0/1342168042 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f923c02ff60 0x7f923c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:12:05.602 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:05.599+0000 7f9246ffd700 1 -- 172.21.15.178:0/1342168042 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f924c041000 con 0x7f92500f5410 2024-02-11T17:12:05.602 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:05.599+0000 7f925582b700 1 --2- 172.21.15.178:0/1342168042 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f923c02ff60 0x7f923c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:12:05.602 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:05.600+0000 7f925582b700 1 --2- 172.21.15.178:0/1342168042 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f923c02ff60 0x7f923c032420 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f92400096a0 tx=0x7f9240006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:12:05.605 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:05.604+0000 7f9246ffd700 1 -- 172.21.15.178:0/1342168042 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f924c014420 con 0x7f92500f5410 2024-02-11T17:12:05.810 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:05.808+0000 7f9258290700 1 -- 172.21.15.178:0/1342168042 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f92500fae60 con 0x7f92500f5410 2024-02-11T17:12:05.811 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:05.809+0000 7f9246ffd700 1 -- 172.21.15.178:0/1342168042 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f924c010230 con 0x7f92500f5410 2024-02-11T17:12:05.811 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:12:05.811 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:12:05.813 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:05.811+0000 7f9258290700 1 -- 172.21.15.178:0/1342168042 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f923c02ff60 msgr2=0x7f923c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:12:05.813 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:05.811+0000 7f9258290700 1 --2- 172.21.15.178:0/1342168042 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f923c02ff60 0x7f923c032420 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f92400096a0 tx=0x7f9240006b40).stop 2024-02-11T17:12:05.813 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:05.811+0000 7f9258290700 1 -- 172.21.15.178:0/1342168042 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f92500f5410 msgr2=0x7f9250105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:12:05.813 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:05.811+0000 7f9258290700 1 --2- 172.21.15.178:0/1342168042 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f92500f5410 0x7f9250105630 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f924c013510 tx=0x7f924c00b040).stop 2024-02-11T17:12:05.813 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:05.811+0000 7f9258290700 1 -- 172.21.15.178:0/1342168042 shutdown_connections 2024-02-11T17:12:05.813 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:05.811+0000 7f9258290700 1 --2- 172.21.15.178:0/1342168042 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f923c02ff60 0x7f923c032420 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:12:05.813 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:05.811+0000 7f9258290700 1 --2- 172.21.15.178:0/1342168042 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f92500f5410 0x7f9250105630 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:12:05.813 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:05.811+0000 7f9258290700 1 -- 172.21.15.178:0/1342168042 >> 172.21.15.178:0/1342168042 conn(0x7f92500f0b70 msgr2=0x7f92500f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:12:05.813 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:05.811+0000 7f9258290700 1 -- 172.21.15.178:0/1342168042 shutdown_connections 2024-02-11T17:12:05.813 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:05.811+0000 7f9258290700 1 -- 172.21.15.178:0/1342168042 wait complete. 2024-02-11T17:12:05.814 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:12:06.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:12:06 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:12:05.810728+0000 mon.smithi037 (mon.0) 338 : audit [DBG] from='client.? 172.21.15.178:0/1342168042' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:12:07.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:12:07 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:12:06.761894+0000 mgr.smithi037.eofnaq (mgr.14162) 372 : cluster [DBG] pgmap v356: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:12:07.565 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:12:07.565 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:12:09.473 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:09.469+0000 7f1f2f10a700 1 -- 172.21.15.178:0/1286717570 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1f280f6510 msgr2=0x7f1f280f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:12:09.473 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:09.469+0000 7f1f2f10a700 1 --2- 172.21.15.178:0/1286717570 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1f280f6510 0x7f1f280f6930 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f1f1c009fb0 tx=0x7f1f1c009b30).stop 2024-02-11T17:12:09.473 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:09.469+0000 7f1f2f10a700 1 -- 172.21.15.178:0/1286717570 shutdown_connections 2024-02-11T17:12:09.473 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:09.469+0000 7f1f2f10a700 1 --2- 172.21.15.178:0/1286717570 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1f280f6510 0x7f1f280f6930 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:12:09.473 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:09.469+0000 7f1f2f10a700 1 -- 172.21.15.178:0/1286717570 >> 172.21.15.178:0/1286717570 conn(0x7f1f280f1c70 msgr2=0x7f1f280f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:12:09.473 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:09.469+0000 7f1f2f10a700 1 -- 172.21.15.178:0/1286717570 shutdown_connections 2024-02-11T17:12:09.473 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:09.469+0000 7f1f2f10a700 1 -- 172.21.15.178:0/1286717570 wait complete. 2024-02-11T17:12:09.473 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:09.470+0000 7f1f2f10a700 1 Processor -- start 2024-02-11T17:12:09.474 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:09.470+0000 7f1f2f10a700 1 -- start start 2024-02-11T17:12:09.474 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:09.470+0000 7f1f2f10a700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1f280f6510 0x7f1f28108910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:12:09.474 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:09.470+0000 7f1f2f10a700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f28108e00 con 0x7f1f280f6510 2024-02-11T17:12:09.474 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:09.470+0000 7f1f2cea6700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1f280f6510 0x7f1f28108910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:12:09.474 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:09.470+0000 7f1f2cea6700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1f280f6510 0x7f1f28108910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:47084/0 (socket says 172.21.15.178:47084) 2024-02-11T17:12:09.474 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:09.470+0000 7f1f2cea6700 1 -- 172.21.15.178:0/1690164674 learned_addr learned my addr 172.21.15.178:0/1690164674 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:12:09.474 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:09.470+0000 7f1f2cea6700 1 -- 172.21.15.178:0/1690164674 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f1c005040 con 0x7f1f280f6510 2024-02-11T17:12:09.474 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:09.471+0000 7f1f2cea6700 1 --2- 172.21.15.178:0/1690164674 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1f280f6510 0x7f1f28108910 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f1f1c013510 tx=0x7f1f1c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:12:09.474 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:09.471+0000 7f1f257fa700 1 -- 172.21.15.178:0/1690164674 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f1c009320 con 0x7f1f280f6510 2024-02-11T17:12:09.474 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:09.471+0000 7f1f2f10a700 1 -- 172.21.15.178:0/1690164674 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f28109000 con 0x7f1f280f6510 2024-02-11T17:12:09.474 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:09.471+0000 7f1f257fa700 1 -- 172.21.15.178:0/1690164674 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1f1c013f00 con 0x7f1f280f6510 2024-02-11T17:12:09.474 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:09.471+0000 7f1f257fa700 1 -- 172.21.15.178:0/1690164674 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f1c014d90 con 0x7f1f280f6510 2024-02-11T17:12:09.475 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:09.471+0000 7f1f2f10a700 1 -- 172.21.15.178:0/1690164674 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f28109490 con 0x7f1f280f6510 2024-02-11T17:12:09.475 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:09.472+0000 7f1f257fa700 1 -- 172.21.15.178:0/1690164674 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1f1c0095f0 con 0x7f1f280f6510 2024-02-11T17:12:09.475 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:09.472+0000 7f1f2f10a700 1 -- 172.21.15.178:0/1690164674 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f28045ad0 con 0x7f1f280f6510 2024-02-11T17:12:09.476 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:09.472+0000 7f1f257fa700 1 --2- 172.21.15.178:0/1690164674 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f1f1802ff60 0x7f1f18032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:12:09.476 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:09.472+0000 7f1f257fa700 1 -- 172.21.15.178:0/1690164674 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1f1c041000 con 0x7f1f280f6510 2024-02-11T17:12:09.476 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:09.472+0000 7f1f27fff700 1 --2- 172.21.15.178:0/1690164674 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f1f1802ff60 0x7f1f18032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:12:09.476 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:09.473+0000 7f1f27fff700 1 --2- 172.21.15.178:0/1690164674 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f1f1802ff60 0x7f1f18032420 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f1f140096a0 tx=0x7f1f14006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:12:09.478 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:09.477+0000 7f1f257fa700 1 -- 172.21.15.178:0/1690164674 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1f1c014420 con 0x7f1f280f6510 2024-02-11T17:12:09.683 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:09.681+0000 7f1f2f10a700 1 -- 172.21.15.178:0/1690164674 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1f2810a340 con 0x7f1f280f6510 2024-02-11T17:12:09.683 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:09.682+0000 7f1f257fa700 1 -- 172.21.15.178:0/1690164674 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1f1c01d7f0 con 0x7f1f280f6510 2024-02-11T17:12:09.684 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:12:09.684 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:12:09.685 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:09.683+0000 7f1f2f10a700 1 -- 172.21.15.178:0/1690164674 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f1f1802ff60 msgr2=0x7f1f18032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:12:09.685 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:09.683+0000 7f1f2f10a700 1 --2- 172.21.15.178:0/1690164674 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f1f1802ff60 0x7f1f18032420 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f1f140096a0 tx=0x7f1f14006b40).stop 2024-02-11T17:12:09.685 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:09.683+0000 7f1f2f10a700 1 -- 172.21.15.178:0/1690164674 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1f280f6510 msgr2=0x7f1f28108910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:12:09.686 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:09.683+0000 7f1f2f10a700 1 --2- 172.21.15.178:0/1690164674 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1f280f6510 0x7f1f28108910 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f1f1c013510 tx=0x7f1f1c00b040).stop 2024-02-11T17:12:09.686 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:09.683+0000 7f1f2f10a700 1 -- 172.21.15.178:0/1690164674 shutdown_connections 2024-02-11T17:12:09.686 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:09.683+0000 7f1f2f10a700 1 --2- 172.21.15.178:0/1690164674 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f1f1802ff60 0x7f1f18032420 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:12:09.686 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:09.683+0000 7f1f2f10a700 1 --2- 172.21.15.178:0/1690164674 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1f280f6510 0x7f1f28108910 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:12:09.686 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:09.683+0000 7f1f2f10a700 1 -- 172.21.15.178:0/1690164674 >> 172.21.15.178:0/1690164674 conn(0x7f1f280f1c70 msgr2=0x7f1f280f2720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:12:09.686 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:09.684+0000 7f1f2f10a700 1 -- 172.21.15.178:0/1690164674 shutdown_connections 2024-02-11T17:12:09.686 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:09.684+0000 7f1f2f10a700 1 -- 172.21.15.178:0/1690164674 wait complete. 2024-02-11T17:12:09.686 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:12:10.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:12:09 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:12:08.762201+0000 mgr.smithi037.eofnaq (mgr.14162) 373 : cluster [DBG] pgmap v357: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:12:10.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:12:09 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:12:09.683080+0000 mon.smithi037 (mon.0) 339 : audit [DBG] from='client.? 172.21.15.178:0/1690164674' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:12:11.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:12:11 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:12:10.762651+0000 mgr.smithi037.eofnaq (mgr.14162) 374 : cluster [DBG] pgmap v358: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:12:11.457 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:12:11.457 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:12:13.378 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:13.375+0000 7f8378c38700 1 -- 172.21.15.178:0/1096683914 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f83740f5410 msgr2=0x7f83740f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:12:13.378 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:13.375+0000 7f8378c38700 1 --2- 172.21.15.178:0/1096683914 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f83740f5410 0x7f83740f5830 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7f8368009fb0 tx=0x7f8368009b30).stop 2024-02-11T17:12:13.378 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:13.375+0000 7f8378c38700 1 -- 172.21.15.178:0/1096683914 shutdown_connections 2024-02-11T17:12:13.379 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:13.375+0000 7f8378c38700 1 --2- 172.21.15.178:0/1096683914 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f83740f5410 0x7f83740f5830 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:12:13.379 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:13.375+0000 7f8378c38700 1 -- 172.21.15.178:0/1096683914 >> 172.21.15.178:0/1096683914 conn(0x7f83740f0b70 msgr2=0x7f83740f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:12:13.379 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:13.375+0000 7f8378c38700 1 -- 172.21.15.178:0/1096683914 shutdown_connections 2024-02-11T17:12:13.379 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:13.375+0000 7f8378c38700 1 -- 172.21.15.178:0/1096683914 wait complete. 2024-02-11T17:12:13.379 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:13.375+0000 7f8378c38700 1 Processor -- start 2024-02-11T17:12:13.379 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:13.375+0000 7f8378c38700 1 -- start start 2024-02-11T17:12:13.379 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:13.375+0000 7f8378c38700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f83740f5410 0x7f83740fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:12:13.379 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:13.375+0000 7f8378c38700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83740fb820 con 0x7f83740f5410 2024-02-11T17:12:13.379 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:13.376+0000 7f837259c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f83740f5410 0x7f83740fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:12:13.379 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:13.376+0000 7f837259c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f83740f5410 0x7f83740fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:47102/0 (socket says 172.21.15.178:47102) 2024-02-11T17:12:13.379 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:13.376+0000 7f837259c700 1 -- 172.21.15.178:0/1921660414 learned_addr learned my addr 172.21.15.178:0/1921660414 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:12:13.379 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:13.376+0000 7f837259c700 1 -- 172.21.15.178:0/1921660414 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8368005040 con 0x7f83740f5410 2024-02-11T17:12:13.379 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:13.376+0000 7f837259c700 1 --2- 172.21.15.178:0/1921660414 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f83740f5410 0x7f83740fd3a0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f8368009f80 tx=0x7f836800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:12:13.379 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:13.376+0000 7f8362ffd700 1 -- 172.21.15.178:0/1921660414 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8368009320 con 0x7f83740f5410 2024-02-11T17:12:13.380 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:13.376+0000 7f8378c38700 1 -- 172.21.15.178:0/1921660414 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83740fb9c0 con 0x7f83740f5410 2024-02-11T17:12:13.380 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:13.376+0000 7f8362ffd700 1 -- 172.21.15.178:0/1921660414 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8368013ec0 con 0x7f83740f5410 2024-02-11T17:12:13.380 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:13.376+0000 7f8362ffd700 1 -- 172.21.15.178:0/1921660414 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f836801ece0 con 0x7f83740f5410 2024-02-11T17:12:13.380 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:13.376+0000 7f8378c38700 1 -- 172.21.15.178:0/1921660414 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83740fbe50 con 0x7f83740f5410 2024-02-11T17:12:13.381 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:13.377+0000 7f8362ffd700 1 -- 172.21.15.178:0/1921660414 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f836801e420 con 0x7f83740f5410 2024-02-11T17:12:13.381 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:13.377+0000 7f8378c38700 1 -- 172.21.15.178:0/1921660414 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8374059100 con 0x7f83740f5410 2024-02-11T17:12:13.381 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:13.377+0000 7f8362ffd700 1 --2- 172.21.15.178:0/1921660414 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f835c02ff60 0x7f835c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:12:13.381 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:13.377+0000 7f8362ffd700 1 -- 172.21.15.178:0/1921660414 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8368040df0 con 0x7f83740f5410 2024-02-11T17:12:13.381 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:13.377+0000 7f8371d9b700 1 --2- 172.21.15.178:0/1921660414 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f835c02ff60 0x7f835c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:12:13.381 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:13.378+0000 7f8371d9b700 1 --2- 172.21.15.178:0/1921660414 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f835c02ff60 0x7f835c032420 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f83640096a0 tx=0x7f8364006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:12:13.384 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:13.383+0000 7f8362ffd700 1 -- 172.21.15.178:0/1921660414 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f836801e730 con 0x7f83740f5410 2024-02-11T17:12:13.590 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:13.590+0000 7f8378c38700 1 -- 172.21.15.178:0/1921660414 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f83740fae60 con 0x7f83740f5410 2024-02-11T17:12:13.591 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:13.590+0000 7f8362ffd700 1 -- 172.21.15.178:0/1921660414 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8368004490 con 0x7f83740f5410 2024-02-11T17:12:13.591 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:12:13.591 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:12:13.593 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:13.592+0000 7f8378c38700 1 -- 172.21.15.178:0/1921660414 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f835c02ff60 msgr2=0x7f835c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:12:13.593 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:13.592+0000 7f8378c38700 1 --2- 172.21.15.178:0/1921660414 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f835c02ff60 0x7f835c032420 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f83640096a0 tx=0x7f8364006b40).stop 2024-02-11T17:12:13.593 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:13.592+0000 7f8378c38700 1 -- 172.21.15.178:0/1921660414 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f83740f5410 msgr2=0x7f83740fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:12:13.593 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:13.592+0000 7f8378c38700 1 --2- 172.21.15.178:0/1921660414 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f83740f5410 0x7f83740fd3a0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f8368009f80 tx=0x7f836800b040).stop 2024-02-11T17:12:13.593 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:13.592+0000 7f8378c38700 1 -- 172.21.15.178:0/1921660414 shutdown_connections 2024-02-11T17:12:13.593 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:13.592+0000 7f8378c38700 1 --2- 172.21.15.178:0/1921660414 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f835c02ff60 0x7f835c032420 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:12:13.593 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:13.592+0000 7f8378c38700 1 --2- 172.21.15.178:0/1921660414 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f83740f5410 0x7f83740fd3a0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:12:13.593 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:13.592+0000 7f8378c38700 1 -- 172.21.15.178:0/1921660414 >> 172.21.15.178:0/1921660414 conn(0x7f83740f0b70 msgr2=0x7f837405fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:12:13.593 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:13.592+0000 7f8378c38700 1 -- 172.21.15.178:0/1921660414 shutdown_connections 2024-02-11T17:12:13.593 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:13.592+0000 7f8378c38700 1 -- 172.21.15.178:0/1921660414 wait complete. 2024-02-11T17:12:13.594 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:12:14.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:12:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:12:12.762894+0000 mgr.smithi037.eofnaq (mgr.14162) 375 : cluster [DBG] pgmap v359: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:12:14.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:12:13 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:12:13.590695+0000 mon.smithi037 (mon.0) 340 : audit [DBG] from='client.? 172.21.15.178:0/1921660414' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:12:15.476 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:12:15.477 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:12:16.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:12:15 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:12:14.763375+0000 mgr.smithi037.eofnaq (mgr.14162) 376 : cluster [DBG] pgmap v360: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:12:17.405 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:17.403+0000 7facfa1b9700 1 -- 172.21.15.178:0/1660928869 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7facf40f7ac0 msgr2=0x7facf40f7ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:12:17.406 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:17.403+0000 7facfa1b9700 1 --2- 172.21.15.178:0/1660928869 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7facf40f7ac0 0x7facf40f7ee0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7face4009fb0 tx=0x7face4009b30).stop 2024-02-11T17:12:17.406 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:17.403+0000 7facfa1b9700 1 -- 172.21.15.178:0/1660928869 shutdown_connections 2024-02-11T17:12:17.406 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:17.403+0000 7facfa1b9700 1 --2- 172.21.15.178:0/1660928869 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7facf40f7ac0 0x7facf40f7ee0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:12:17.406 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:17.403+0000 7facfa1b9700 1 -- 172.21.15.178:0/1660928869 >> 172.21.15.178:0/1660928869 conn(0x7facf40f0260 msgr2=0x7facf40f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:12:17.406 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:17.404+0000 7facfa1b9700 1 -- 172.21.15.178:0/1660928869 shutdown_connections 2024-02-11T17:12:17.406 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:17.404+0000 7facfa1b9700 1 -- 172.21.15.178:0/1660928869 wait complete. 2024-02-11T17:12:17.406 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:17.404+0000 7facfa1b9700 1 Processor -- start 2024-02-11T17:12:17.406 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:17.404+0000 7facfa1b9700 1 -- start start 2024-02-11T17:12:17.406 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:17.404+0000 7facfa1b9700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7facf40f7ac0 0x7facf41066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:12:17.406 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:17.404+0000 7facfa1b9700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facf4106bd0 con 0x7facf40f7ac0 2024-02-11T17:12:17.406 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:17.405+0000 7facf37fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7facf40f7ac0 0x7facf41066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:12:17.406 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:17.405+0000 7facf37fe700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7facf40f7ac0 0x7facf41066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:47124/0 (socket says 172.21.15.178:47124) 2024-02-11T17:12:17.406 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:17.405+0000 7facf37fe700 1 -- 172.21.15.178:0/1867660741 learned_addr learned my addr 172.21.15.178:0/1867660741 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:12:17.407 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:17.405+0000 7facf37fe700 1 -- 172.21.15.178:0/1867660741 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7face4005040 con 0x7facf40f7ac0 2024-02-11T17:12:17.407 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:17.405+0000 7facf37fe700 1 --2- 172.21.15.178:0/1867660741 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7facf40f7ac0 0x7facf41066e0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7face4013510 tx=0x7face400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:12:17.408 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:17.405+0000 7facdbfff700 1 -- 172.21.15.178:0/1867660741 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7face4009320 con 0x7facf40f7ac0 2024-02-11T17:12:17.408 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:17.405+0000 7facfa1b9700 1 -- 172.21.15.178:0/1867660741 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facf4106dd0 con 0x7facf40f7ac0 2024-02-11T17:12:17.408 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:17.405+0000 7facdbfff700 1 -- 172.21.15.178:0/1867660741 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7face4013f00 con 0x7facf40f7ac0 2024-02-11T17:12:17.408 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:17.405+0000 7facdbfff700 1 -- 172.21.15.178:0/1867660741 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7face4014d90 con 0x7facf40f7ac0 2024-02-11T17:12:17.408 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:17.405+0000 7facfa1b9700 1 -- 172.21.15.178:0/1867660741 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facf4107260 con 0x7facf40f7ac0 2024-02-11T17:12:17.408 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:17.406+0000 7facdbfff700 1 -- 172.21.15.178:0/1867660741 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7face4014420 con 0x7facf40f7ac0 2024-02-11T17:12:17.408 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:17.406+0000 7facfa1b9700 1 -- 172.21.15.178:0/1867660741 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facf4045ad0 con 0x7facf40f7ac0 2024-02-11T17:12:17.408 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:17.406+0000 7facdbfff700 1 --2- 172.21.15.178:0/1867660741 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7facdc02ff60 0x7facdc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:12:17.408 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:17.406+0000 7facdbfff700 1 -- 172.21.15.178:0/1867660741 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7face4041110 con 0x7facf40f7ac0 2024-02-11T17:12:17.408 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:17.407+0000 7facf2ffd700 1 --2- 172.21.15.178:0/1867660741 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7facdc02ff60 0x7facdc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:12:17.408 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:17.407+0000 7facf2ffd700 1 --2- 172.21.15.178:0/1867660741 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7facdc02ff60 0x7facdc032420 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7face80096a0 tx=0x7face8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:12:17.412 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:17.411+0000 7facdbfff700 1 -- 172.21.15.178:0/1867660741 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7face40095f0 con 0x7facf40f7ac0 2024-02-11T17:12:17.616 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:17.616+0000 7facfa1b9700 1 -- 172.21.15.178:0/1867660741 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7facf4108110 con 0x7facf40f7ac0 2024-02-11T17:12:17.617 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:17.616+0000 7facdbfff700 1 -- 172.21.15.178:0/1867660741 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7face401db70 con 0x7facf40f7ac0 2024-02-11T17:12:17.617 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:12:17.617 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:12:17.618 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:17.618+0000 7facfa1b9700 1 -- 172.21.15.178:0/1867660741 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7facdc02ff60 msgr2=0x7facdc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:12:17.619 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:17.618+0000 7facfa1b9700 1 --2- 172.21.15.178:0/1867660741 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7facdc02ff60 0x7facdc032420 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7face80096a0 tx=0x7face8006b40).stop 2024-02-11T17:12:17.619 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:17.618+0000 7facfa1b9700 1 -- 172.21.15.178:0/1867660741 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7facf40f7ac0 msgr2=0x7facf41066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:12:17.619 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:17.618+0000 7facfa1b9700 1 --2- 172.21.15.178:0/1867660741 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7facf40f7ac0 0x7facf41066e0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7face4013510 tx=0x7face400b040).stop 2024-02-11T17:12:17.619 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:17.618+0000 7facfa1b9700 1 -- 172.21.15.178:0/1867660741 shutdown_connections 2024-02-11T17:12:17.619 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:17.618+0000 7facfa1b9700 1 --2- 172.21.15.178:0/1867660741 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7facdc02ff60 0x7facdc032420 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:12:17.619 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:17.618+0000 7facfa1b9700 1 --2- 172.21.15.178:0/1867660741 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7facf40f7ac0 0x7facf41066e0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:12:17.619 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:17.618+0000 7facfa1b9700 1 -- 172.21.15.178:0/1867660741 >> 172.21.15.178:0/1867660741 conn(0x7facf40f0260 msgr2=0x7facf40f0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:12:17.619 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:17.618+0000 7facfa1b9700 1 -- 172.21.15.178:0/1867660741 shutdown_connections 2024-02-11T17:12:17.619 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:17.618+0000 7facfa1b9700 1 -- 172.21.15.178:0/1867660741 wait complete. 2024-02-11T17:12:17.619 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:12:18.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:12:17 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:12:16.763796+0000 mgr.smithi037.eofnaq (mgr.14162) 377 : cluster [DBG] pgmap v361: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:12:18.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:12:17 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:12:17.616620+0000 mon.smithi037 (mon.0) 341 : audit [DBG] from='client.? 172.21.15.178:0/1867660741' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:12:19.388 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-11T17:12:19.388 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 -- ceph mon dump -f json 2024-02-11T17:12:20.183 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:12:19 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:12:18.764074+0000 mgr.smithi037.eofnaq (mgr.14162) 378 : cluster [DBG] pgmap v362: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:12:21.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:12:21 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: cluster 2024-02-11T17:12:20.764493+0000 mgr.smithi037.eofnaq (mgr.14162) 379 : cluster [DBG] pgmap v363: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-11T17:12:21.451 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:21.448+0000 7f85888b8700 1 -- 172.21.15.178:0/2651472038 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f85840f5410 msgr2=0x7f85840f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:12:21.451 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:21.448+0000 7f85888b8700 1 --2- 172.21.15.178:0/2651472038 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f85840f5410 0x7f85840f5830 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7f856c009fb0 tx=0x7f856c009b30).stop 2024-02-11T17:12:21.451 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:21.449+0000 7f85888b8700 1 -- 172.21.15.178:0/2651472038 shutdown_connections 2024-02-11T17:12:21.451 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:21.449+0000 7f85888b8700 1 --2- 172.21.15.178:0/2651472038 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f85840f5410 0x7f85840f5830 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:12:21.451 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:21.449+0000 7f85888b8700 1 -- 172.21.15.178:0/2651472038 >> 172.21.15.178:0/2651472038 conn(0x7f85840f0b70 msgr2=0x7f85840f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:12:21.451 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:21.449+0000 7f85888b8700 1 -- 172.21.15.178:0/2651472038 shutdown_connections 2024-02-11T17:12:21.451 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:21.449+0000 7f85888b8700 1 -- 172.21.15.178:0/2651472038 wait complete. 2024-02-11T17:12:21.451 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:21.449+0000 7f85888b8700 1 Processor -- start 2024-02-11T17:12:21.451 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:21.449+0000 7f85888b8700 1 -- start start 2024-02-11T17:12:21.451 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:21.449+0000 7f85888b8700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f85840f5410 0x7f85840fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:12:21.452 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:21.449+0000 7f85888b8700 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85840fb820 con 0x7f85840f5410 2024-02-11T17:12:21.453 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:21.450+0000 7f858259c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f85840f5410 0x7f85840fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:12:21.453 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:21.450+0000 7f858259c700 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f85840f5410 0x7f85840fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.178:52060/0 (socket says 172.21.15.178:52060) 2024-02-11T17:12:21.453 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:21.450+0000 7f858259c700 1 -- 172.21.15.178:0/1163270633 learned_addr learned my addr 172.21.15.178:0/1163270633 (peer_addr_for_me v2:172.21.15.178:0/0) 2024-02-11T17:12:21.453 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:21.450+0000 7f858259c700 1 -- 172.21.15.178:0/1163270633 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f856c005040 con 0x7f85840f5410 2024-02-11T17:12:21.453 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:21.450+0000 7f858259c700 1 --2- 172.21.15.178:0/1163270633 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f85840f5410 0x7f85840fd3a0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f856c009f80 tx=0x7f856c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:12:21.454 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:21.450+0000 7f857affd700 1 -- 172.21.15.178:0/1163270633 <== mon.0 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f856c009320 con 0x7f85840f5410 2024-02-11T17:12:21.454 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:21.450+0000 7f85888b8700 1 -- 172.21.15.178:0/1163270633 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85840fb9c0 con 0x7f85840f5410 2024-02-11T17:12:21.454 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:21.450+0000 7f857affd700 1 -- 172.21.15.178:0/1163270633 <== mon.0 v2:172.21.15.37:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f856c013ec0 con 0x7f85840f5410 2024-02-11T17:12:21.454 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:21.450+0000 7f857affd700 1 -- 172.21.15.178:0/1163270633 <== mon.0 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f856c01ece0 con 0x7f85840f5410 2024-02-11T17:12:21.454 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:21.450+0000 7f85888b8700 1 -- 172.21.15.178:0/1163270633 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85840fbe50 con 0x7f85840f5410 2024-02-11T17:12:21.454 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:21.451+0000 7f857affd700 1 -- 172.21.15.178:0/1163270633 <== mon.0 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f856c01e420 con 0x7f85840f5410 2024-02-11T17:12:21.454 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:21.451+0000 7f85888b8700 1 -- 172.21.15.178:0/1163270633 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8584059100 con 0x7f85840f5410 2024-02-11T17:12:21.454 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:21.451+0000 7f857affd700 1 --2- 172.21.15.178:0/1163270633 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f857002ff60 0x7f8570032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-11T17:12:21.454 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:21.451+0000 7f857affd700 1 -- 172.21.15.178:0/1163270633 <== mon.0 v2:172.21.15.37:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f856c040ee0 con 0x7f85840f5410 2024-02-11T17:12:21.454 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:21.452+0000 7f8581d9b700 1 --2- 172.21.15.178:0/1163270633 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f857002ff60 0x7f8570032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-11T17:12:21.455 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:21.452+0000 7f8581d9b700 1 --2- 172.21.15.178:0/1163270633 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f857002ff60 0x7f8570032420 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f85740096a0 tx=0x7f8574006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-11T17:12:21.458 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:21.456+0000 7f857affd700 1 -- 172.21.15.178:0/1163270633 <== mon.0 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f856c01e6f0 con 0x7f85840f5410 2024-02-11T17:12:21.666 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:21.665+0000 7f85888b8700 1 -- 172.21.15.178:0/1163270633 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f85840fae60 con 0x7f85840f5410 2024-02-11T17:12:21.667 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:21.666+0000 7f857affd700 1 -- 172.21.15.178:0/1163270633 <== mon.0 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f856c02c2d0 con 0x7f85840f5410 2024-02-11T17:12:21.667 INFO:teuthology.orchestra.run.smithi178.stdout: 2024-02-11T17:12:21.667 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"c65cd888-c8fe-11ee-95b9-87774f69a715","modified":"2024-02-11T16:58:56.764219Z","created":"2024-02-11T16:58:56.764219Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi037","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:3300","nonce":0},{"type":"v1","addr":"172.21.15.37:6789","nonce":0}]},"addr":"172.21.15.37:6789/0","public_addr":"172.21.15.37:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-11T17:12:21.669 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:21.667+0000 7f85888b8700 1 -- 172.21.15.178:0/1163270633 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f857002ff60 msgr2=0x7f8570032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:12:21.669 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:21.667+0000 7f85888b8700 1 --2- 172.21.15.178:0/1163270633 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f857002ff60 0x7f8570032420 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f85740096a0 tx=0x7f8574006b40).stop 2024-02-11T17:12:21.669 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:21.667+0000 7f85888b8700 1 -- 172.21.15.178:0/1163270633 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f85840f5410 msgr2=0x7f85840fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-11T17:12:21.669 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:21.667+0000 7f85888b8700 1 --2- 172.21.15.178:0/1163270633 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f85840f5410 0x7f85840fd3a0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f856c009f80 tx=0x7f856c00b040).stop 2024-02-11T17:12:21.669 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:21.667+0000 7f85888b8700 1 -- 172.21.15.178:0/1163270633 shutdown_connections 2024-02-11T17:12:21.669 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:21.667+0000 7f85888b8700 1 --2- 172.21.15.178:0/1163270633 >> [v2:172.21.15.37:6800/3714037575,v1:172.21.15.37:6801/3714037575] conn(0x7f857002ff60 0x7f8570032420 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:12:21.670 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:21.667+0000 7f85888b8700 1 --2- 172.21.15.178:0/1163270633 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f85840f5410 0x7f85840fd3a0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-11T17:12:21.670 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:21.667+0000 7f85888b8700 1 -- 172.21.15.178:0/1163270633 >> 172.21.15.178:0/1163270633 conn(0x7f85840f0b70 msgr2=0x7f858405fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-11T17:12:21.670 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:21.667+0000 7f85888b8700 1 -- 172.21.15.178:0/1163270633 shutdown_connections 2024-02-11T17:12:21.670 INFO:teuthology.orchestra.run.smithi178.stderr:2024-02-11T17:12:21.667+0000 7f85888b8700 1 -- 172.21.15.178:0/1163270633 wait complete. 2024-02-11T17:12:21.670 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2024-02-11T17:12:22.413 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_5db25abe00e218b6920215c0bef70af7e47c7152/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds 2024-02-11T17:12:22.415 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-02-11T17:12:22.415 DEBUG:teuthology.orchestra.run.smithi037:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-11T17:12:22.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:12:22 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: audit 2024-02-11T17:12:21.666850+0000 mon.smithi037 (mon.0) 342 : audit [DBG] from='client.? 172.21.15.178:0/1163270633' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-11T17:12:22.440 DEBUG:teuthology.orchestra.run.smithi178:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-11T17:12:22.459 INFO:tasks.cephadm:Stopping all daemons... 2024-02-11T17:12:22.459 INFO:tasks.cephadm.mon.smithi037:Stopping mon.smithi037... 2024-02-11T17:12:22.459 DEBUG:teuthology.orchestra.run.smithi037:> sudo systemctl stop ceph-c65cd888-c8fe-11ee-95b9-87774f69a715@mon.smithi037 2024-02-11T17:12:22.933 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:12:22 smithi037 systemd[1]: Stopping Ceph mon.smithi037 for c65cd888-c8fe-11ee-95b9-87774f69a715... 2024-02-11T17:12:22.933 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:12:22 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T17:12:22.652+0000 7f9066364700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi037 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-02-11T17:12:22.933 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:12:22 smithi037 ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037[39355]: debug 2024-02-11T17:12:22.652+0000 7f9066364700 -1 mon.smithi037@0(leader) e1 *** Got Signal Terminated *** 2024-02-11T17:12:23.433 INFO:journalctl@ceph.mon.smithi037.smithi037.stdout:Feb 11 17:12:23 smithi037 podman[45328]: ceph-c65cd888-c8fe-11ee-95b9-87774f69a715-mon.smithi037 2024-02-11T17:12:23.657 DEBUG:teuthology.orchestra.run.smithi037:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c65cd888-c8fe-11ee-95b9-87774f69a715@mon.smithi037.service' 2024-02-11T17:12:23.746 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-11T17:12:23.747 INFO:tasks.cephadm.mon.smithi037:Stopped mon.smithi037 2024-02-11T17:12:23.747 INFO:tasks.cephadm.mon.smithi178:Stopping mon.smithi178... 2024-02-11T17:12:23.747 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ceph-c65cd888-c8fe-11ee-95b9-87774f69a715@mon.smithi178 2024-02-11T17:12:23.785 INFO:teuthology.orchestra.run.smithi178.stderr:Failed to stop ceph-c65cd888-c8fe-11ee-95b9-87774f69a715@mon.smithi178.service: Unit ceph-c65cd888-c8fe-11ee-95b9-87774f69a715@mon.smithi178.service not loaded. 2024-02-11T17:12:23.786 DEBUG:teuthology.orchestra.run:got remote process result: 5 2024-02-11T17:12:23.787 ERROR:tasks.cephadm:Failed to stop "ceph.mon.smithi178" Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_5db25abe00e218b6920215c0bef70af7e47c7152/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_5db25abe00e218b6920215c0bef70af7e47c7152/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_5db25abe00e218b6920215c0bef70af7e47c7152/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi178 with status 5: 'sudo systemctl stop ceph-c65cd888-c8fe-11ee-95b9-87774f69a715@mon.smithi178' 2024-02-11T17:12:23.787 INFO:tasks.cephadm:Archiving crash dumps... 2024-02-11T17:12:23.789 DEBUG:teuthology.misc:Transferring archived files from smithi037:/var/lib/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-11_15:26:36-rados-wip-yuri8-testing-2024-02-07-0820-pacific-distro-default-smithi/7556240/remote/ubuntu@smithi037.front.sepia.ceph.com/crash 2024-02-11T17:12:23.790 DEBUG:teuthology.orchestra.run.smithi037:> sudo tar c -f - -C /var/lib/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715/crash -- . 2024-02-11T17:12:23.832 DEBUG:teuthology.misc:Transferring archived files from smithi178:/var/lib/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-11_15:26:36-rados-wip-yuri8-testing-2024-02-07-0820-pacific-distro-default-smithi/7556240/remote/ubuntu@smithi178.front.sepia.ceph.com/crash 2024-02-11T17:12:23.832 DEBUG:teuthology.orchestra.run.smithi178:> sudo tar c -f - -C /var/lib/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715/crash -- . 2024-02-11T17:12:23.876 INFO:teuthology.orchestra.run.smithi178.stderr:tar: /var/lib/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715/crash: Cannot open: No such file or directory 2024-02-11T17:12:23.876 INFO:teuthology.orchestra.run.smithi178.stderr:tar: Error is not recoverable: exiting now 2024-02-11T17:12:23.877 INFO:tasks.cephadm:Checking cluster log for badness... 2024-02-11T17:12:23.877 DEBUG:teuthology.orchestra.run.smithi037:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-02-11T17:12:23.911 INFO:teuthology.orchestra.run.smithi037.stderr:grep: /var/log/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715/ceph.log: No such file or directory 2024-02-11T17:12:23.913 INFO:tasks.cephadm:Compressing logs... 2024-02-11T17:12:23.913 DEBUG:teuthology.orchestra.run.smithi037:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-11T17:12:23.955 DEBUG:teuthology.orchestra.run.smithi178:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-11T17:12:23.983 INFO:teuthology.orchestra.run.smithi037.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-11T17:12:23.983 INFO:teuthology.orchestra.run.smithi037.stderr:: No such file or directory 2024-02-11T17:12:23.983 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715/ceph-volume.log 2024-02-11T17:12:23.984 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715/ceph-mon.smithi037.log 2024-02-11T17:12:23.984 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715/ceph-mgr.smithi037.eofnaq.log 2024-02-11T17:12:23.984 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715/ceph-mon.smithi037.log: 88.3% -- replaced with /var/log/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715/ceph-volume.log.gz 2024-02-11T17:12:23.985 INFO:teuthology.orchestra.run.smithi178.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-11T17:12:23.985 INFO:teuthology.orchestra.run.smithi178.stderr:: No such file or directory 2024-02-11T17:12:23.987 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715/ceph-volume.log 2024-02-11T17:12:23.987 INFO:teuthology.orchestra.run.smithi178.stderr: 95.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-11T17:12:23.988 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715/ceph-volume.log: 88.4% -- replaced with /var/log/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715/ceph-volume.log.gz 2024-02-11T17:12:23.988 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715/ceph-mgr.smithi037.eofnaq.log: 91.7% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-11T17:12:23.989 INFO:teuthology.orchestra.run.smithi178.stderr: 2024-02-11T17:12:23.990 INFO:teuthology.orchestra.run.smithi178.stderr:real 0m0.015s 2024-02-11T17:12:23.990 INFO:teuthology.orchestra.run.smithi178.stderr:user 0m0.012s 2024-02-11T17:12:23.990 INFO:teuthology.orchestra.run.smithi178.stderr:sys 0m0.013s 2024-02-11T17:12:24.084 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /var/log/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715/ceph-mgr.smithi037.eofnaq.log: file size changed while zipping 2024-02-11T17:12:24.084 INFO:teuthology.orchestra.run.smithi037.stderr: 92.8% -- replaced with /var/log/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715/ceph-mgr.smithi037.eofnaq.log.gz 2024-02-11T17:12:24.161 INFO:teuthology.orchestra.run.smithi037.stderr: 90.8% -- replaced with /var/log/ceph/c65cd888-c8fe-11ee-95b9-87774f69a715/ceph-mon.smithi037.log.gz 2024-02-11T17:12:24.163 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-02-11T17:12:24.163 INFO:teuthology.orchestra.run.smithi037.stderr:real 0m0.191s 2024-02-11T17:12:24.163 INFO:teuthology.orchestra.run.smithi037.stderr:user 0m0.283s 2024-02-11T17:12:24.163 INFO:teuthology.orchestra.run.smithi037.stderr:sys 0m0.024s 2024-02-11T17:12:24.163 INFO:tasks.cephadm:Archiving logs... 2024-02-11T17:12:24.163 DEBUG:teuthology.misc:Transferring archived files from smithi037:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-02-11_15:26:36-rados-wip-yuri8-testing-2024-02-07-0820-pacific-distro-default-smithi/7556240/remote/ubuntu@smithi037.front.sepia.ceph.com/log 2024-02-11T17:12:24.164 DEBUG:teuthology.orchestra.run.smithi037:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-11T17:12:24.285 DEBUG:teuthology.misc:Transferring archived files from smithi178:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-02-11_15:26:36-rados-wip-yuri8-testing-2024-02-07-0820-pacific-distro-default-smithi/7556240/remote/ubuntu@smithi178.front.sepia.ceph.com/log 2024-02-11T17:12:24.286 DEBUG:teuthology.orchestra.run.smithi178:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-11T17:12:24.320 INFO:tasks.cephadm:Removing cluster... 2024-02-11T17:12:24.320 DEBUG:teuthology.orchestra.run.smithi037:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 --force 2024-02-11T17:15:07.626 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c65cd888-c8fe-11ee-95b9-87774f69a715 --force 2024-02-11T17:15:07.939 INFO:tasks.cephadm:Removing cephadm ... 2024-02-11T17:15:07.940 DEBUG:teuthology.orchestra.run.smithi037:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-11T17:15:07.960 DEBUG:teuthology.orchestra.run.smithi178:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-11T17:15:07.978 INFO:tasks.cephadm:Teardown complete 2024-02-11T17:15:07.978 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_5db25abe00e218b6920215c0bef70af7e47c7152/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_5db25abe00e218b6920215c0bef70af7e47c7152/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_5db25abe00e218b6920215c0bef70af7e47c7152/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_5db25abe00e218b6920215c0bef70af7e47c7152/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_5db25abe00e218b6920215c0bef70af7e47c7152/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi178 with status 5: 'sudo systemctl stop ceph-c65cd888-c8fe-11ee-95b9-87774f69a715@mon.smithi178' 2024-02-11T17:15:08.197 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=82dba0d89c784a73ad092a8a6d688db9 Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_5db25abe00e218b6920215c0bef70af7e47c7152/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_5db25abe00e218b6920215c0bef70af7e47c7152/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_5db25abe00e218b6920215c0bef70af7e47c7152/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_5db25abe00e218b6920215c0bef70af7e47c7152/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_5db25abe00e218b6920215c0bef70af7e47c7152/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi178 with status 5: 'sudo systemctl stop ceph-c65cd888-c8fe-11ee-95b9-87774f69a715@mon.smithi178' 2024-02-11T17:15:08.202 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-02-11T17:15:08.212 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-02-11T17:15:08.231 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-02-11T17:15:08.232 DEBUG:teuthology.orchestra.run.smithi037:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-11T17:15:08.247 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-11T17:15:08.321 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-02-11T17:15:08.331 INFO:teuthology.task.clock:Checking final clock skew... 2024-02-11T17:15:08.331 DEBUG:teuthology.orchestra.run.smithi037:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-11T17:15:08.334 DEBUG:teuthology.orchestra.run.smithi178:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-11T17:15:08.350 INFO:teuthology.orchestra.run.smithi178.stderr:bash: ntpq: command not found 2024-02-11T17:15:08.351 INFO:teuthology.orchestra.run.smithi037.stderr:bash: ntpq: command not found 2024-02-11T17:15:08.355 INFO:teuthology.orchestra.run.smithi178.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-11T17:15:08.355 INFO:teuthology.orchestra.run.smithi178.stdout:=============================================================================== 2024-02-11T17:15:08.356 INFO:teuthology.orchestra.run.smithi178.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 30 +1032us[+1032us] +/- 92ms 2024-02-11T17:15:08.356 INFO:teuthology.orchestra.run.smithi178.stdout:^* hv02.front.sepia.ceph.com 3 6 377 31 -1249us[-1252us] +/- 73ms 2024-02-11T17:15:08.356 INFO:teuthology.orchestra.run.smithi178.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 29 +371us[ +371us] +/- 65ms 2024-02-11T17:15:08.356 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-11T17:15:08.356 INFO:teuthology.orchestra.run.smithi037.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-11T17:15:08.356 INFO:teuthology.orchestra.run.smithi037.stdout:=============================================================================== 2024-02-11T17:15:08.356 INFO:teuthology.orchestra.run.smithi037.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 30 +1036us[+1023us] +/- 92ms 2024-02-11T17:15:08.357 INFO:teuthology.orchestra.run.smithi037.stdout:^* hv02.front.sepia.ceph.com 3 6 377 27 -1251us[-1264us] +/- 73ms 2024-02-11T17:15:08.357 INFO:teuthology.orchestra.run.smithi037.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 31 +368us[ +355us] +/- 65ms 2024-02-11T17:15:08.357 INFO:teuthology.orchestra.run.smithi037.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-11T17:15:08.357 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-02-11T17:15:08.368 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-02-11T17:15:08.369 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-02-11T17:15:08.397 DEBUG:teuthology.orchestra.run.smithi037:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-11T17:15:08.462 DEBUG:teuthology.orchestra.run.smithi178:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-11T17:15:08.525 DEBUG:teuthology.orchestra.run.smithi037:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-11T17:15:08.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-11T17:15:08.557 DEBUG:teuthology.orchestra.run.smithi178:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-11T17:15:08.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-11T17:15:08.586 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-02-11T17:15:08.596 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-02-11T17:15:08.624 INFO:teuthology.task.internal:Duration was 1551.619681 seconds 2024-02-11T17:15:08.624 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-02-11T17:15:08.664 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-02-11T17:15:08.664 DEBUG:teuthology.orchestra.run.smithi037:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-11T17:15:08.667 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-11T17:15:08.712 INFO:teuthology.orchestra.run.smithi178.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-11T17:15:08.713 INFO:teuthology.orchestra.run.smithi037.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-11T17:15:08.946 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-02-11T17:15:08.946 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi037.front.sepia.ceph.com 2024-02-11T17:15:08.946 DEBUG:teuthology.orchestra.run.smithi037:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-11T17:15:08.970 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi178.front.sepia.ceph.com 2024-02-11T17:15:08.970 DEBUG:teuthology.orchestra.run.smithi178:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-11T17:15:08.992 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-02-11T17:15:08.992 DEBUG:teuthology.orchestra.run.smithi037:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-11T17:15:09.012 DEBUG:teuthology.orchestra.run.smithi178:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-11T17:15:09.082 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-02-11T17:15:09.082 DEBUG:teuthology.orchestra.run.smithi037:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-11T17:15:09.084 DEBUG:teuthology.orchestra.run.smithi178:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-11T17:15:09.174 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-02-11T17:15:09.184 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-02-11T17:15:09.184 DEBUG:teuthology.orchestra.run.smithi037:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-11T17:15:09.211 DEBUG:teuthology.orchestra.run.smithi178:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-11T17:15:09.238 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-02-11T17:15:09.249 DEBUG:teuthology.orchestra.run.smithi037:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-11T17:15:09.254 DEBUG:teuthology.orchestra.run.smithi178:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-11T17:15:09.279 INFO:teuthology.orchestra.run.smithi037.stdout:kernel.core_pattern = core 2024-02-11T17:15:09.304 INFO:teuthology.orchestra.run.smithi178.stdout:kernel.core_pattern = core 2024-02-11T17:15:09.329 DEBUG:teuthology.orchestra.run.smithi037:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-11T17:15:09.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-11T17:15:09.354 DEBUG:teuthology.orchestra.run.smithi178:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-11T17:15:09.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-11T17:15:09.386 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-02-11T17:15:09.398 INFO:teuthology.task.internal:Transferring archived files... 2024-02-11T17:15:09.398 DEBUG:teuthology.misc:Transferring archived files from smithi037:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-11_15:26:36-rados-wip-yuri8-testing-2024-02-07-0820-pacific-distro-default-smithi/7556240/remote/smithi037 2024-02-11T17:15:09.399 DEBUG:teuthology.orchestra.run.smithi037:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-11T17:15:09.440 DEBUG:teuthology.misc:Transferring archived files from smithi178:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-11_15:26:36-rados-wip-yuri8-testing-2024-02-07-0820-pacific-distro-default-smithi/7556240/remote/smithi178 2024-02-11T17:15:09.441 DEBUG:teuthology.orchestra.run.smithi178:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-11T17:15:09.481 INFO:teuthology.task.internal:Removing archive directory... 2024-02-11T17:15:09.481 DEBUG:teuthology.orchestra.run.smithi037:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-11T17:15:09.483 DEBUG:teuthology.orchestra.run.smithi178:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-11T17:15:09.531 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-02-11T17:15:09.542 INFO:teuthology.task.internal:Not uploading archives. 2024-02-11T17:15:09.542 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-02-11T17:15:09.607 INFO:teuthology.task.internal:Tidying up after the test... 2024-02-11T17:15:09.608 DEBUG:teuthology.orchestra.run.smithi037:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-11T17:15:09.610 DEBUG:teuthology.orchestra.run.smithi178:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-11T17:15:09.626 INFO:teuthology.orchestra.run.smithi037.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 11 17:15 /home/ubuntu/cephtest 2024-02-11T17:15:09.628 INFO:teuthology.orchestra.run.smithi178.stdout: 525357 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 11 17:15 /home/ubuntu/cephtest 2024-02-11T17:15:09.629 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-02-11T17:15:09.639 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-02-11T17:15:09.739 INFO:teuthology.nuke:Checking targets against current locks 2024-02-11T17:15:09.765 DEBUG:teuthology.nuke:shortname: smithi037 2024-02-11T17:15:09.766 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-11T17:15:09.792 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi037.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-11_15:26:36-rados-wip-yuri8-testing-2024-02-07-0820-pacific-distro-default-smithi/7556240', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-11 16:40:31.448701', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKvJIQHP1Zs60NA9NBl5unD1csME7/q5BnQh7W9nrh82Zli5jZdvbRPSf6Z5S1PSAzAueP0oWi0nyOKITw1FfHE='} 2024-02-11T17:15:09.793 DEBUG:teuthology.nuke:shortname: smithi178 2024-02-11T17:15:09.793 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-11T17:15:09.821 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi178.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-11_15:26:36-rados-wip-yuri8-testing-2024-02-07-0820-pacific-distro-default-smithi/7556240', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-11 16:40:31.450230', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJdUbnVQDJMpnVjj+CNMfLqoADVfhGpoCxaA8bq8ZQ9GwbRU37n0/uWKqIGL5JrZMUIZjOgDJy3XUtm92aFr/v8='} 2024-02-11T17:15:09.846 INFO:teuthology.orchestra.console.smithi037:Power off 2024-02-11T17:15:09.846 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-11T17:15:09.872 DEBUG:teuthology.orchestra.console.smithi037:power off output: Chassis Power Control: Down/Off 2024-02-11T17:15:09.872 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-11T17:15:09.887 INFO:teuthology.orchestra.console.smithi178:Power off 2024-02-11T17:15:09.888 DEBUG:teuthology.orchestra.console.smithi178:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-11T17:15:09.896 DEBUG:teuthology.orchestra.console.smithi037:check power output: Chassis Power is on 2024-02-11T17:15:09.913 DEBUG:teuthology.orchestra.console.smithi178:power off output: Chassis Power Control: Down/Off 2024-02-11T17:15:09.913 DEBUG:teuthology.orchestra.console.smithi178:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-11T17:15:09.938 DEBUG:teuthology.orchestra.console.smithi178:check power output: Chassis Power is on 2024-02-11T17:15:13.897 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-11T17:15:13.940 DEBUG:teuthology.orchestra.console.smithi178:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-11T17:15:14.010 DEBUG:teuthology.orchestra.console.smithi037:check power output: Chassis Power is on 2024-02-11T17:15:14.054 DEBUG:teuthology.orchestra.console.smithi178:check power output: Chassis Power is on 2024-02-11T17:15:18.012 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-11T17:15:18.055 DEBUG:teuthology.orchestra.console.smithi178:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-11T17:15:18.123 DEBUG:teuthology.orchestra.console.smithi037:check power output: Chassis Power is on 2024-02-11T17:15:18.165 DEBUG:teuthology.orchestra.console.smithi178:check power output: Chassis Power is off 2024-02-11T17:15:18.265 INFO:teuthology.orchestra.console.smithi178:Power off completed 2024-02-11T17:15:22.124 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-11T17:15:22.235 DEBUG:teuthology.orchestra.console.smithi037:check power output: Chassis Power is off 2024-02-11T17:15:22.336 INFO:teuthology.orchestra.console.smithi037:Power off completed 2024-02-11T17:15:22.441 INFO:teuthology.run:Summary data: description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} duration: 1551.6196806430817 failure_reason: 'Command failed on smithi178 with status 5: ''sudo systemctl stop ceph-c65cd888-c8fe-11ee-95b9-87774f69a715@mon.smithi178''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=82dba0d89c784a73ad092a8a6d688db9 status: fail success: false 2024-02-11T17:15:22.442 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-11T17:15:22.523 INFO:teuthology.run:FAIL