2024-02-10T08:31:39.031 INFO:root:teuthology version: 0.0.1.dev256+gd9fdb22 2024-02-10T08:31:39.032 DEBUG:teuthology.run:Teuthology command: teuthology --name yuriw-2024-02-09_21:34:44-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi --archive /home/teuthworker/archive/yuriw-2024-02-09_21:34:44-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7555330 --owner scheduled_yuriw@teuthology --description rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} --verbose -- /home/teuthworker/archive/yuriw-2024-02-09_21:34:44-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7555330/orig.config.yaml 2024-02-10T08:31:39.083 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-10T08:31:39.184 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-02-09_21:34:44-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7555330 branch: wip-yuri10-testing-2024-02-08-0854-pacific description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} email: null first_in_suite: false job_id: '7555330' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/pacific v16.2.4 ' name: yuriw-2024-02-09_21:34:44-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri10-testing-2024-02-08-0854-pacific ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(PG_DEGRADED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 0e714d9a4bd2a821113e6318adb87bd06cf81ec1 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 0e714d9a4bd2a821113e6318adb87bd06cf81ec1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-tracker-64343-pacific sha1: 57bd6abdec7bb457ae7999d9c96682e9ac678e27 owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 964 sha1: 0e714d9a4bd2a821113e6318adb87bd06cf81ec1 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-tracker-64343-pacific suite_path: /home/teuthworker/src/github.com_ljflores_ceph_57bd6abdec7bb457ae7999d9c96682e9ac678e27/qa suite_relpath: qa suite_repo: https://github.com/ljflores/ceph.git suite_sha1: 57bd6abdec7bb457ae7999d9c96682e9ac678e27 targets: smithi077.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOkHI/0YKMx3NR9cd+Rv5sFx5wEZthIIzyZxxo+3z4ApnP9wcOXcnzolJWz1MPWdK32Q4vEuZIOvPtoC69iERC0= smithi155.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLG3o25k+pBMjkJgMe4uujFxi29ZlQcrYW+WhAldqCGVL5blpah59RRoilRfwNzrFs/5war+okFvzmcnzMkaSnc= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - pexec: clients: - sudo modprobe -r ceph - sudo modprobe ceph disable_send_metrics=on - install: exclude_packages: - ceph-volume tag: v16.2.4 - print: '**** done install task...' - cephadm: avoid_pacific_features: true cephadm_branch: v16.2.4 cephadm_git_url: https://github.com/ceph/ceph image: docker.io/ceph/ceph:v16.2.4 roleless: true - print: '**** done starting v16.2.4' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 1 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay false - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: d9fdb2209e15b39d9f061fd85399f352ce0f0894 timestamp: 2024-02-09_21:34:44 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-02-10T08:31:39.184 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ljflores_ceph_57bd6abdec7bb457ae7999d9c96682e9ac678e27/qa; will attempt to use it 2024-02-10T08:31:39.185 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ljflores_ceph_57bd6abdec7bb457ae7999d9c96682e9ac678e27/qa/tasks 2024-02-10T08:31:39.185 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-02-10T08:31:39.187 INFO:teuthology.task.internal:Checking packages... 2024-02-10T08:31:39.209 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '0e714d9a4bd2a821113e6318adb87bd06cf81ec1' 2024-02-10T08:31:39.209 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-10T08:31:39.209 INFO:teuthology.packaging:ref: None 2024-02-10T08:31:39.209 INFO:teuthology.packaging:tag: None 2024-02-10T08:31:39.210 INFO:teuthology.packaging:branch: wip-yuri10-testing-2024-02-08-0854-pacific 2024-02-10T08:31:39.210 INFO:teuthology.packaging:sha1: 0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-10T08:31:39.210 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri10-testing-2024-02-08-0854-pacific 2024-02-10T08:31:39.503 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-521.g0e714d9a 2024-02-10T08:31:39.505 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-02-10T08:31:39.514 INFO:teuthology.task.internal:no buildpackages task found 2024-02-10T08:31:39.514 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-10T08:31:39.540 INFO:teuthology.task.internal:Saving configuration 2024-02-10T08:31:39.557 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-10T08:31:39.567 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-10T08:31:39.592 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi077.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_21:34:44-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7555330', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-10 08:25:48.722617', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOkHI/0YKMx3NR9cd+Rv5sFx5wEZthIIzyZxxo+3z4ApnP9wcOXcnzolJWz1MPWdK32Q4vEuZIOvPtoC69iERC0='} 2024-02-10T08:31:39.614 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi155.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_21:34:44-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7555330', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-10 08:25:48.721253', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLG3o25k+pBMjkJgMe4uujFxi29ZlQcrYW+WhAldqCGVL5blpah59RRoilRfwNzrFs/5war+okFvzmcnzMkaSnc='} 2024-02-10T08:31:39.614 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-10T08:31:39.622 INFO:teuthology.task.internal:roles: ubuntu@smithi077.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-02-10T08:31:39.622 INFO:teuthology.task.internal:roles: ubuntu@smithi155.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-02-10T08:31:39.622 INFO:teuthology.run_tasks:Running task console_log... 2024-02-10T08:31:39.695 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f0afbf0bf70>, signals=[15]) 2024-02-10T08:31:39.695 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-10T08:31:39.706 INFO:teuthology.task.internal:Opening connections... 2024-02-10T08:31:39.706 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi077.front.sepia.ceph.com 2024-02-10T08:31:39.708 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T08:31:39.780 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi155.front.sepia.ceph.com 2024-02-10T08:31:39.781 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T08:31:39.852 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-10T08:31:39.859 DEBUG:teuthology.orchestra.run.smithi077:> uname -m 2024-02-10T08:31:39.881 INFO:teuthology.orchestra.run.smithi077.stdout:x86_64 2024-02-10T08:31:39.881 DEBUG:teuthology.orchestra.run.smithi077:> cat /etc/os-release 2024-02-10T08:31:39.941 INFO:teuthology.orchestra.run.smithi077.stdout:NAME="CentOS Stream" 2024-02-10T08:31:39.941 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION="8" 2024-02-10T08:31:39.942 INFO:teuthology.orchestra.run.smithi077.stdout:ID="centos" 2024-02-10T08:31:39.942 INFO:teuthology.orchestra.run.smithi077.stdout:ID_LIKE="rhel fedora" 2024-02-10T08:31:39.942 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION_ID="8" 2024-02-10T08:31:39.942 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM_ID="platform:el8" 2024-02-10T08:31:39.942 INFO:teuthology.orchestra.run.smithi077.stdout:PRETTY_NAME="CentOS Stream 8" 2024-02-10T08:31:39.942 INFO:teuthology.orchestra.run.smithi077.stdout:ANSI_COLOR="0;31" 2024-02-10T08:31:39.942 INFO:teuthology.orchestra.run.smithi077.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-02-10T08:31:39.942 INFO:teuthology.orchestra.run.smithi077.stdout:HOME_URL="https://centos.org/" 2024-02-10T08:31:39.942 INFO:teuthology.orchestra.run.smithi077.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-10T08:31:39.942 INFO:teuthology.orchestra.run.smithi077.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-02-10T08:31:39.942 INFO:teuthology.orchestra.run.smithi077.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-10T08:31:39.943 INFO:teuthology.lock.ops:Updating smithi077.front.sepia.ceph.com on lock server 2024-02-10T08:31:39.963 DEBUG:teuthology.orchestra.run.smithi155:> uname -m 2024-02-10T08:31:39.985 INFO:teuthology.orchestra.run.smithi155.stdout:x86_64 2024-02-10T08:31:39.985 DEBUG:teuthology.orchestra.run.smithi155:> cat /etc/os-release 2024-02-10T08:31:40.045 INFO:teuthology.orchestra.run.smithi155.stdout:NAME="CentOS Stream" 2024-02-10T08:31:40.045 INFO:teuthology.orchestra.run.smithi155.stdout:VERSION="8" 2024-02-10T08:31:40.045 INFO:teuthology.orchestra.run.smithi155.stdout:ID="centos" 2024-02-10T08:31:40.045 INFO:teuthology.orchestra.run.smithi155.stdout:ID_LIKE="rhel fedora" 2024-02-10T08:31:40.045 INFO:teuthology.orchestra.run.smithi155.stdout:VERSION_ID="8" 2024-02-10T08:31:40.045 INFO:teuthology.orchestra.run.smithi155.stdout:PLATFORM_ID="platform:el8" 2024-02-10T08:31:40.046 INFO:teuthology.orchestra.run.smithi155.stdout:PRETTY_NAME="CentOS Stream 8" 2024-02-10T08:31:40.046 INFO:teuthology.orchestra.run.smithi155.stdout:ANSI_COLOR="0;31" 2024-02-10T08:31:40.046 INFO:teuthology.orchestra.run.smithi155.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-02-10T08:31:40.046 INFO:teuthology.orchestra.run.smithi155.stdout:HOME_URL="https://centos.org/" 2024-02-10T08:31:40.046 INFO:teuthology.orchestra.run.smithi155.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-10T08:31:40.046 INFO:teuthology.orchestra.run.smithi155.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-02-10T08:31:40.046 INFO:teuthology.orchestra.run.smithi155.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-10T08:31:40.046 INFO:teuthology.lock.ops:Updating smithi155.front.sepia.ceph.com on lock server 2024-02-10T08:31:40.071 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-10T08:31:40.082 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-10T08:31:40.089 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-10T08:31:40.090 DEBUG:teuthology.orchestra.run.smithi077:> test '!' -e /home/ubuntu/cephtest 2024-02-10T08:31:40.092 DEBUG:teuthology.orchestra.run.smithi155:> test '!' -e /home/ubuntu/cephtest 2024-02-10T08:31:40.111 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-10T08:31:40.119 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-10T08:31:40.119 DEBUG:teuthology.orchestra.run.smithi077:> test -z $(ls -A /var/lib/ceph) 2024-02-10T08:31:40.153 DEBUG:teuthology.orchestra.run.smithi155:> test -z $(ls -A /var/lib/ceph) 2024-02-10T08:31:40.178 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-10T08:31:40.230 INFO:teuthology.run_tasks:Running task kernel... 2024-02-10T08:31:40.246 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-02-10T08:31:40.246 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-02-10T08:31:40.246 DEBUG:teuthology.orchestra.run.smithi077:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-10T08:31:40.247 DEBUG:teuthology.orchestra.run.smithi155:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-10T08:31:40.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T08:31:40.266 DEBUG:teuthology.orchestra.run.smithi155:> uname -r 2024-02-10T08:31:40.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T08:31:40.268 DEBUG:teuthology.orchestra.run.smithi077:> uname -r 2024-02-10T08:31:40.326 INFO:teuthology.orchestra.run.smithi155.stdout:4.18.0-489.el8.x86_64 2024-02-10T08:31:40.326 INFO:teuthology.task.kernel:Running kernel on smithi155: 4.18.0-489.el8.x86_64 2024-02-10T08:31:40.326 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum install -y kernel 2024-02-10T08:31:40.330 INFO:teuthology.orchestra.run.smithi077.stdout:4.18.0-489.el8.x86_64 2024-02-10T08:31:40.330 INFO:teuthology.task.kernel:Running kernel on smithi077: 4.18.0-489.el8.x86_64 2024-02-10T08:31:40.330 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum install -y kernel 2024-02-10T08:31:55.242 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 8 - AppStream 2.1 MB/s | 27 MB 00:12 2024-02-10T08:31:56.632 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 8 - AppStream 1.9 MB/s | 27 MB 00:13 2024-02-10T08:36:17.123 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 8 - BaseOS 41 kB/s | 10 MB 04:17 2024-02-10T08:36:18.882 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 8 - Extras 119 kB/s | 18 kB 00:00 2024-02-10T08:36:19.174 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 8 - Extras common packages 38 kB/s | 6.9 kB 00:00 2024-02-10T08:36:19.409 INFO:teuthology.orchestra.run.smithi155.stdout:Copr repo for python3-asyncssh owned by ceph 24 kB/s | 3.5 kB 00:00 2024-02-10T08:36:20.434 INFO:teuthology.orchestra.run.smithi155.stdout:Extra Packages for Enterprise Linux 17 MB/s | 16 MB 00:00 2024-02-10T08:36:24.159 INFO:teuthology.orchestra.run.smithi155.stdout:lab-extras 352 kB/s | 24 kB 00:00 2024-02-10T08:36:27.089 INFO:teuthology.orchestra.run.smithi155.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-02-10T08:36:27.090 INFO:teuthology.orchestra.run.smithi155.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-02-10T08:36:27.211 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-02-10T08:36:27.212 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-02-10T08:36:27.212 INFO:teuthology.orchestra.run.smithi155.stdout: Package Architecture Version Repository Size 2024-02-10T08:36:27.212 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-02-10T08:36:27.213 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-02-10T08:36:27.213 INFO:teuthology.orchestra.run.smithi155.stdout: kernel x86_64 4.18.0-536.el8 baseos 10 M 2024-02-10T08:36:27.213 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-02-10T08:36:27.213 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-core x86_64 4.18.0-536.el8 baseos 44 M 2024-02-10T08:36:27.213 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-modules x86_64 4.18.0-536.el8 baseos 36 M 2024-02-10T08:36:27.213 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:36:27.213 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-02-10T08:36:27.213 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-02-10T08:36:27.213 INFO:teuthology.orchestra.run.smithi155.stdout:Install 3 Packages 2024-02-10T08:36:27.213 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:36:27.214 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 90 M 2024-02-10T08:36:27.214 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 96 M 2024-02-10T08:36:27.214 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-02-10T08:36:27.279 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 8 - BaseOS 40 kB/s | 10 MB 04:25 2024-02-10T08:36:29.246 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 8 - Extras 54 kB/s | 18 kB 00:00 2024-02-10T08:36:29.546 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 8 - Extras common packages 37 kB/s | 6.9 kB 00:00 2024-02-10T08:36:29.781 INFO:teuthology.orchestra.run.smithi077.stdout:Copr repo for python3-asyncssh owned by ceph 24 kB/s | 3.5 kB 00:00 2024-02-10T08:36:33.643 INFO:teuthology.orchestra.run.smithi155.stdout:(1/3): kernel-4.18.0-536.el8.x86_64.rpm 1.6 MB/s | 10 MB 00:06 2024-02-10T08:36:33.893 INFO:teuthology.orchestra.run.smithi077.stdout:Extra Packages for Enterprise Linux 4.0 MB/s | 16 MB 00:04 2024-02-10T08:36:37.666 INFO:teuthology.orchestra.run.smithi077.stdout:lab-extras 406 kB/s | 24 kB 00:00 2024-02-10T08:36:38.293 INFO:teuthology.orchestra.run.smithi155.stdout:(2/3): kernel-modules-4.18.0-536.el8.x86_64.rpm 3.3 MB/s | 36 MB 00:10 2024-02-10T08:36:40.608 INFO:teuthology.orchestra.run.smithi077.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-02-10T08:36:40.608 INFO:teuthology.orchestra.run.smithi077.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-02-10T08:36:40.717 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-02-10T08:36:40.718 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-10T08:36:40.718 INFO:teuthology.orchestra.run.smithi077.stdout: Package Architecture Version Repository Size 2024-02-10T08:36:40.718 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-10T08:36:40.718 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2024-02-10T08:36:40.718 INFO:teuthology.orchestra.run.smithi077.stdout: kernel x86_64 4.18.0-536.el8 baseos 10 M 2024-02-10T08:36:40.719 INFO:teuthology.orchestra.run.smithi077.stdout:Installing dependencies: 2024-02-10T08:36:40.719 INFO:teuthology.orchestra.run.smithi077.stdout: kernel-core x86_64 4.18.0-536.el8 baseos 44 M 2024-02-10T08:36:40.719 INFO:teuthology.orchestra.run.smithi077.stdout: kernel-modules x86_64 4.18.0-536.el8 baseos 36 M 2024-02-10T08:36:40.719 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:36:40.719 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2024-02-10T08:36:40.719 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-10T08:36:40.720 INFO:teuthology.orchestra.run.smithi077.stdout:Install 3 Packages 2024-02-10T08:36:40.720 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:36:40.720 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 90 M 2024-02-10T08:36:40.721 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 96 M 2024-02-10T08:36:40.721 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2024-02-10T08:36:44.517 INFO:teuthology.orchestra.run.smithi155.stdout:(3/3): kernel-core-4.18.0-536.el8.x86_64.rpm 2.5 MB/s | 44 MB 00:17 2024-02-10T08:36:44.518 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:36:44.518 INFO:teuthology.orchestra.run.smithi155.stdout:Total 5.2 MB/s | 90 MB 00:17 2024-02-10T08:36:45.195 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-02-10T08:36:45.350 INFO:teuthology.orchestra.run.smithi077.stdout:(1/3): kernel-4.18.0-536.el8.x86_64.rpm 2.3 MB/s | 10 MB 00:04 2024-02-10T08:36:45.619 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-02-10T08:36:45.619 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-02-10T08:36:46.852 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-02-10T08:36:46.883 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-02-10T08:36:49.678 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-02-10T08:36:50.188 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : kernel-core-4.18.0-536.el8.x86_64 1/3 2024-02-10T08:36:53.811 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: kernel-core-4.18.0-536.el8.x86_64 1/3 2024-02-10T08:36:54.204 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : kernel-modules-4.18.0-536.el8.x86_64 2/3 2024-02-10T08:36:54.249 INFO:teuthology.orchestra.run.smithi077.stdout:(2/3): kernel-modules-4.18.0-536.el8.x86_64.rpm 2.7 MB/s | 36 MB 00:13 2024-02-10T08:36:55.500 INFO:teuthology.orchestra.run.smithi077.stdout:(3/3): kernel-core-4.18.0-536.el8.x86_64.rpm 3.0 MB/s | 44 MB 00:14 2024-02-10T08:36:55.500 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:36:55.500 INFO:teuthology.orchestra.run.smithi077.stdout:Total 6.1 MB/s | 90 MB 00:14 2024-02-10T08:36:56.184 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2024-02-10T08:36:56.587 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2024-02-10T08:36:56.587 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2024-02-10T08:36:57.640 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: kernel-modules-4.18.0-536.el8.x86_64 2/3 2024-02-10T08:36:57.816 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2024-02-10T08:36:57.840 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2024-02-10T08:36:58.069 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : kernel-4.18.0-536.el8.x86_64 3/3 2024-02-10T08:37:00.639 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2024-02-10T08:37:01.234 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : kernel-core-4.18.0-536.el8.x86_64 1/3 2024-02-10T08:37:04.803 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: kernel-core-4.18.0-536.el8.x86_64 1/3 2024-02-10T08:37:05.208 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : kernel-modules-4.18.0-536.el8.x86_64 2/3 2024-02-10T08:37:08.645 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: kernel-modules-4.18.0-536.el8.x86_64 2/3 2024-02-10T08:37:08.916 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : kernel-4.18.0-536.el8.x86_64 3/3 2024-02-10T08:37:19.912 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: kernel-core-4.18.0-536.el8.x86_64 3/3 2024-02-10T08:37:19.936 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: kernel-modules-4.18.0-536.el8.x86_64 3/3 2024-02-10T08:37:20.291 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: kernel-4.18.0-536.el8.x86_64 3/3 2024-02-10T08:37:20.291 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : kernel-4.18.0-536.el8.x86_64 1/3 2024-02-10T08:37:20.291 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : kernel-core-4.18.0-536.el8.x86_64 2/3 2024-02-10T08:37:20.726 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : kernel-modules-4.18.0-536.el8.x86_64 3/3 2024-02-10T08:37:20.727 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:37:20.727 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-02-10T08:37:20.727 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-4.18.0-536.el8.x86_64 kernel-core-4.18.0-536.el8.x86_64 2024-02-10T08:37:20.727 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-modules-4.18.0-536.el8.x86_64 2024-02-10T08:37:20.727 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:37:20.727 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-02-10T08:37:20.871 DEBUG:teuthology.orchestra.run.smithi155:> rpm -q kernel | sort -rV | head -n 1 2024-02-10T08:37:21.090 INFO:teuthology.orchestra.run.smithi155.stdout:kernel-4.18.0-536.el8.x86_64 2024-02-10T08:37:21.091 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-536.el8.x86_64 2024-02-10T08:37:21.091 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-536.el8.x86_64 2024-02-10T08:37:21.091 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-10T08:37:21.091 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-10T08:37:21.091 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-02-10T08:37:21.091 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi155.front.sepia.ceph.com, path=None, version=distro) 2024-02-10T08:37:21.092 DEBUG:teuthology.orchestra.run.smithi155:> rpm -q kernel | sort -rV | head -n 1 2024-02-10T08:37:21.302 INFO:teuthology.orchestra.run.smithi155.stdout:kernel-4.18.0-536.el8.x86_64 2024-02-10T08:37:21.302 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-536.el8.x86_64 2024-02-10T08:37:21.302 DEBUG:teuthology.orchestra.run.smithi155:> sudo rpm -qi grub2-tools 2024-02-10T08:37:21.336 INFO:teuthology.orchestra.run.smithi155.stdout:Name : grub2-tools 2024-02-10T08:37:21.336 INFO:teuthology.orchestra.run.smithi155.stdout:Epoch : 1 2024-02-10T08:37:21.336 INFO:teuthology.orchestra.run.smithi155.stdout:Version : 2.02 2024-02-10T08:37:21.336 INFO:teuthology.orchestra.run.smithi155.stdout:Release : 129.el8 2024-02-10T08:37:21.336 INFO:teuthology.orchestra.run.smithi155.stdout:Architecture: x86_64 2024-02-10T08:37:21.337 INFO:teuthology.orchestra.run.smithi155.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-02-10T08:37:21.337 INFO:teuthology.orchestra.run.smithi155.stdout:Group : System Environment/Base 2024-02-10T08:37:21.337 INFO:teuthology.orchestra.run.smithi155.stdout:Size : 9265586 2024-02-10T08:37:21.337 INFO:teuthology.orchestra.run.smithi155.stdout:License : GPLv3+ 2024-02-10T08:37:21.337 INFO:teuthology.orchestra.run.smithi155.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-02-10T08:37:21.337 INFO:teuthology.orchestra.run.smithi155.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-02-10T08:37:21.337 INFO:teuthology.orchestra.run.smithi155.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-02-10T08:37:21.337 INFO:teuthology.orchestra.run.smithi155.stdout:Build Host : kbuilder.bsys.centos.org 2024-02-10T08:37:21.337 INFO:teuthology.orchestra.run.smithi155.stdout:Relocations : (not relocatable) 2024-02-10T08:37:21.337 INFO:teuthology.orchestra.run.smithi155.stdout:Packager : CentOS BuildSystem 2024-02-10T08:37:21.337 INFO:teuthology.orchestra.run.smithi155.stdout:Vendor : CentOS 2024-02-10T08:37:21.337 INFO:teuthology.orchestra.run.smithi155.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-10T08:37:21.338 INFO:teuthology.orchestra.run.smithi155.stdout:Summary : Support tools for GRUB. 2024-02-10T08:37:21.338 INFO:teuthology.orchestra.run.smithi155.stdout:Description : 2024-02-10T08:37:21.338 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:37:21.338 INFO:teuthology.orchestra.run.smithi155.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-10T08:37:21.338 INFO:teuthology.orchestra.run.smithi155.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-10T08:37:21.338 INFO:teuthology.orchestra.run.smithi155.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-10T08:37:21.338 INFO:teuthology.orchestra.run.smithi155.stdout:hardware devices. 2024-02-10T08:37:21.338 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:37:21.338 INFO:teuthology.orchestra.run.smithi155.stdout:This subpackage provides tools for support of all platforms. 2024-02-10T08:37:21.339 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-10T08:37:21.339 INFO:teuthology.task.kernel:Updating grub on smithi155 to boot 4.18.0-536.el8.x86_64 2024-02-10T08:37:21.339 DEBUG:teuthology.orchestra.run.smithi155:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-10T08:37:23.074 INFO:teuthology.orchestra.run.smithi155.stderr:Generating grub configuration file ... 2024-02-10T08:37:24.929 INFO:teuthology.orchestra.run.smithi155.stderr:done 2024-02-10T08:37:24.931 DEBUG:teuthology.orchestra.run.smithi155:> mktemp 2024-02-10T08:37:24.956 INFO:teuthology.orchestra.run.smithi155.stdout:/tmp/tmp.I5NIWmCt6C 2024-02-10T08:37:24.956 DEBUG:teuthology.orchestra.run.smithi155:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.I5NIWmCt6C 2024-02-10T08:37:25.029 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod 0666 /tmp/tmp.I5NIWmCt6C 2024-02-10T08:37:25.189 DEBUG:teuthology.orchestra.remote:smithi155:/tmp/tmp.I5NIWmCt6C is 6KB 2024-02-10T08:37:25.237 DEBUG:teuthology.orchestra.run.smithi155:> rm -fr /tmp/tmp.I5NIWmCt6C 2024-02-10T08:37:25.254 DEBUG:teuthology.orchestra.run.smithi155:> sudo /bin/ls /boot/loader/entries || true 2024-02-10T08:37:25.325 INFO:teuthology.orchestra.run.smithi155.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-02-10T08:37:25.325 INFO:teuthology.orchestra.run.smithi155.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-02-10T08:37:25.325 INFO:teuthology.orchestra.run.smithi155.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-02-10T08:37:25.325 INFO:teuthology.orchestra.run.smithi155.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-536.el8.x86_64.conf 2024-02-10T08:37:25.327 DEBUG:teuthology.orchestra.run.smithi155:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-536.el8.x86_64 2024-02-10T08:37:25.489 DEBUG:teuthology.orchestra.run.smithi155:> sudo shutdown -r now 2024-02-10T08:37:30.726 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: kernel-core-4.18.0-536.el8.x86_64 3/3 2024-02-10T08:37:30.749 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: kernel-modules-4.18.0-536.el8.x86_64 3/3 2024-02-10T08:37:31.124 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: kernel-4.18.0-536.el8.x86_64 3/3 2024-02-10T08:37:31.124 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : kernel-4.18.0-536.el8.x86_64 1/3 2024-02-10T08:37:31.124 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : kernel-core-4.18.0-536.el8.x86_64 2/3 2024-02-10T08:37:31.686 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : kernel-modules-4.18.0-536.el8.x86_64 3/3 2024-02-10T08:37:31.686 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:37:31.687 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2024-02-10T08:37:31.687 INFO:teuthology.orchestra.run.smithi077.stdout: kernel-4.18.0-536.el8.x86_64 kernel-core-4.18.0-536.el8.x86_64 2024-02-10T08:37:31.687 INFO:teuthology.orchestra.run.smithi077.stdout: kernel-modules-4.18.0-536.el8.x86_64 2024-02-10T08:37:31.687 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:37:31.687 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-02-10T08:37:32.160 DEBUG:teuthology.orchestra.run.smithi077:> rpm -q kernel | sort -rV | head -n 1 2024-02-10T08:37:32.380 INFO:teuthology.orchestra.run.smithi077.stdout:kernel-4.18.0-536.el8.x86_64 2024-02-10T08:37:32.380 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-536.el8.x86_64 2024-02-10T08:37:32.380 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-536.el8.x86_64 2024-02-10T08:37:32.380 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-10T08:37:32.380 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-10T08:37:32.380 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-02-10T08:37:32.381 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi077.front.sepia.ceph.com, path=None, version=distro) 2024-02-10T08:37:32.381 DEBUG:teuthology.orchestra.run.smithi077:> rpm -q kernel | sort -rV | head -n 1 2024-02-10T08:37:32.590 INFO:teuthology.orchestra.run.smithi077.stdout:kernel-4.18.0-536.el8.x86_64 2024-02-10T08:37:32.590 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-536.el8.x86_64 2024-02-10T08:37:32.590 DEBUG:teuthology.orchestra.run.smithi077:> sudo rpm -qi grub2-tools 2024-02-10T08:37:32.623 INFO:teuthology.orchestra.run.smithi077.stdout:Name : grub2-tools 2024-02-10T08:37:32.624 INFO:teuthology.orchestra.run.smithi077.stdout:Epoch : 1 2024-02-10T08:37:32.624 INFO:teuthology.orchestra.run.smithi077.stdout:Version : 2.02 2024-02-10T08:37:32.624 INFO:teuthology.orchestra.run.smithi077.stdout:Release : 129.el8 2024-02-10T08:37:32.624 INFO:teuthology.orchestra.run.smithi077.stdout:Architecture: x86_64 2024-02-10T08:37:32.624 INFO:teuthology.orchestra.run.smithi077.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-02-10T08:37:32.624 INFO:teuthology.orchestra.run.smithi077.stdout:Group : System Environment/Base 2024-02-10T08:37:32.624 INFO:teuthology.orchestra.run.smithi077.stdout:Size : 9265586 2024-02-10T08:37:32.624 INFO:teuthology.orchestra.run.smithi077.stdout:License : GPLv3+ 2024-02-10T08:37:32.624 INFO:teuthology.orchestra.run.smithi077.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-02-10T08:37:32.624 INFO:teuthology.orchestra.run.smithi077.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-02-10T08:37:32.625 INFO:teuthology.orchestra.run.smithi077.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-02-10T08:37:32.625 INFO:teuthology.orchestra.run.smithi077.stdout:Build Host : kbuilder.bsys.centos.org 2024-02-10T08:37:32.625 INFO:teuthology.orchestra.run.smithi077.stdout:Relocations : (not relocatable) 2024-02-10T08:37:32.625 INFO:teuthology.orchestra.run.smithi077.stdout:Packager : CentOS BuildSystem 2024-02-10T08:37:32.625 INFO:teuthology.orchestra.run.smithi077.stdout:Vendor : CentOS 2024-02-10T08:37:32.625 INFO:teuthology.orchestra.run.smithi077.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-10T08:37:32.625 INFO:teuthology.orchestra.run.smithi077.stdout:Summary : Support tools for GRUB. 2024-02-10T08:37:32.625 INFO:teuthology.orchestra.run.smithi077.stdout:Description : 2024-02-10T08:37:32.625 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:37:32.625 INFO:teuthology.orchestra.run.smithi077.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-10T08:37:32.625 INFO:teuthology.orchestra.run.smithi077.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-10T08:37:32.625 INFO:teuthology.orchestra.run.smithi077.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-10T08:37:32.625 INFO:teuthology.orchestra.run.smithi077.stdout:hardware devices. 2024-02-10T08:37:32.626 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:37:32.626 INFO:teuthology.orchestra.run.smithi077.stdout:This subpackage provides tools for support of all platforms. 2024-02-10T08:37:32.627 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-10T08:37:32.627 INFO:teuthology.task.kernel:Updating grub on smithi077 to boot 4.18.0-536.el8.x86_64 2024-02-10T08:37:32.627 DEBUG:teuthology.orchestra.run.smithi077:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-10T08:37:33.919 INFO:teuthology.orchestra.run.smithi077.stderr:Generating grub configuration file ... 2024-02-10T08:37:35.676 INFO:teuthology.orchestra.run.smithi077.stderr:done 2024-02-10T08:37:35.677 DEBUG:teuthology.orchestra.run.smithi077:> mktemp 2024-02-10T08:37:35.700 INFO:teuthology.orchestra.run.smithi077.stdout:/tmp/tmp.aEknMWttLN 2024-02-10T08:37:35.700 DEBUG:teuthology.orchestra.run.smithi077:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.aEknMWttLN 2024-02-10T08:37:35.774 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 0666 /tmp/tmp.aEknMWttLN 2024-02-10T08:37:35.923 DEBUG:teuthology.orchestra.remote:smithi077:/tmp/tmp.aEknMWttLN is 6KB 2024-02-10T08:37:35.972 DEBUG:teuthology.orchestra.run.smithi077:> rm -fr /tmp/tmp.aEknMWttLN 2024-02-10T08:37:35.989 DEBUG:teuthology.orchestra.run.smithi077:> sudo /bin/ls /boot/loader/entries || true 2024-02-10T08:37:36.057 INFO:teuthology.orchestra.run.smithi077.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-02-10T08:37:36.057 INFO:teuthology.orchestra.run.smithi077.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-02-10T08:37:36.057 INFO:teuthology.orchestra.run.smithi077.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-02-10T08:37:36.057 INFO:teuthology.orchestra.run.smithi077.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-536.el8.x86_64.conf 2024-02-10T08:37:36.058 DEBUG:teuthology.orchestra.run.smithi077:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-536.el8.x86_64 2024-02-10T08:37:36.217 DEBUG:teuthology.orchestra.run.smithi077:> sudo shutdown -r now 2024-02-10T08:37:55.504 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-02-10T08:37:55.505 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-02-10T08:37:55.506 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T08:38:06.231 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-02-10T08:38:06.231 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-02-10T08:38:06.231 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T08:38:13.820 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-02-10T08:38:22.828 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-02-10T08:38:22.829 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T08:38:24.576 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-02-10T08:38:25.888 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-02-10T08:38:33.584 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-02-10T08:38:33.585 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T08:38:36.636 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-02-10T08:38:37.891 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-02-10T08:38:37.891 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T08:38:48.644 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-02-10T08:38:48.645 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T08:38:56.316 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-02-10T08:39:11.322 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-02-10T08:39:11.323 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T08:39:11.770 DEBUG:teuthology.orchestra.run.smithi155:> true 2024-02-10T08:39:12.065 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-02-10T08:39:12.065 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "4.18.0-536.el8.x86_64"... 2024-02-10T08:39:12.065 DEBUG:teuthology.orchestra.run.smithi155:> uname -r 2024-02-10T08:39:12.082 INFO:teuthology.orchestra.run.smithi155.stdout:4.18.0-536.el8.x86_64 2024-02-10T08:39:12.083 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-536.el8.x86_64 vs 4.18.0-536.el8.x86_64 2024-02-10T08:39:12.083 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-10T08:39:12.083 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-10T08:39:13.083 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-02-10T08:39:13.084 DEBUG:teuthology.orchestra.run.smithi155:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-10T08:39:13.153 INFO:teuthology.orchestra.run.smithi155.stdout:ttyS1 2024-02-10T08:39:13.168 DEBUG:teuthology.parallel:result is None 2024-02-10T08:39:21.207 DEBUG:teuthology.orchestra.run.smithi077:> true 2024-02-10T08:39:21.848 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-02-10T08:39:21.848 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "4.18.0-536.el8.x86_64"... 2024-02-10T08:39:21.849 DEBUG:teuthology.orchestra.run.smithi077:> uname -r 2024-02-10T08:39:21.866 INFO:teuthology.orchestra.run.smithi077.stdout:4.18.0-536.el8.x86_64 2024-02-10T08:39:21.866 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-536.el8.x86_64 vs 4.18.0-536.el8.x86_64 2024-02-10T08:39:21.866 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-10T08:39:21.867 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-10T08:39:22.868 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-02-10T08:39:22.868 DEBUG:teuthology.orchestra.run.smithi077:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-10T08:39:22.948 INFO:teuthology.orchestra.run.smithi077.stdout:ttyS1 2024-02-10T08:39:22.960 DEBUG:teuthology.parallel:result is None 2024-02-10T08:39:22.960 INFO:teuthology.run_tasks:Running task internal.base... 2024-02-10T08:39:22.968 INFO:teuthology.task.internal:Creating test directory... 2024-02-10T08:39:22.968 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-10T08:39:22.971 DEBUG:teuthology.orchestra.run.smithi155:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-10T08:39:22.994 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-02-10T08:39:23.002 INFO:teuthology.run_tasks:Running task internal.archive... 2024-02-10T08:39:23.011 INFO:teuthology.task.internal:Creating archive directory... 2024-02-10T08:39:23.011 DEBUG:teuthology.orchestra.run.smithi077:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-10T08:39:23.031 DEBUG:teuthology.orchestra.run.smithi155:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-10T08:39:23.076 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-02-10T08:39:23.083 INFO:teuthology.task.internal:Enabling coredump saving... 2024-02-10T08:39:23.083 DEBUG:teuthology.orchestra.run.smithi077:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-10T08:39:23.100 DEBUG:teuthology.orchestra.run.smithi155:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-10T08:39:23.140 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-10T08:39:23.149 INFO:teuthology.orchestra.run.smithi155.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-10T08:39:23.152 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-10T08:39:23.160 INFO:teuthology.orchestra.run.smithi155.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-10T08:39:23.162 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-02-10T08:39:23.169 INFO:teuthology.task.internal:Configuring sudo... 2024-02-10T08:39:23.169 DEBUG:teuthology.orchestra.run.smithi077:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-10T08:39:23.171 DEBUG:teuthology.orchestra.run.smithi155:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-10T08:39:23.231 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-02-10T08:39:23.241 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-02-10T08:39:23.241 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-10T08:39:23.243 DEBUG:teuthology.orchestra.run.smithi155:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-10T08:39:23.289 DEBUG:teuthology.orchestra.run.smithi077:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-10T08:39:23.351 DEBUG:teuthology.orchestra.run.smithi077:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-10T08:39:23.408 DEBUG:teuthology.orchestra.run.smithi077:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-10T08:39:23.454 DEBUG:teuthology.orchestra.run.smithi077:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-10T08:39:23.497 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-02-10T08:39:23.497 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-10T08:39:23.563 DEBUG:teuthology.orchestra.run.smithi155:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-10T08:39:23.608 DEBUG:teuthology.orchestra.run.smithi155:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-10T08:39:23.662 DEBUG:teuthology.orchestra.run.smithi155:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-10T08:39:23.749 DEBUG:teuthology.orchestra.run.smithi155:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-10T08:39:23.790 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-02-10T08:39:23.790 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-10T08:39:23.856 DEBUG:teuthology.orchestra.run.smithi077:> sudo service rsyslog restart 2024-02-10T08:39:23.859 DEBUG:teuthology.orchestra.run.smithi155:> sudo service rsyslog restart 2024-02-10T08:39:23.950 INFO:teuthology.orchestra.run.smithi077.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-10T08:39:23.959 INFO:teuthology.orchestra.run.smithi155.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-10T08:39:24.433 INFO:teuthology.run_tasks:Running task internal.timer... 2024-02-10T08:39:24.441 INFO:teuthology.task.internal:Starting timer... 2024-02-10T08:39:24.441 INFO:teuthology.run_tasks:Running task pcp... 2024-02-10T08:39:24.451 INFO:teuthology.run_tasks:Running task selinux... 2024-02-10T08:39:24.458 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-02-10T08:39:24.459 DEBUG:teuthology.orchestra.run.smithi077:> sudo service auditd rotate 2024-02-10T08:39:24.533 INFO:teuthology.orchestra.run.smithi077.stdout:Rotating logs: 2024-02-10T08:39:24.535 DEBUG:teuthology.orchestra.run.smithi155:> sudo service auditd rotate 2024-02-10T08:39:24.624 INFO:teuthology.orchestra.run.smithi155.stdout:Rotating logs: 2024-02-10T08:39:24.624 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-02-10T08:39:24.625 DEBUG:teuthology.orchestra.run.smithi077:> /usr/sbin/getenforce 2024-02-10T08:39:24.660 INFO:teuthology.orchestra.run.smithi077.stdout:Permissive 2024-02-10T08:39:24.661 DEBUG:teuthology.orchestra.run.smithi155:> /usr/sbin/getenforce 2024-02-10T08:39:24.687 INFO:teuthology.orchestra.run.smithi155.stdout:Permissive 2024-02-10T08:39:24.687 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi077.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi155.front.sepia.ceph.com': 'permissive'} 2024-02-10T08:39:24.687 DEBUG:teuthology.orchestra.run.smithi077:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-10T08:39:24.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T08:39:24.728 DEBUG:teuthology.orchestra.run.smithi155:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-10T08:39:24.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T08:39:24.755 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-02-10T08:39:24.755 DEBUG:teuthology.orchestra.run.smithi077:> sudo /usr/sbin/setenforce permissive 2024-02-10T08:39:24.794 DEBUG:teuthology.orchestra.run.smithi155:> sudo /usr/sbin/setenforce permissive 2024-02-10T08:39:24.829 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-02-10T08:39:24.840 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-02-10T08:39:24.857 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-02-10T08:39:24.859 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi077.front.sepia.ceph.com,smithi155.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-02-10T08:45:31.314 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi077.front.sepia.ceph.com'), Remote(name='ubuntu@smithi155.front.sepia.ceph.com')] 2024-02-10T08:45:31.315 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-02-10T08:45:31.316 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T08:45:31.388 DEBUG:teuthology.orchestra.run.smithi077:> true 2024-02-10T08:45:31.454 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-02-10T08:45:31.455 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-02-10T08:45:31.455 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T08:45:31.533 DEBUG:teuthology.orchestra.run.smithi155:> true 2024-02-10T08:45:31.603 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-02-10T08:45:31.603 INFO:teuthology.run_tasks:Running task clock... 2024-02-10T08:45:31.613 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-02-10T08:45:31.614 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-10T08:45:31.614 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-10T08:45:31.616 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-10T08:45:31.617 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-10T08:45:31.646 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-10T08:45:31.660 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-10T08:45:31.674 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-10T08:45:31.687 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-10T08:45:31.728 INFO:teuthology.orchestra.run.smithi077.stderr:sudo: ntpd: command not found 2024-02-10T08:45:31.739 INFO:teuthology.orchestra.run.smithi077.stdout:506 Cannot talk to daemon 2024-02-10T08:45:31.752 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-10T08:45:31.763 INFO:teuthology.orchestra.run.smithi155.stderr:sudo: ntpd: command not found 2024-02-10T08:45:31.766 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-10T08:45:31.774 INFO:teuthology.orchestra.run.smithi155.stdout:506 Cannot talk to daemon 2024-02-10T08:45:31.788 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-10T08:45:31.801 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-10T08:45:31.830 INFO:teuthology.orchestra.run.smithi077.stderr:bash: ntpq: command not found 2024-02-10T08:45:31.834 INFO:teuthology.orchestra.run.smithi077.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-10T08:45:31.834 INFO:teuthology.orchestra.run.smithi077.stdout:=============================================================================== 2024-02-10T08:45:31.834 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T08:45:31.834 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T08:45:31.834 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T08:45:31.834 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T08:45:31.862 INFO:teuthology.orchestra.run.smithi155.stderr:bash: ntpq: command not found 2024-02-10T08:45:31.866 INFO:teuthology.orchestra.run.smithi155.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-10T08:45:31.866 INFO:teuthology.orchestra.run.smithi155.stdout:=============================================================================== 2024-02-10T08:45:31.866 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T08:45:31.866 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T08:45:31.866 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T08:45:31.866 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T08:45:31.867 INFO:teuthology.run_tasks:Running task pexec... 2024-02-10T08:45:31.876 INFO:teuthology.task.pexec:Executing custom commands... 2024-02-10T08:45:31.876 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi077.front.sepia.ceph.com 2024-02-10T08:45:31.876 DEBUG:teuthology.orchestra.run.smithi077:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-10T08:45:31.877 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi155.front.sepia.ceph.com 2024-02-10T08:45:31.877 DEBUG:teuthology.orchestra.run.smithi155:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-10T08:45:32.664 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:01:33 ago on Sat 10 Feb 2024 08:43:59 AM UTC. 2024-02-10T08:45:32.689 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:32 ago on Sat 10 Feb 2024 08:44:00 AM UTC. 2024-02-10T08:45:34.650 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-02-10T08:45:34.651 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-02-10T08:45:34.651 INFO:teuthology.orchestra.run.smithi155.stdout: Package Architecture Version Repository Size 2024-02-10T08:45:34.651 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-02-10T08:45:34.651 INFO:teuthology.orchestra.run.smithi155.stdout:Resetting modules: 2024-02-10T08:45:34.651 INFO:teuthology.orchestra.run.smithi155.stdout: container-tools 2024-02-10T08:45:34.651 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:45:34.651 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-02-10T08:45:34.651 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-02-10T08:45:34.651 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:45:34.659 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-02-10T08:45:34.660 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-10T08:45:34.660 INFO:teuthology.orchestra.run.smithi077.stdout: Package Architecture Version Repository Size 2024-02-10T08:45:34.660 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-10T08:45:34.661 INFO:teuthology.orchestra.run.smithi077.stdout:Resetting modules: 2024-02-10T08:45:34.661 INFO:teuthology.orchestra.run.smithi077.stdout: container-tools 2024-02-10T08:45:34.661 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:45:34.661 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2024-02-10T08:45:34.661 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-10T08:45:34.661 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:45:34.908 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-02-10T08:45:34.920 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-02-10T08:45:35.757 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:01:36 ago on Sat 10 Feb 2024 08:43:59 AM UTC. 2024-02-10T08:45:35.789 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:35 ago on Sat 10 Feb 2024 08:44:00 AM UTC. 2024-02-10T08:45:37.750 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-02-10T08:45:37.759 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-02-10T08:45:37.786 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-10T08:45:37.786 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repository Size 2024-02-10T08:45:37.786 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-10T08:45:37.786 INFO:teuthology.orchestra.run.smithi077.stdout:Installing group/module packages: 2024-02-10T08:45:37.786 INFO:teuthology.orchestra.run.smithi077.stdout: aardvark-dns x86_64 2:1.9.0-1.module_el8+804+f131391c appstream 1.0 M 2024-02-10T08:45:37.786 INFO:teuthology.orchestra.run.smithi077.stdout: buildah x86_64 1:1.34.0-1.module_el8+804+f131391c appstream 9.7 M 2024-02-10T08:45:37.786 INFO:teuthology.orchestra.run.smithi077.stdout: cockpit-podman noarch 82-1.module_el8+804+f131391c appstream 759 k 2024-02-10T08:45:37.786 INFO:teuthology.orchestra.run.smithi077.stdout: crun x86_64 1.12-1.module_el8+804+f131391c appstream 248 k 2024-02-10T08:45:37.786 INFO:teuthology.orchestra.run.smithi077.stdout: netavark x86_64 2:1.9.0-1.module_el8+804+f131391c appstream 4.0 M 2024-02-10T08:45:37.786 INFO:teuthology.orchestra.run.smithi077.stdout: python3-podman noarch 4.8.2-1.module_el8+804+f131391c appstream 179 k 2024-02-10T08:45:37.786 INFO:teuthology.orchestra.run.smithi077.stdout: skopeo x86_64 2:1.14.0-2.module_el8+804+f131391c appstream 8.8 M 2024-02-10T08:45:37.787 INFO:teuthology.orchestra.run.smithi077.stdout: toolbox x86_64 0.0.99.4-8.module_el8+804+f131391c appstream 2.5 M 2024-02-10T08:45:37.787 INFO:teuthology.orchestra.run.smithi077.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-02-10T08:45:37.787 INFO:teuthology.orchestra.run.smithi077.stdout:Installing dependencies: 2024-02-10T08:45:37.787 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-02-10T08:45:37.787 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-02-10T08:45:37.787 INFO:teuthology.orchestra.run.smithi077.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-02-10T08:45:37.787 INFO:teuthology.orchestra.run.smithi077.stdout:Installing weak dependencies: 2024-02-10T08:45:37.787 INFO:teuthology.orchestra.run.smithi077.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-02-10T08:45:37.787 INFO:teuthology.orchestra.run.smithi077.stdout:Installing module profiles: 2024-02-10T08:45:37.787 INFO:teuthology.orchestra.run.smithi077.stdout: container-tools/common 2024-02-10T08:45:37.787 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:45:37.787 INFO:teuthology.orchestra.run.smithi077.stdout:Enabling module streams: 2024-02-10T08:45:37.787 INFO:teuthology.orchestra.run.smithi077.stdout: container-tools rhel8 2024-02-10T08:45:37.787 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:45:37.787 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2024-02-10T08:45:37.788 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-10T08:45:37.788 INFO:teuthology.orchestra.run.smithi077.stdout:Install 13 Packages 2024-02-10T08:45:37.788 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:45:37.788 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 27 M 2024-02-10T08:45:37.788 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 91 M 2024-02-10T08:45:37.789 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2024-02-10T08:45:37.795 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-02-10T08:45:37.795 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-02-10T08:45:37.795 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-02-10T08:45:37.796 INFO:teuthology.orchestra.run.smithi155.stdout:Installing group/module packages: 2024-02-10T08:45:37.796 INFO:teuthology.orchestra.run.smithi155.stdout: aardvark-dns x86_64 2:1.9.0-1.module_el8+804+f131391c appstream 1.0 M 2024-02-10T08:45:37.796 INFO:teuthology.orchestra.run.smithi155.stdout: buildah x86_64 1:1.34.0-1.module_el8+804+f131391c appstream 9.7 M 2024-02-10T08:45:37.796 INFO:teuthology.orchestra.run.smithi155.stdout: cockpit-podman noarch 82-1.module_el8+804+f131391c appstream 759 k 2024-02-10T08:45:37.796 INFO:teuthology.orchestra.run.smithi155.stdout: crun x86_64 1.12-1.module_el8+804+f131391c appstream 248 k 2024-02-10T08:45:37.796 INFO:teuthology.orchestra.run.smithi155.stdout: netavark x86_64 2:1.9.0-1.module_el8+804+f131391c appstream 4.0 M 2024-02-10T08:45:37.796 INFO:teuthology.orchestra.run.smithi155.stdout: python3-podman noarch 4.8.2-1.module_el8+804+f131391c appstream 179 k 2024-02-10T08:45:37.796 INFO:teuthology.orchestra.run.smithi155.stdout: skopeo x86_64 2:1.14.0-2.module_el8+804+f131391c appstream 8.8 M 2024-02-10T08:45:37.796 INFO:teuthology.orchestra.run.smithi155.stdout: toolbox x86_64 0.0.99.4-8.module_el8+804+f131391c appstream 2.5 M 2024-02-10T08:45:37.796 INFO:teuthology.orchestra.run.smithi155.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-02-10T08:45:37.796 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-02-10T08:45:37.797 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-02-10T08:45:37.797 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-02-10T08:45:37.797 INFO:teuthology.orchestra.run.smithi155.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-02-10T08:45:37.797 INFO:teuthology.orchestra.run.smithi155.stdout:Installing weak dependencies: 2024-02-10T08:45:37.797 INFO:teuthology.orchestra.run.smithi155.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-02-10T08:45:37.797 INFO:teuthology.orchestra.run.smithi155.stdout:Installing module profiles: 2024-02-10T08:45:37.797 INFO:teuthology.orchestra.run.smithi155.stdout: container-tools/common 2024-02-10T08:45:37.797 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:45:37.797 INFO:teuthology.orchestra.run.smithi155.stdout:Enabling module streams: 2024-02-10T08:45:37.797 INFO:teuthology.orchestra.run.smithi155.stdout: container-tools rhel8 2024-02-10T08:45:37.797 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:45:37.797 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-02-10T08:45:37.797 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-02-10T08:45:37.797 INFO:teuthology.orchestra.run.smithi155.stdout:Install 13 Packages 2024-02-10T08:45:37.797 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:45:37.798 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 27 M 2024-02-10T08:45:37.798 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 91 M 2024-02-10T08:45:37.798 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-02-10T08:45:38.184 INFO:teuthology.orchestra.run.smithi077.stdout:(1/13): cockpit-podman-82-1.module_el8+804+f131 2.1 MB/s | 759 kB 00:00 2024-02-10T08:45:38.234 INFO:teuthology.orchestra.run.smithi077.stdout:(2/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 760 kB/s | 38 kB 00:00 2024-02-10T08:45:38.309 INFO:teuthology.orchestra.run.smithi077.stdout:(3/13): aardvark-dns-1.9.0-1.module_el8+804+f13 2.1 MB/s | 1.0 MB 00:00 2024-02-10T08:45:38.360 INFO:teuthology.orchestra.run.smithi077.stdout:(4/13): crun-1.12-1.module_el8+804+f131391c.x86 1.9 MB/s | 248 kB 00:00 2024-02-10T08:45:38.436 INFO:teuthology.orchestra.run.smithi155.stdout:(1/13): aardvark-dns-1.9.0-1.module_el8+804+f13 1.7 MB/s | 1.0 MB 00:00 2024-02-10T08:45:38.443 INFO:teuthology.orchestra.run.smithi077.stdout:(5/13): python3-podman-4.8.2-1.module_el8+804+f 2.1 MB/s | 179 kB 00:00 2024-02-10T08:45:38.469 INFO:teuthology.orchestra.run.smithi155.stdout:(2/13): cockpit-podman-82-1.module_el8+804+f131 1.2 MB/s | 759 kB 00:00 2024-02-10T08:45:38.494 INFO:teuthology.orchestra.run.smithi155.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 649 kB/s | 38 kB 00:00 2024-02-10T08:45:38.510 INFO:teuthology.orchestra.run.smithi077.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 372 kB/s | 25 kB 00:00 2024-02-10T08:45:38.577 INFO:teuthology.orchestra.run.smithi077.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.4 MB/s | 94 kB 00:00 2024-02-10T08:45:38.595 INFO:teuthology.orchestra.run.smithi155.stdout:(4/13): crun-1.12-1.module_el8+804+f131391c.x86 1.9 MB/s | 248 kB 00:00 2024-02-10T08:45:38.779 INFO:teuthology.orchestra.run.smithi155.stdout:(5/13): python3-podman-4.8.2-1.module_el8+804+f 981 kB/s | 179 kB 00:00 2024-02-10T08:45:38.845 INFO:teuthology.orchestra.run.smithi155.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 371 kB/s | 25 kB 00:00 2024-02-10T08:45:38.903 INFO:teuthology.orchestra.run.smithi077.stdout:(8/13): netavark-1.9.0-1.module_el8+804+f131391 6.7 MB/s | 4.0 MB 00:00 2024-02-10T08:45:38.921 INFO:teuthology.orchestra.run.smithi155.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.2 MB/s | 94 kB 00:00 2024-02-10T08:45:39.096 INFO:teuthology.orchestra.run.smithi155.stdout:(8/13): buildah-1.34.0-1.module_el8+804+f131391 7.7 MB/s | 9.7 MB 00:01 2024-02-10T08:45:39.221 INFO:teuthology.orchestra.run.smithi155.stdout:(9/13): netavark-1.9.0-1.module_el8+804+f131391 5.4 MB/s | 4.0 MB 00:00 2024-02-10T08:45:39.280 INFO:teuthology.orchestra.run.smithi155.stdout:(10/13): udica-0.2.6-20.module_el8+804+f131391c 901 kB/s | 53 kB 00:00 2024-02-10T08:45:39.339 INFO:teuthology.orchestra.run.smithi155.stdout:(11/13): yajl-2.1.0-12.el8.x86_64.rpm 704 kB/s | 41 kB 00:00 2024-02-10T08:45:39.353 INFO:teuthology.orchestra.run.smithi077.stdout:(9/13): skopeo-1.14.0-2.module_el8+804+f131391c 11 MB/s | 8.8 MB 00:00 2024-02-10T08:45:39.403 INFO:teuthology.orchestra.run.smithi077.stdout:(10/13): toolbox-0.0.99.4-8.module_el8+804+f131 5.0 MB/s | 2.5 MB 00:00 2024-02-10T08:45:39.428 INFO:teuthology.orchestra.run.smithi077.stdout:(11/13): udica-0.2.6-20.module_el8+804+f131391c 701 kB/s | 53 kB 00:00 2024-02-10T08:45:39.462 INFO:teuthology.orchestra.run.smithi077.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 695 kB/s | 41 kB 00:00 2024-02-10T08:45:39.580 INFO:teuthology.orchestra.run.smithi077.stdout:(13/13): buildah-1.34.0-1.module_el8+804+f13139 5.5 MB/s | 9.7 MB 00:01 2024-02-10T08:45:39.580 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:45:39.580 INFO:teuthology.orchestra.run.smithi077.stdout:Total 15 MB/s | 27 MB 00:01 2024-02-10T08:45:39.645 INFO:teuthology.orchestra.run.smithi155.stdout:(12/13): toolbox-0.0.99.4-8.module_el8+804+f131 4.6 MB/s | 2.5 MB 00:00 2024-02-10T08:45:39.847 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2024-02-10T08:45:39.868 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2024-02-10T08:45:39.868 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2024-02-10T08:45:40.129 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2024-02-10T08:45:40.132 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2024-02-10T08:45:40.564 INFO:teuthology.orchestra.run.smithi155.stdout:(13/13): skopeo-1.14.0-2.module_el8+804+f131391 5.4 MB/s | 8.8 MB 00:01 2024-02-10T08:45:40.565 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:45:40.565 INFO:teuthology.orchestra.run.smithi155.stdout:Total 9.9 MB/s | 27 MB 00:02 2024-02-10T08:45:40.837 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2024-02-10T08:45:40.838 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-02-10T08:45:40.860 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-02-10T08:45:40.860 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-02-10T08:45:41.083 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-02-10T08:45:41.118 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-02-10T08:45:41.122 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-02-10T08:45:41.286 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-02-10T08:45:41.484 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2024-02-10T08:45:41.739 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-02-10T08:45:41.855 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-02-10T08:45:42.080 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-02-10T08:45:42.300 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-02-10T08:45:42.302 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : aardvark-dns-2:1.9.0-1.module_el8+804+f131391c.x86 5/13 2024-02-10T08:45:42.510 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2024-02-10T08:45:42.776 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-02-10T08:45:43.258 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : netavark-2:1.9.0-1.module_el8+804+f131391c.x86_64 6/13 2024-02-10T08:45:43.365 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : aardvark-dns-2:1.9.0-1.module_el8+804+f131391c.x86 5/13 2024-02-10T08:45:43.515 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : buildah-1:1.34.0-1.module_el8+804+f131391c.x86_64 7/13 2024-02-10T08:45:43.752 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : crun-1.12-1.module_el8+804+f131391c.x86_64 8/13 2024-02-10T08:45:44.032 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-podman-4.8.2-1.module_el8+804+f131391c.noa 9/13 2024-02-10T08:45:44.362 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : netavark-2:1.9.0-1.module_el8+804+f131391c.x86_64 6/13 2024-02-10T08:45:44.460 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 10/13 2024-02-10T08:45:44.579 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : buildah-1:1.34.0-1.module_el8+804+f131391c.x86_64 7/13 2024-02-10T08:45:44.824 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : crun-1.12-1.module_el8+804+f131391c.x86_64 8/13 2024-02-10T08:45:45.112 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-podman-4.8.2-1.module_el8+804+f131391c.noa 9/13 2024-02-10T08:45:45.426 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : toolbox-0.0.99.4-8.module_el8+804+f131391c.x86_64 11/13 2024-02-10T08:45:45.536 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 10/13 2024-02-10T08:45:45.633 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : skopeo-2:1.14.0-2.module_el8+804+f131391c.x86_64 12/13 2024-02-10T08:45:45.827 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : cockpit-podman-82-1.module_el8+804+f131391c.noarch 13/13 2024-02-10T08:45:46.273 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: cockpit-podman-82-1.module_el8+804+f131391c.noarch 13/13 2024-02-10T08:45:46.273 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : aardvark-dns-2:1.9.0-1.module_el8+804+f131391c.x86 1/13 2024-02-10T08:45:46.274 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : buildah-1:1.34.0-1.module_el8+804+f131391c.x86_64 2/13 2024-02-10T08:45:46.274 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : cockpit-podman-82-1.module_el8+804+f131391c.noarch 3/13 2024-02-10T08:45:46.274 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-02-10T08:45:46.274 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : crun-1.12-1.module_el8+804+f131391c.x86_64 5/13 2024-02-10T08:45:46.274 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : netavark-2:1.9.0-1.module_el8+804+f131391c.x86_64 6/13 2024-02-10T08:45:46.274 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-podman-4.8.2-1.module_el8+804+f131391c.noa 7/13 2024-02-10T08:45:46.274 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-02-10T08:45:46.274 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-02-10T08:45:46.274 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : skopeo-2:1.14.0-2.module_el8+804+f131391c.x86_64 10/13 2024-02-10T08:45:46.274 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : toolbox-0.0.99.4-8.module_el8+804+f131391c.x86_64 11/13 2024-02-10T08:45:46.274 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-10T08:45:46.502 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : toolbox-0.0.99.4-8.module_el8+804+f131391c.x86_64 11/13 2024-02-10T08:45:46.761 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : skopeo-2:1.14.0-2.module_el8+804+f131391c.x86_64 12/13 2024-02-10T08:45:46.802 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-02-10T08:45:46.802 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:45:46.802 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2024-02-10T08:45:46.803 INFO:teuthology.orchestra.run.smithi077.stdout: aardvark-dns-2:1.9.0-1.module_el8+804+f131391c.x86_64 2024-02-10T08:45:46.803 INFO:teuthology.orchestra.run.smithi077.stdout: buildah-1:1.34.0-1.module_el8+804+f131391c.x86_64 2024-02-10T08:45:46.803 INFO:teuthology.orchestra.run.smithi077.stdout: cockpit-podman-82-1.module_el8+804+f131391c.noarch 2024-02-10T08:45:46.803 INFO:teuthology.orchestra.run.smithi077.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-02-10T08:45:46.803 INFO:teuthology.orchestra.run.smithi077.stdout: crun-1.12-1.module_el8+804+f131391c.x86_64 2024-02-10T08:45:46.803 INFO:teuthology.orchestra.run.smithi077.stdout: netavark-2:1.9.0-1.module_el8+804+f131391c.x86_64 2024-02-10T08:45:46.803 INFO:teuthology.orchestra.run.smithi077.stdout: python3-podman-4.8.2-1.module_el8+804+f131391c.noarch 2024-02-10T08:45:46.803 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-02-10T08:45:46.803 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-02-10T08:45:46.803 INFO:teuthology.orchestra.run.smithi077.stdout: skopeo-2:1.14.0-2.module_el8+804+f131391c.x86_64 2024-02-10T08:45:46.803 INFO:teuthology.orchestra.run.smithi077.stdout: toolbox-0.0.99.4-8.module_el8+804+f131391c.x86_64 2024-02-10T08:45:46.803 INFO:teuthology.orchestra.run.smithi077.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-02-10T08:45:46.803 INFO:teuthology.orchestra.run.smithi077.stdout: yajl-2.1.0-12.el8.x86_64 2024-02-10T08:45:46.803 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:45:46.803 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-02-10T08:45:46.998 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : cockpit-podman-82-1.module_el8+804+f131391c.noarch 13/13 2024-02-10T08:45:47.123 DEBUG:teuthology.parallel:result is None 2024-02-10T08:45:47.423 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: cockpit-podman-82-1.module_el8+804+f131391c.noarch 13/13 2024-02-10T08:45:47.424 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : aardvark-dns-2:1.9.0-1.module_el8+804+f131391c.x86 1/13 2024-02-10T08:45:47.424 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : buildah-1:1.34.0-1.module_el8+804+f131391c.x86_64 2/13 2024-02-10T08:45:47.424 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : cockpit-podman-82-1.module_el8+804+f131391c.noarch 3/13 2024-02-10T08:45:47.424 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-02-10T08:45:47.424 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : crun-1.12-1.module_el8+804+f131391c.x86_64 5/13 2024-02-10T08:45:47.424 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : netavark-2:1.9.0-1.module_el8+804+f131391c.x86_64 6/13 2024-02-10T08:45:47.424 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-podman-4.8.2-1.module_el8+804+f131391c.noa 7/13 2024-02-10T08:45:47.424 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-02-10T08:45:47.424 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-02-10T08:45:47.424 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : skopeo-2:1.14.0-2.module_el8+804+f131391c.x86_64 10/13 2024-02-10T08:45:47.424 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : toolbox-0.0.99.4-8.module_el8+804+f131391c.x86_64 11/13 2024-02-10T08:45:47.424 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-10T08:45:47.945 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-02-10T08:45:47.946 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:45:47.946 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-02-10T08:45:47.946 INFO:teuthology.orchestra.run.smithi155.stdout: aardvark-dns-2:1.9.0-1.module_el8+804+f131391c.x86_64 2024-02-10T08:45:47.946 INFO:teuthology.orchestra.run.smithi155.stdout: buildah-1:1.34.0-1.module_el8+804+f131391c.x86_64 2024-02-10T08:45:47.946 INFO:teuthology.orchestra.run.smithi155.stdout: cockpit-podman-82-1.module_el8+804+f131391c.noarch 2024-02-10T08:45:47.946 INFO:teuthology.orchestra.run.smithi155.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-02-10T08:45:47.946 INFO:teuthology.orchestra.run.smithi155.stdout: crun-1.12-1.module_el8+804+f131391c.x86_64 2024-02-10T08:45:47.946 INFO:teuthology.orchestra.run.smithi155.stdout: netavark-2:1.9.0-1.module_el8+804+f131391c.x86_64 2024-02-10T08:45:47.946 INFO:teuthology.orchestra.run.smithi155.stdout: python3-podman-4.8.2-1.module_el8+804+f131391c.noarch 2024-02-10T08:45:47.946 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-02-10T08:45:47.946 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-02-10T08:45:47.947 INFO:teuthology.orchestra.run.smithi155.stdout: skopeo-2:1.14.0-2.module_el8+804+f131391c.x86_64 2024-02-10T08:45:47.947 INFO:teuthology.orchestra.run.smithi155.stdout: toolbox-0.0.99.4-8.module_el8+804+f131391c.x86_64 2024-02-10T08:45:47.947 INFO:teuthology.orchestra.run.smithi155.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-02-10T08:45:47.947 INFO:teuthology.orchestra.run.smithi155.stdout: yajl-2.1.0-12.el8.x86_64 2024-02-10T08:45:47.947 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:45:47.947 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-02-10T08:45:48.172 DEBUG:teuthology.parallel:result is None 2024-02-10T08:45:48.172 INFO:teuthology.run_tasks:Running task pexec... 2024-02-10T08:45:48.181 INFO:teuthology.task.pexec:Executing custom commands... 2024-02-10T08:45:48.181 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi077.front.sepia.ceph.com 2024-02-10T08:45:48.181 DEBUG:teuthology.orchestra.run.smithi077:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-10T08:45:48.182 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi155.front.sepia.ceph.com 2024-02-10T08:45:48.182 DEBUG:teuthology.orchestra.run.smithi155:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-10T08:45:48.336 DEBUG:teuthology.parallel:result is None 2024-02-10T08:45:48.392 DEBUG:teuthology.parallel:result is None 2024-02-10T08:45:48.392 INFO:teuthology.run_tasks:Running task install... 2024-02-10T08:45:48.401 DEBUG:teuthology.task.install:project ceph 2024-02-10T08:45:48.401 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '0e714d9a4bd2a821113e6318adb87bd06cf81ec1'}} 2024-02-10T08:45:48.401 DEBUG:teuthology.task.install:config {'exclude_packages': ['ceph-volume'], 'tag': 'v16.2.4', 'flavor': 'default', 'sha1': '0e714d9a4bd2a821113e6318adb87bd06cf81ec1'} 2024-02-10T08:45:48.401 INFO:teuthology.task.install:Using flavor: default 2024-02-10T08:45:48.409 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-02-10T08:45:48.409 INFO:teuthology.task.install:extra packages: [] 2024-02-10T08:45:48.409 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '0e714d9a4bd2a821113e6318adb87bd06cf81ec1', 'tag': 'v16.2.4', 'wait_for_package': False} 2024-02-10T08:45:48.409 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-10T08:45:48.410 INFO:teuthology.packaging:ref: None 2024-02-10T08:45:48.410 INFO:teuthology.packaging:tag: v16.2.4 2024-02-10T08:45:48.410 INFO:teuthology.packaging:branch: None 2024-02-10T08:45:48.410 INFO:teuthology.packaging:sha1: 0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-10T08:45:48.839 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-10T08:45:48.840 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-10T08:45:48.843 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '0e714d9a4bd2a821113e6318adb87bd06cf81ec1', 'tag': 'v16.2.4', 'wait_for_package': False} 2024-02-10T08:45:48.843 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-10T08:45:48.843 INFO:teuthology.packaging:ref: None 2024-02-10T08:45:48.843 INFO:teuthology.packaging:tag: v16.2.4 2024-02-10T08:45:48.843 INFO:teuthology.packaging:branch: None 2024-02-10T08:45:48.843 INFO:teuthology.packaging:sha1: 0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-10T08:45:48.844 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-10T08:45:49.063 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2024-02-10T08:45:49.064 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2024-02-10T08:45:49.078 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2024-02-10T08:45:49.078 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2024-02-10T08:45:49.101 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-10T08:45:49.101 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-02-10T08:45:49.101 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-10T08:45:49.109 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-10T08:45:49.109 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-02-10T08:45:49.109 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-10T08:45:49.133 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-10T08:45:49.134 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-10T08:45:49.134 INFO:teuthology.packaging:ref: None 2024-02-10T08:45:49.134 INFO:teuthology.packaging:tag: v16.2.4 2024-02-10T08:45:49.134 INFO:teuthology.packaging:branch: None 2024-02-10T08:45:49.134 INFO:teuthology.packaging:sha1: 0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-10T08:45:49.134 DEBUG:teuthology.orchestra.run.smithi077:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-10T08:45:49.139 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-10T08:45:49.139 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-10T08:45:49.139 INFO:teuthology.packaging:ref: None 2024-02-10T08:45:49.139 INFO:teuthology.packaging:tag: v16.2.4 2024-02-10T08:45:49.139 INFO:teuthology.packaging:branch: None 2024-02-10T08:45:49.139 INFO:teuthology.packaging:sha1: 0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-10T08:45:49.140 DEBUG:teuthology.orchestra.run.smithi155:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-10T08:45:49.206 DEBUG:teuthology.orchestra.run.smithi077:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-10T08:45:49.209 DEBUG:teuthology.orchestra.run.smithi155:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-10T08:45:49.287 DEBUG:teuthology.orchestra.run.smithi077:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-10T08:45:49.289 DEBUG:teuthology.orchestra.run.smithi155:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-10T08:45:49.356 INFO:teuthology.orchestra.run.smithi077.stdout:check_obsoletes = 1 2024-02-10T08:45:49.356 INFO:teuthology.orchestra.run.smithi155.stdout:check_obsoletes = 1 2024-02-10T08:45:49.357 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum clean all 2024-02-10T08:45:49.358 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum clean all 2024-02-10T08:45:49.623 INFO:teuthology.orchestra.run.smithi077.stdout:76 files removed 2024-02-10T08:45:49.624 INFO:teuthology.orchestra.run.smithi155.stdout:76 files removed 2024-02-10T08:45:49.651 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph-radosgw 2024-02-10T08:45:49.652 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-radosgw 2024-02-10T08:45:50.137 INFO:teuthology.orchestra.run.smithi077.stdout:ceph packages for x86_64 923 kB/s | 74 kB 00:00 2024-02-10T08:45:50.139 INFO:teuthology.orchestra.run.smithi155.stdout:ceph packages for x86_64 944 kB/s | 74 kB 00:00 2024-02-10T08:45:50.254 INFO:teuthology.orchestra.run.smithi077.stdout:ceph noarch packages 192 kB/s | 15 kB 00:00 2024-02-10T08:45:50.265 INFO:teuthology.orchestra.run.smithi155.stdout:ceph noarch packages 173 kB/s | 15 kB 00:00 2024-02-10T08:45:50.364 INFO:teuthology.orchestra.run.smithi077.stdout:ceph source packages 22 kB/s | 1.7 kB 00:00 2024-02-10T08:45:50.366 INFO:teuthology.orchestra.run.smithi155.stdout:ceph source packages 24 kB/s | 1.7 kB 00:00 2024-02-10T08:45:52.348 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 8 - AppStream 14 MB/s | 27 MB 00:01 2024-02-10T08:45:53.300 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 8 - AppStream 9.2 MB/s | 27 MB 00:02 2024-02-10T08:46:01.298 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 8 - BaseOS 2.5 MB/s | 10 MB 00:04 2024-02-10T08:46:01.509 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 8 - BaseOS 3.1 MB/s | 10 MB 00:03 2024-02-10T08:46:03.107 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 8 - Extras 88 kB/s | 18 kB 00:00 2024-02-10T08:46:03.433 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 8 - Extras common packages 32 kB/s | 6.9 kB 00:00 2024-02-10T08:46:03.476 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 8 - Extras 53 kB/s | 18 kB 00:00 2024-02-10T08:46:03.935 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 8 - Extras common packages 20 kB/s | 6.9 kB 00:00 2024-02-10T08:46:05.717 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 8 - PowerTools 2.2 MB/s | 4.8 MB 00:02 2024-02-10T08:46:05.895 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 8 - PowerTools 2.5 MB/s | 4.8 MB 00:01 2024-02-10T08:46:06.744 INFO:teuthology.orchestra.run.smithi077.stdout:Copr repo for python3-asyncssh owned by ceph 27 kB/s | 3.5 kB 00:00 2024-02-10T08:46:06.963 INFO:teuthology.orchestra.run.smithi155.stdout:Copr repo for python3-asyncssh owned by ceph 23 kB/s | 3.5 kB 00:00 2024-02-10T08:46:08.080 INFO:teuthology.orchestra.run.smithi155.stdout:Extra Packages for Enterprise Linux 16 MB/s | 16 MB 00:01 2024-02-10T08:46:08.338 INFO:teuthology.orchestra.run.smithi077.stdout:Extra Packages for Enterprise Linux 11 MB/s | 16 MB 00:01 2024-02-10T08:46:11.939 INFO:teuthology.orchestra.run.smithi155.stdout:lab-extras 278 kB/s | 24 kB 00:00 2024-02-10T08:46:12.144 INFO:teuthology.orchestra.run.smithi077.stdout:lab-extras 396 kB/s | 24 kB 00:00 2024-02-10T08:46:15.582 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-02-10T08:46:15.585 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-02-10T08:46:15.585 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-02-10T08:46:15.585 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-02-10T08:46:15.585 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-02-10T08:46:15.585 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2024-02-10T08:46:15.585 INFO:teuthology.orchestra.run.smithi155.stdout:Upgrading: 2024-02-10T08:46:15.585 INFO:teuthology.orchestra.run.smithi155.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-02-10T08:46:15.585 INFO:teuthology.orchestra.run.smithi155.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2024-02-10T08:46:15.585 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-02-10T08:46:15.585 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2024-02-10T08:46:15.586 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2024-02-10T08:46:15.586 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2024-02-10T08:46:15.586 INFO:teuthology.orchestra.run.smithi155.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2024-02-10T08:46:15.586 INFO:teuthology.orchestra.run.smithi155.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2024-02-10T08:46:15.586 INFO:teuthology.orchestra.run.smithi155.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-02-10T08:46:15.586 INFO:teuthology.orchestra.run.smithi155.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-02-10T08:46:15.586 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2024-02-10T08:46:15.586 INFO:teuthology.orchestra.run.smithi155.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-02-10T08:46:15.586 INFO:teuthology.orchestra.run.smithi155.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2024-02-10T08:46:15.586 INFO:teuthology.orchestra.run.smithi155.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-02-10T08:46:15.586 INFO:teuthology.orchestra.run.smithi155.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2024-02-10T08:46:15.586 INFO:teuthology.orchestra.run.smithi155.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-02-10T08:46:15.586 INFO:teuthology.orchestra.run.smithi155.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2024-02-10T08:46:15.587 INFO:teuthology.orchestra.run.smithi155.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-02-10T08:46:15.587 INFO:teuthology.orchestra.run.smithi155.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-02-10T08:46:15.587 INFO:teuthology.orchestra.run.smithi155.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2024-02-10T08:46:15.587 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2024-02-10T08:46:15.587 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2024-02-10T08:46:15.587 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2024-02-10T08:46:15.587 INFO:teuthology.orchestra.run.smithi155.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-02-10T08:46:15.587 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2024-02-10T08:46:15.587 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2024-02-10T08:46:15.587 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2024-02-10T08:46:15.587 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:46:15.587 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-02-10T08:46:15.587 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-02-10T08:46:15.587 INFO:teuthology.orchestra.run.smithi155.stdout:Install 25 Packages 2024-02-10T08:46:15.587 INFO:teuthology.orchestra.run.smithi155.stdout:Upgrade 2 Packages 2024-02-10T08:46:15.588 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:46:15.588 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 59 M 2024-02-10T08:46:15.588 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-02-10T08:46:15.642 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-02-10T08:46:15.646 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-10T08:46:15.646 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repository Size 2024-02-10T08:46:15.646 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-10T08:46:15.646 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2024-02-10T08:46:15.646 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2024-02-10T08:46:15.646 INFO:teuthology.orchestra.run.smithi077.stdout:Upgrading: 2024-02-10T08:46:15.646 INFO:teuthology.orchestra.run.smithi077.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-02-10T08:46:15.646 INFO:teuthology.orchestra.run.smithi077.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2024-02-10T08:46:15.646 INFO:teuthology.orchestra.run.smithi077.stdout:Installing dependencies: 2024-02-10T08:46:15.646 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2024-02-10T08:46:15.646 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2024-02-10T08:46:15.646 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2024-02-10T08:46:15.647 INFO:teuthology.orchestra.run.smithi077.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2024-02-10T08:46:15.647 INFO:teuthology.orchestra.run.smithi077.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2024-02-10T08:46:15.647 INFO:teuthology.orchestra.run.smithi077.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-02-10T08:46:15.647 INFO:teuthology.orchestra.run.smithi077.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-02-10T08:46:15.647 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2024-02-10T08:46:15.647 INFO:teuthology.orchestra.run.smithi077.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-02-10T08:46:15.647 INFO:teuthology.orchestra.run.smithi077.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2024-02-10T08:46:15.647 INFO:teuthology.orchestra.run.smithi077.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-02-10T08:46:15.647 INFO:teuthology.orchestra.run.smithi077.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2024-02-10T08:46:15.647 INFO:teuthology.orchestra.run.smithi077.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-02-10T08:46:15.648 INFO:teuthology.orchestra.run.smithi077.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2024-02-10T08:46:15.648 INFO:teuthology.orchestra.run.smithi077.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-02-10T08:46:15.648 INFO:teuthology.orchestra.run.smithi077.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-02-10T08:46:15.648 INFO:teuthology.orchestra.run.smithi077.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2024-02-10T08:46:15.648 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2024-02-10T08:46:15.648 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2024-02-10T08:46:15.648 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2024-02-10T08:46:15.648 INFO:teuthology.orchestra.run.smithi077.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-02-10T08:46:15.648 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2024-02-10T08:46:15.648 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2024-02-10T08:46:15.648 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2024-02-10T08:46:15.648 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:46:15.648 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2024-02-10T08:46:15.649 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-10T08:46:15.649 INFO:teuthology.orchestra.run.smithi077.stdout:Install 25 Packages 2024-02-10T08:46:15.649 INFO:teuthology.orchestra.run.smithi077.stdout:Upgrade 2 Packages 2024-02-10T08:46:15.649 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:46:15.649 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 59 M 2024-02-10T08:46:15.649 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2024-02-10T08:46:16.013 INFO:teuthology.orchestra.run.smithi077.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 38 MB/s | 6.0 MB 00:00 2024-02-10T08:46:16.030 INFO:teuthology.orchestra.run.smithi077.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 1.5 MB/s | 25 kB 00:00 2024-02-10T08:46:16.063 INFO:teuthology.orchestra.run.smithi077.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 24 MB/s | 801 kB 00:00 2024-02-10T08:46:16.097 INFO:teuthology.orchestra.run.smithi077.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 14 MB/s | 486 kB 00:00 2024-02-10T08:46:16.140 INFO:teuthology.orchestra.run.smithi155.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 45 MB/s | 6.0 MB 00:00 2024-02-10T08:46:16.165 INFO:teuthology.orchestra.run.smithi155.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 1.0 MB/s | 25 kB 00:00 2024-02-10T08:46:16.197 INFO:teuthology.orchestra.run.smithi077.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 41 MB/s | 4.1 MB 00:00 2024-02-10T08:46:16.199 INFO:teuthology.orchestra.run.smithi155.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 23 MB/s | 801 kB 00:00 2024-02-10T08:46:16.223 INFO:teuthology.orchestra.run.smithi077.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 1.8 MB/s | 44 kB 00:00 2024-02-10T08:46:16.232 INFO:teuthology.orchestra.run.smithi155.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 14 MB/s | 486 kB 00:00 2024-02-10T08:46:16.256 INFO:teuthology.orchestra.run.smithi077.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 2.3 MB/s | 80 kB 00:00 2024-02-10T08:46:16.273 INFO:teuthology.orchestra.run.smithi077.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 13 MB/s | 213 kB 00:00 2024-02-10T08:46:16.337 INFO:teuthology.orchestra.run.smithi077.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 5.9 MB/s | 387 kB 00:00 2024-02-10T08:46:16.349 INFO:teuthology.orchestra.run.smithi155.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 35 MB/s | 4.1 MB 00:00 2024-02-10T08:46:16.374 INFO:teuthology.orchestra.run.smithi077.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 10 MB/s | 368 kB 00:00 2024-02-10T08:46:16.374 INFO:teuthology.orchestra.run.smithi155.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 1.8 MB/s | 44 kB 00:00 2024-02-10T08:46:16.399 INFO:teuthology.orchestra.run.smithi077.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 4.5 MB/s | 114 kB 00:00 2024-02-10T08:46:16.400 INFO:teuthology.orchestra.run.smithi155.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 3.1 MB/s | 80 kB 00:00 2024-02-10T08:46:16.425 INFO:teuthology.orchestra.run.smithi155.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 8.3 MB/s | 213 kB 00:00 2024-02-10T08:46:16.459 INFO:teuthology.orchestra.run.smithi155.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 11 MB/s | 387 kB 00:00 2024-02-10T08:46:16.493 INFO:teuthology.orchestra.run.smithi155.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 11 MB/s | 368 kB 00:00 2024-02-10T08:46:16.518 INFO:teuthology.orchestra.run.smithi155.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 4.5 MB/s | 114 kB 00:00 2024-02-10T08:46:16.557 INFO:teuthology.orchestra.run.smithi077.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 18 MB/s | 13 MB 00:00 2024-02-10T08:46:16.677 INFO:teuthology.orchestra.run.smithi155.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 19 MB/s | 13 MB 00:00 2024-02-10T08:46:16.841 INFO:teuthology.orchestra.run.smithi077.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 24 MB/s | 24 MB 00:00 2024-02-10T08:46:16.866 INFO:teuthology.orchestra.run.smithi077.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 311 kB/s | 145 kB 00:00 2024-02-10T08:46:16.933 INFO:teuthology.orchestra.run.smithi077.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 658 kB/s | 44 kB 00:00 2024-02-10T08:46:16.977 INFO:teuthology.orchestra.run.smithi155.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 25 MB/s | 24 MB 00:00 2024-02-10T08:46:17.002 INFO:teuthology.orchestra.run.smithi155.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 300 kB/s | 145 kB 00:00 2024-02-10T08:46:17.042 INFO:teuthology.orchestra.run.smithi077.stdout:(16/27): daxctl-libs-71.1-7.el8.x86_64.rpm 388 kB/s | 42 kB 00:00 2024-02-10T08:46:17.067 INFO:teuthology.orchestra.run.smithi077.stdout:(17/27): librdkafka-1.6.1-1.el8.x86_64.rpm 1.3 MB/s | 677 kB 00:00 2024-02-10T08:46:17.069 INFO:teuthology.orchestra.run.smithi155.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 660 kB/s | 44 kB 00:00 2024-02-10T08:46:17.101 INFO:teuthology.orchestra.run.smithi077.stdout:(18/27): librabbitmq-0.9.0-5.el8.x86_64.rpm 803 kB/s | 47 kB 00:00 2024-02-10T08:46:17.136 INFO:teuthology.orchestra.run.smithi155.stdout:(16/27): daxctl-libs-71.1-7.el8.x86_64.rpm 633 kB/s | 42 kB 00:00 2024-02-10T08:46:17.168 INFO:teuthology.orchestra.run.smithi077.stdout:(19/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 795 kB/s | 259 kB 00:00 2024-02-10T08:46:17.169 INFO:teuthology.orchestra.run.smithi155.stdout:(17/27): librdkafka-1.6.1-1.el8.x86_64.rpm 1.3 MB/s | 677 kB 00:00 2024-02-10T08:46:17.193 INFO:teuthology.orchestra.run.smithi077.stdout:(20/27): ndctl-libs-71.1-7.el8.x86_64.rpm 638 kB/s | 80 kB 00:00 2024-02-10T08:46:17.245 INFO:teuthology.orchestra.run.smithi155.stdout:(18/27): librabbitmq-0.9.0-5.el8.x86_64.rpm 435 kB/s | 47 kB 00:00 2024-02-10T08:46:17.279 INFO:teuthology.orchestra.run.smithi155.stdout:(19/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 862 kB/s | 259 kB 00:00 2024-02-10T08:46:17.304 INFO:teuthology.orchestra.run.smithi155.stdout:(20/27): ndctl-libs-71.1-7.el8.x86_64.rpm 598 kB/s | 80 kB 00:00 2024-02-10T08:46:17.404 INFO:teuthology.orchestra.run.smithi155.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 732 kB/s | 116 kB 00:00 2024-02-10T08:46:17.429 INFO:teuthology.orchestra.run.smithi155.stdout:(22/27): leveldb-1.22-1.el8.x86_64.rpm 1.4 MB/s | 181 kB 00:00 2024-02-10T08:46:17.455 INFO:teuthology.orchestra.run.smithi155.stdout:(23/27): gperftools-libs-2.7-9.el8.x86_64.rpm 1.7 MB/s | 306 kB 00:00 2024-02-10T08:46:17.480 INFO:teuthology.orchestra.run.smithi155.stdout:(24/27): liboath-2.6.2-3.el8.x86_64.rpm 780 kB/s | 59 kB 00:00 2024-02-10T08:46:17.530 INFO:teuthology.orchestra.run.smithi155.stdout:(25/27): libunwind-1.3.1-3.el8.x86_64.rpm 748 kB/s | 75 kB 00:00 2024-02-10T08:46:17.614 INFO:teuthology.orchestra.run.smithi155.stdout:(26/27): librados2-16.2.4-0.el8.x86_64.rpm 28 MB/s | 3.7 MB 00:00 2024-02-10T08:46:17.672 INFO:teuthology.orchestra.run.smithi155.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 21 MB/s | 4.0 MB 00:00 2024-02-10T08:46:17.674 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:46:17.675 INFO:teuthology.orchestra.run.smithi155.stdout:Total 28 MB/s | 59 MB 00:02 2024-02-10T08:46:17.744 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-02-10T08:46:17.792 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-02-10T08:46:17.793 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-02-10T08:46:18.140 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-02-10T08:46:18.143 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-02-10T08:46:18.385 INFO:teuthology.orchestra.run.smithi077.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 91 kB/s | 116 kB 00:01 2024-02-10T08:46:18.569 INFO:teuthology.orchestra.run.smithi077.stdout:(22/27): leveldb-1.22-1.el8.x86_64.rpm 132 kB/s | 181 kB 00:01 2024-02-10T08:46:18.644 INFO:teuthology.orchestra.run.smithi077.stdout:(23/27): gperftools-libs-2.7-9.el8.x86_64.rpm 207 kB/s | 306 kB 00:01 2024-02-10T08:46:18.670 INFO:teuthology.orchestra.run.smithi077.stdout:(24/27): liboath-2.6.2-3.el8.x86_64.rpm 207 kB/s | 59 kB 00:00 2024-02-10T08:46:18.762 INFO:teuthology.orchestra.run.smithi077.stdout:(25/27): librados2-16.2.4-0.el8.x86_64.rpm 32 MB/s | 3.7 MB 00:00 2024-02-10T08:46:18.828 INFO:teuthology.orchestra.run.smithi077.stdout:(26/27): librbd1-16.2.4-0.el8.x86_64.rpm 26 MB/s | 4.0 MB 00:00 2024-02-10T08:46:18.853 INFO:teuthology.orchestra.run.smithi077.stdout:(27/27): libunwind-1.3.1-3.el8.x86_64.rpm 264 kB/s | 75 kB 00:00 2024-02-10T08:46:18.858 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:46:18.861 INFO:teuthology.orchestra.run.smithi077.stdout:Total 18 MB/s | 59 MB 00:03 2024-02-10T08:46:18.929 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2024-02-10T08:46:18.978 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2024-02-10T08:46:18.978 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2024-02-10T08:46:19.322 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2024-02-10T08:46:19.325 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2024-02-10T08:46:19.340 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-02-10T08:46:19.590 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2024-02-10T08:46:19.827 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2024-02-10T08:46:20.019 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-02-10T08:46:20.430 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-02-10T08:46:20.752 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2024-02-10T08:46:20.857 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2024-02-10T08:46:20.921 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2024-02-10T08:46:21.127 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2024-02-10T08:46:21.139 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2024-02-10T08:46:21.302 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-02-10T08:46:21.352 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-02-10T08:46:21.385 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2024-02-10T08:46:21.540 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-02-10T08:46:21.595 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/29 2024-02-10T08:46:21.971 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-02-10T08:46:22.161 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/29 2024-02-10T08:46:22.182 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2024-02-10T08:46:22.299 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2024-02-10T08:46:22.344 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-02-10T08:46:22.426 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-02-10T08:46:22.531 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2024-02-10T08:46:22.640 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2024-02-10T08:46:22.748 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-02-10T08:46:22.794 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2024-02-10T08:46:22.816 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-02-10T08:46:22.891 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2024-02-10T08:46:23.049 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/29 2024-02-10T08:46:23.065 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2024-02-10T08:46:23.277 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2024-02-10T08:46:23.469 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2024-02-10T08:46:23.542 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2024-02-10T08:46:23.584 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/29 2024-02-10T08:46:23.736 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-02-10T08:46:23.748 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-02-10T08:46:23.815 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-02-10T08:46:24.011 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2024-02-10T08:46:24.165 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2024-02-10T08:46:24.171 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-02-10T08:46:24.229 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2024-02-10T08:46:24.402 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-02-10T08:46:24.461 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2024-02-10T08:46:24.499 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-02-10T08:46:24.640 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2024-02-10T08:46:24.689 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2024-02-10T08:46:24.823 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2024-02-10T08:46:24.897 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2024-02-10T08:46:24.920 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2024-02-10T08:46:25.085 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-02-10T08:46:25.100 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-02-10T08:46:25.160 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-02-10T08:46:25.411 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2024-02-10T08:46:25.628 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-02-10T08:46:25.659 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2024-02-10T08:46:25.848 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-02-10T08:46:25.884 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2024-02-10T08:46:25.945 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-02-10T08:46:26.127 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2024-02-10T08:46:26.135 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2024-02-10T08:46:26.341 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2024-02-10T08:46:26.498 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-02-10T08:46:26.565 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-02-10T08:46:26.798 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2024-02-10T08:46:26.998 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2024-02-10T08:46:27.214 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2024-02-10T08:46:27.431 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2024-02-10T08:46:28.749 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-10T08:46:29.018 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-10T08:46:29.710 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-10T08:46:29.979 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-10T08:46:30.059 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-02-10T08:46:30.172 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-02-10T08:46:30.198 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-10T08:46:30.343 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-02-10T08:46:30.898 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-10T08:46:31.206 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-02-10T08:46:31.314 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-02-10T08:46:31.455 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-02-10T08:46:49.800 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-02-10T08:46:49.800 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /sys 2024-02-10T08:46:49.800 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /proc 2024-02-10T08:46:49.800 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /dev 2024-02-10T08:46:49.800 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /run 2024-02-10T08:46:49.800 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /mnt 2024-02-10T08:46:49.800 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /var/tmp 2024-02-10T08:46:49.800 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /home 2024-02-10T08:46:49.800 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /tmp 2024-02-10T08:46:49.800 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /dev 2024-02-10T08:46:49.801 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:46:50.427 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-02-10T08:46:50.427 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /sys 2024-02-10T08:46:50.427 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /proc 2024-02-10T08:46:50.427 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /dev 2024-02-10T08:46:50.427 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /run 2024-02-10T08:46:50.428 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /mnt 2024-02-10T08:46:50.428 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /var/tmp 2024-02-10T08:46:50.428 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /home 2024-02-10T08:46:50.428 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /tmp 2024-02-10T08:46:50.428 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /dev 2024-02-10T08:46:50.428 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:46:51.156 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-02-10T08:46:51.264 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-02-10T08:46:51.264 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-02-10T08:46:51.467 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-02-10T08:46:51.467 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2024-02-10T08:46:51.753 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-02-10T08:46:51.829 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-02-10T08:46:51.829 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-02-10T08:46:52.072 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-02-10T08:46:52.072 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2024-02-10T08:46:52.158 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2024-02-10T08:46:52.158 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2024-02-10T08:46:52.158 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2024-02-10T08:46:52.159 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2024-02-10T08:46:52.159 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2024-02-10T08:46:52.159 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2024-02-10T08:46:52.159 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2024-02-10T08:46:52.159 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2024-02-10T08:46:52.159 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2024-02-10T08:46:52.159 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2024-02-10T08:46:52.159 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2024-02-10T08:46:52.159 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2024-02-10T08:46:52.159 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2024-02-10T08:46:52.159 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2024-02-10T08:46:52.159 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2024-02-10T08:46:52.159 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 15/29 2024-02-10T08:46:52.160 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2024-02-10T08:46:52.160 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2024-02-10T08:46:52.160 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2024-02-10T08:46:52.160 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 19/29 2024-02-10T08:46:52.160 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2024-02-10T08:46:52.160 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2024-02-10T08:46:52.160 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2024-02-10T08:46:52.160 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2024-02-10T08:46:52.160 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2024-02-10T08:46:52.160 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2024-02-10T08:46:52.160 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2024-02-10T08:46:52.160 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2024-02-10T08:46:52.160 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2024-02-10T08:46:52.780 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2024-02-10T08:46:52.780 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2024-02-10T08:46:52.780 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2024-02-10T08:46:52.780 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2024-02-10T08:46:52.780 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2024-02-10T08:46:52.780 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2024-02-10T08:46:52.781 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2024-02-10T08:46:52.781 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2024-02-10T08:46:52.781 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2024-02-10T08:46:52.781 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2024-02-10T08:46:52.781 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2024-02-10T08:46:52.781 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2024-02-10T08:46:52.781 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2024-02-10T08:46:52.781 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2024-02-10T08:46:52.781 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2024-02-10T08:46:52.781 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 15/29 2024-02-10T08:46:52.781 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2024-02-10T08:46:52.781 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2024-02-10T08:46:52.781 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2024-02-10T08:46:52.782 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 19/29 2024-02-10T08:46:52.782 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2024-02-10T08:46:52.782 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2024-02-10T08:46:52.782 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2024-02-10T08:46:52.782 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2024-02-10T08:46:52.782 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2024-02-10T08:46:52.782 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2024-02-10T08:46:52.782 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2024-02-10T08:46:52.782 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2024-02-10T08:46:52.782 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2024-02-10T08:46:53.156 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2024-02-10T08:46:53.156 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:46:53.156 INFO:teuthology.orchestra.run.smithi155.stdout:Upgraded: 2024-02-10T08:46:53.156 INFO:teuthology.orchestra.run.smithi155.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2024-02-10T08:46:53.156 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-02-10T08:46:53.156 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2024-02-10T08:46:53.156 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2024-02-10T08:46:53.157 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2024-02-10T08:46:53.157 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2024-02-10T08:46:53.157 INFO:teuthology.orchestra.run.smithi155.stdout: daxctl-libs-71.1-7.el8.x86_64 2024-02-10T08:46:53.157 INFO:teuthology.orchestra.run.smithi155.stdout: fmt-6.2.1-1.el8.x86_64 2024-02-10T08:46:53.157 INFO:teuthology.orchestra.run.smithi155.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-02-10T08:46:53.157 INFO:teuthology.orchestra.run.smithi155.stdout: leveldb-1.22-1.el8.x86_64 2024-02-10T08:46:53.157 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2024-02-10T08:46:53.157 INFO:teuthology.orchestra.run.smithi155.stdout: liboath-2.6.2-3.el8.x86_64 2024-02-10T08:46:53.157 INFO:teuthology.orchestra.run.smithi155.stdout: libpmemobj-1.6.1-1.el8.x86_64 2024-02-10T08:46:53.157 INFO:teuthology.orchestra.run.smithi155.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-02-10T08:46:53.158 INFO:teuthology.orchestra.run.smithi155.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2024-02-10T08:46:53.158 INFO:teuthology.orchestra.run.smithi155.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-02-10T08:46:53.158 INFO:teuthology.orchestra.run.smithi155.stdout: librgw2-2:16.2.4-0.el8.x86_64 2024-02-10T08:46:53.158 INFO:teuthology.orchestra.run.smithi155.stdout: libunwind-1.3.1-3.el8.x86_64 2024-02-10T08:46:53.158 INFO:teuthology.orchestra.run.smithi155.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-02-10T08:46:53.158 INFO:teuthology.orchestra.run.smithi155.stdout: ndctl-libs-71.1-7.el8.x86_64 2024-02-10T08:46:53.158 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2024-02-10T08:46:53.158 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2024-02-10T08:46:53.158 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2024-02-10T08:46:53.158 INFO:teuthology.orchestra.run.smithi155.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-02-10T08:46:53.158 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2024-02-10T08:46:53.158 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2024-02-10T08:46:53.158 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2024-02-10T08:46:53.159 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:46:53.159 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-02-10T08:46:53.454 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-test 2024-02-10T08:46:53.684 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2024-02-10T08:46:53.685 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:46:53.685 INFO:teuthology.orchestra.run.smithi077.stdout:Upgraded: 2024-02-10T08:46:53.685 INFO:teuthology.orchestra.run.smithi077.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2024-02-10T08:46:53.685 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2024-02-10T08:46:53.685 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2024-02-10T08:46:53.685 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2024-02-10T08:46:53.685 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2024-02-10T08:46:53.685 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2024-02-10T08:46:53.685 INFO:teuthology.orchestra.run.smithi077.stdout: daxctl-libs-71.1-7.el8.x86_64 2024-02-10T08:46:53.685 INFO:teuthology.orchestra.run.smithi077.stdout: fmt-6.2.1-1.el8.x86_64 2024-02-10T08:46:53.685 INFO:teuthology.orchestra.run.smithi077.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-02-10T08:46:53.686 INFO:teuthology.orchestra.run.smithi077.stdout: leveldb-1.22-1.el8.x86_64 2024-02-10T08:46:53.686 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2024-02-10T08:46:53.686 INFO:teuthology.orchestra.run.smithi077.stdout: liboath-2.6.2-3.el8.x86_64 2024-02-10T08:46:53.686 INFO:teuthology.orchestra.run.smithi077.stdout: libpmemobj-1.6.1-1.el8.x86_64 2024-02-10T08:46:53.686 INFO:teuthology.orchestra.run.smithi077.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-02-10T08:46:53.686 INFO:teuthology.orchestra.run.smithi077.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2024-02-10T08:46:53.686 INFO:teuthology.orchestra.run.smithi077.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-02-10T08:46:53.686 INFO:teuthology.orchestra.run.smithi077.stdout: librgw2-2:16.2.4-0.el8.x86_64 2024-02-10T08:46:53.686 INFO:teuthology.orchestra.run.smithi077.stdout: libunwind-1.3.1-3.el8.x86_64 2024-02-10T08:46:53.686 INFO:teuthology.orchestra.run.smithi077.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-02-10T08:46:53.686 INFO:teuthology.orchestra.run.smithi077.stdout: ndctl-libs-71.1-7.el8.x86_64 2024-02-10T08:46:53.686 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2024-02-10T08:46:53.686 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2024-02-10T08:46:53.686 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2024-02-10T08:46:53.687 INFO:teuthology.orchestra.run.smithi077.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-02-10T08:46:53.687 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2024-02-10T08:46:53.687 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2024-02-10T08:46:53.687 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2024-02-10T08:46:53.687 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:46:53.687 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-02-10T08:46:53.990 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph-test 2024-02-10T08:46:54.217 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:00:43 ago on Sat 10 Feb 2024 08:46:11 AM UTC. 2024-02-10T08:46:54.748 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:00:42 ago on Sat 10 Feb 2024 08:46:12 AM UTC. 2024-02-10T08:46:55.733 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-02-10T08:46:55.734 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-02-10T08:46:55.734 INFO:teuthology.orchestra.run.smithi155.stdout: Package Architecture Version Repository Size 2024-02-10T08:46:55.734 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-02-10T08:46:55.734 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-02-10T08:46:55.734 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2024-02-10T08:46:55.734 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-02-10T08:46:55.734 INFO:teuthology.orchestra.run.smithi155.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-02-10T08:46:55.734 INFO:teuthology.orchestra.run.smithi155.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2024-02-10T08:46:55.735 INFO:teuthology.orchestra.run.smithi155.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-02-10T08:46:55.735 INFO:teuthology.orchestra.run.smithi155.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-02-10T08:46:55.735 INFO:teuthology.orchestra.run.smithi155.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-02-10T08:46:55.735 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:46:55.735 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-02-10T08:46:55.735 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-02-10T08:46:55.735 INFO:teuthology.orchestra.run.smithi155.stdout:Install 6 Packages 2024-02-10T08:46:55.735 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:46:55.736 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 53 M 2024-02-10T08:46:55.736 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 229 M 2024-02-10T08:46:55.736 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-02-10T08:46:56.014 INFO:teuthology.orchestra.run.smithi155.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 2.0 MB/s | 169 kB 00:00 2024-02-10T08:46:56.131 INFO:teuthology.orchestra.run.smithi155.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 1.0 MB/s | 203 kB 00:00 2024-02-10T08:46:56.223 INFO:teuthology.orchestra.run.smithi155.stdout:(3/6): oniguruma-6.8.2-3.el8.x86_64.rpm 902 kB/s | 188 kB 00:00 2024-02-10T08:46:56.253 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-02-10T08:46:56.254 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-10T08:46:56.254 INFO:teuthology.orchestra.run.smithi077.stdout: Package Architecture Version Repository Size 2024-02-10T08:46:56.254 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-10T08:46:56.254 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2024-02-10T08:46:56.254 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2024-02-10T08:46:56.254 INFO:teuthology.orchestra.run.smithi077.stdout:Installing dependencies: 2024-02-10T08:46:56.254 INFO:teuthology.orchestra.run.smithi077.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-02-10T08:46:56.254 INFO:teuthology.orchestra.run.smithi077.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2024-02-10T08:46:56.254 INFO:teuthology.orchestra.run.smithi077.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-02-10T08:46:56.254 INFO:teuthology.orchestra.run.smithi077.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-02-10T08:46:56.255 INFO:teuthology.orchestra.run.smithi077.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-02-10T08:46:56.255 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:46:56.255 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2024-02-10T08:46:56.255 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-10T08:46:56.255 INFO:teuthology.orchestra.run.smithi077.stdout:Install 6 Packages 2024-02-10T08:46:56.255 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:46:56.255 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 53 M 2024-02-10T08:46:56.256 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 229 M 2024-02-10T08:46:56.256 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2024-02-10T08:46:56.379 INFO:teuthology.orchestra.run.smithi077.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 2.0 MB/s | 169 kB 00:00 2024-02-10T08:46:56.546 INFO:teuthology.orchestra.run.smithi077.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 814 kB/s | 203 kB 00:00 2024-02-10T08:46:56.580 INFO:teuthology.orchestra.run.smithi077.stdout:(3/6): oniguruma-6.8.2-3.el8.x86_64.rpm 941 kB/s | 188 kB 00:00 2024-02-10T08:46:56.646 INFO:teuthology.orchestra.run.smithi077.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.0 MB/s | 71 kB 00:00 2024-02-10T08:46:56.672 INFO:teuthology.orchestra.run.smithi077.stdout:(5/6): socat-1.7.4.1-1.el8.x86_64.rpm 2.5 MB/s | 323 kB 00:00 2024-02-10T08:46:56.715 INFO:teuthology.orchestra.run.smithi155.stdout:(4/6): ceph-test-16.2.4-0.el8.x86_64.rpm 66 MB/s | 52 MB 00:00 2024-02-10T08:46:56.740 INFO:teuthology.orchestra.run.smithi155.stdout:(5/6): socat-1.7.4.1-1.el8.x86_64.rpm 530 kB/s | 323 kB 00:00 2024-02-10T08:46:56.774 INFO:teuthology.orchestra.run.smithi155.stdout:(6/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 128 kB/s | 71 kB 00:00 2024-02-10T08:46:56.776 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:46:56.776 INFO:teuthology.orchestra.run.smithi155.stdout:Total 51 MB/s | 53 MB 00:01 2024-02-10T08:46:56.818 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-02-10T08:46:56.839 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-02-10T08:46:56.840 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-02-10T08:46:57.154 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-02-10T08:46:57.155 INFO:teuthology.orchestra.run.smithi077.stdout:(6/6): ceph-test-16.2.4-0.el8.x86_64.rpm 60 MB/s | 52 MB 00:00 2024-02-10T08:46:57.157 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:46:57.158 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-02-10T08:46:57.158 INFO:teuthology.orchestra.run.smithi077.stdout:Total 59 MB/s | 53 MB 00:00 2024-02-10T08:46:57.200 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2024-02-10T08:46:57.221 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2024-02-10T08:46:57.222 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2024-02-10T08:46:57.540 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2024-02-10T08:46:57.543 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2024-02-10T08:46:57.660 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-02-10T08:46:57.908 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-02-10T08:46:58.019 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2024-02-10T08:46:58.132 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-02-10T08:46:58.272 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-10T08:46:58.272 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-02-10T08:46:58.379 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-10T08:46:58.458 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-02-10T08:46:58.554 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-02-10T08:46:58.652 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-10T08:46:58.730 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-02-10T08:46:58.734 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-10T08:46:58.944 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-02-10T08:46:59.094 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-02-10T08:47:03.611 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-02-10T08:47:03.878 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-02-10T08:47:04.010 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-02-10T08:47:04.186 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-02-10T08:47:04.187 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2024-02-10T08:47:04.187 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2024-02-10T08:47:04.187 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-02-10T08:47:04.187 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-02-10T08:47:04.187 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-02-10T08:47:04.233 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-02-10T08:47:04.558 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-02-10T08:47:04.558 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2024-02-10T08:47:04.559 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2024-02-10T08:47:04.559 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-02-10T08:47:04.559 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-02-10T08:47:04.559 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-02-10T08:47:04.616 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-02-10T08:47:04.616 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:47:04.616 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-02-10T08:47:04.616 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-02-10T08:47:04.616 INFO:teuthology.orchestra.run.smithi155.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-3.el8.x86_64 2024-02-10T08:47:04.616 INFO:teuthology.orchestra.run.smithi155.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2024-02-10T08:47:04.616 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:47:04.616 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-02-10T08:47:04.828 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph 2024-02-10T08:47:05.003 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-02-10T08:47:05.003 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:47:05.004 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2024-02-10T08:47:05.004 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-02-10T08:47:05.004 INFO:teuthology.orchestra.run.smithi077.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-3.el8.x86_64 2024-02-10T08:47:05.004 INFO:teuthology.orchestra.run.smithi077.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2024-02-10T08:47:05.004 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:47:05.004 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-02-10T08:47:05.213 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph 2024-02-10T08:47:05.583 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:00:54 ago on Sat 10 Feb 2024 08:46:11 AM UTC. 2024-02-10T08:47:05.968 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:00:53 ago on Sat 10 Feb 2024 08:46:12 AM UTC. 2024-02-10T08:47:07.115 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-02-10T08:47:07.118 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-02-10T08:47:07.118 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-02-10T08:47:07.118 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-02-10T08:47:07.118 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-02-10T08:47:07.118 INFO:teuthology.orchestra.run.smithi155.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2024-02-10T08:47:07.118 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-02-10T08:47:07.118 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2024-02-10T08:47:07.119 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2024-02-10T08:47:07.119 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2024-02-10T08:47:07.119 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2024-02-10T08:47:07.119 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2024-02-10T08:47:07.119 INFO:teuthology.orchestra.run.smithi155.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-02-10T08:47:07.119 INFO:teuthology.orchestra.run.smithi155.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-02-10T08:47:07.119 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-02-10T08:47:07.119 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-02-10T08:47:07.119 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-02-10T08:47:07.119 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-02-10T08:47:07.119 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-02-10T08:47:07.119 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-02-10T08:47:07.120 INFO:teuthology.orchestra.run.smithi155.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-02-10T08:47:07.120 INFO:teuthology.orchestra.run.smithi155.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-02-10T08:47:07.120 INFO:teuthology.orchestra.run.smithi155.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-02-10T08:47:07.120 INFO:teuthology.orchestra.run.smithi155.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-02-10T08:47:07.120 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-02-10T08:47:07.120 INFO:teuthology.orchestra.run.smithi155.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-02-10T08:47:07.120 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-02-10T08:47:07.120 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-02-10T08:47:07.120 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-02-10T08:47:07.120 INFO:teuthology.orchestra.run.smithi155.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-02-10T08:47:07.120 INFO:teuthology.orchestra.run.smithi155.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-02-10T08:47:07.120 INFO:teuthology.orchestra.run.smithi155.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-02-10T08:47:07.121 INFO:teuthology.orchestra.run.smithi155.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-02-10T08:47:07.121 INFO:teuthology.orchestra.run.smithi155.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-02-10T08:47:07.121 INFO:teuthology.orchestra.run.smithi155.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-02-10T08:47:07.121 INFO:teuthology.orchestra.run.smithi155.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-02-10T08:47:07.121 INFO:teuthology.orchestra.run.smithi155.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-02-10T08:47:07.121 INFO:teuthology.orchestra.run.smithi155.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-02-10T08:47:07.121 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:47:07.121 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-02-10T08:47:07.121 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-02-10T08:47:07.121 INFO:teuthology.orchestra.run.smithi155.stdout:Install 32 Packages 2024-02-10T08:47:07.121 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:47:07.122 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 31 M 2024-02-10T08:47:07.122 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 111 M 2024-02-10T08:47:07.122 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-02-10T08:47:07.494 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-02-10T08:47:07.497 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-10T08:47:07.497 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repository Size 2024-02-10T08:47:07.497 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-10T08:47:07.498 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2024-02-10T08:47:07.498 INFO:teuthology.orchestra.run.smithi077.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2024-02-10T08:47:07.498 INFO:teuthology.orchestra.run.smithi077.stdout:Installing dependencies: 2024-02-10T08:47:07.498 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2024-02-10T08:47:07.498 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2024-02-10T08:47:07.498 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2024-02-10T08:47:07.498 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2024-02-10T08:47:07.498 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2024-02-10T08:47:07.498 INFO:teuthology.orchestra.run.smithi077.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-02-10T08:47:07.498 INFO:teuthology.orchestra.run.smithi077.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-02-10T08:47:07.498 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-02-10T08:47:07.498 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-02-10T08:47:07.498 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-02-10T08:47:07.498 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-02-10T08:47:07.499 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-02-10T08:47:07.499 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-02-10T08:47:07.499 INFO:teuthology.orchestra.run.smithi077.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-02-10T08:47:07.499 INFO:teuthology.orchestra.run.smithi077.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-02-10T08:47:07.499 INFO:teuthology.orchestra.run.smithi077.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-02-10T08:47:07.499 INFO:teuthology.orchestra.run.smithi077.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-02-10T08:47:07.499 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-02-10T08:47:07.499 INFO:teuthology.orchestra.run.smithi077.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-02-10T08:47:07.499 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-02-10T08:47:07.499 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-02-10T08:47:07.499 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-02-10T08:47:07.499 INFO:teuthology.orchestra.run.smithi077.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-02-10T08:47:07.499 INFO:teuthology.orchestra.run.smithi077.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-02-10T08:47:07.499 INFO:teuthology.orchestra.run.smithi077.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-02-10T08:47:07.499 INFO:teuthology.orchestra.run.smithi077.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-02-10T08:47:07.500 INFO:teuthology.orchestra.run.smithi077.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-02-10T08:47:07.500 INFO:teuthology.orchestra.run.smithi077.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-02-10T08:47:07.500 INFO:teuthology.orchestra.run.smithi077.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-02-10T08:47:07.500 INFO:teuthology.orchestra.run.smithi077.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-02-10T08:47:07.500 INFO:teuthology.orchestra.run.smithi077.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-02-10T08:47:07.500 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:47:07.500 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2024-02-10T08:47:07.500 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-10T08:47:07.500 INFO:teuthology.orchestra.run.smithi077.stdout:Install 32 Packages 2024-02-10T08:47:07.500 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:47:07.501 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 31 M 2024-02-10T08:47:07.501 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 111 M 2024-02-10T08:47:07.501 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2024-02-10T08:47:07.678 INFO:teuthology.orchestra.run.smithi155.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 126 kB/s | 6.3 kB 00:00 2024-02-10T08:47:07.728 INFO:teuthology.orchestra.run.smithi155.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-02-10T08:47:07.765 INFO:teuthology.orchestra.run.smithi077.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 119 kB/s | 6.3 kB 00:00 2024-02-10T08:47:07.780 INFO:teuthology.orchestra.run.smithi155.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 16 MB/s | 2.4 MB 00:00 2024-02-10T08:47:07.804 INFO:teuthology.orchestra.run.smithi155.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 8.6 MB/s | 220 kB 00:00 2024-02-10T08:47:07.816 INFO:teuthology.orchestra.run.smithi077.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-02-10T08:47:07.866 INFO:teuthology.orchestra.run.smithi077.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 16 MB/s | 2.4 MB 00:00 2024-02-10T08:47:07.900 INFO:teuthology.orchestra.run.smithi077.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 6.4 MB/s | 220 kB 00:00 2024-02-10T08:47:07.913 INFO:teuthology.orchestra.run.smithi155.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 21 MB/s | 5.0 MB 00:00 2024-02-10T08:47:07.975 INFO:teuthology.orchestra.run.smithi077.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 24 MB/s | 5.0 MB 00:00 2024-02-10T08:47:08.155 INFO:teuthology.orchestra.run.smithi155.stdout:(6/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 43 MB/s | 18 MB 00:00 2024-02-10T08:47:08.205 INFO:teuthology.orchestra.run.smithi155.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 133 kB/s | 39 kB 00:00 2024-02-10T08:47:08.217 INFO:teuthology.orchestra.run.smithi077.stdout:(6/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 45 MB/s | 18 MB 00:00 2024-02-10T08:47:08.230 INFO:teuthology.orchestra.run.smithi155.stdout:(8/32): python3-mako-1.0.6-14.el8.noarch.rpm 369 kB/s | 157 kB 00:00 2024-02-10T08:47:08.242 INFO:teuthology.orchestra.run.smithi077.stdout:(7/32): python3-mako-1.0.6-14.el8.noarch.rpm 459 kB/s | 157 kB 00:00 2024-02-10T08:47:08.297 INFO:teuthology.orchestra.run.smithi155.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 619 kB/s | 57 kB 00:00 2024-02-10T08:47:08.309 INFO:teuthology.orchestra.run.smithi077.stdout:(8/32): python3-pytz-2017.2-11.el8.noarch.rpm 853 kB/s | 57 kB 00:00 2024-02-10T08:47:08.335 INFO:teuthology.orchestra.run.smithi077.stdout:(9/32): python3-markupsafe-0.23-19.el8.x86_64.r 108 kB/s | 39 kB 00:00 2024-02-10T08:47:08.381 INFO:teuthology.orchestra.run.smithi155.stdout:(10/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 457 kB/s | 103 kB 00:00 2024-02-10T08:47:08.407 INFO:teuthology.orchestra.run.smithi155.stdout:(11/32): python3-werkzeug-0.12.2-4.el8.noarch.r 2.5 MB/s | 457 kB 00:00 2024-02-10T08:47:08.440 INFO:teuthology.orchestra.run.smithi077.stdout:(10/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 465 kB/s | 103 kB 00:00 2024-02-10T08:47:08.490 INFO:teuthology.orchestra.run.smithi155.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.3 MB/s | 249 kB 00:00 2024-02-10T08:47:08.493 INFO:teuthology.orchestra.run.smithi077.stdout:(11/32): python3-werkzeug-0.12.2-4.el8.noarch.r 2.4 MB/s | 457 kB 00:00 2024-02-10T08:47:08.515 INFO:teuthology.orchestra.run.smithi155.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 1.0 MB/s | 109 kB 00:00 2024-02-10T08:47:08.519 INFO:teuthology.orchestra.run.smithi077.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.3 MB/s | 249 kB 00:00 2024-02-10T08:47:08.594 INFO:teuthology.orchestra.run.smithi077.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 1.1 MB/s | 109 kB 00:00 2024-02-10T08:47:08.620 INFO:teuthology.orchestra.run.smithi077.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 434 kB/s | 44 kB 00:00 2024-02-10T08:47:08.666 INFO:teuthology.orchestra.run.smithi155.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 249 kB/s | 44 kB 00:00 2024-02-10T08:47:08.703 INFO:teuthology.orchestra.run.smithi077.stdout:(15/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 2.0 MB/s | 173 kB 00:00 2024-02-10T08:47:08.737 INFO:teuthology.orchestra.run.smithi077.stdout:(16/32): python3-cryptography-3.2.1-7.el8.x86_6 2.1 MB/s | 641 kB 00:00 2024-02-10T08:47:08.739 INFO:teuthology.orchestra.run.smithi155.stdout:(15/32): python3-cryptography-3.2.1-7.el8.x86_6 1.7 MB/s | 641 kB 00:00 2024-02-10T08:47:08.775 INFO:teuthology.orchestra.run.smithi155.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 715 kB/s | 185 kB 00:00 2024-02-10T08:47:08.785 INFO:teuthology.orchestra.run.smithi077.stdout:(17/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 970 kB/s | 185 kB 00:00 2024-02-10T08:47:08.800 INFO:teuthology.orchestra.run.smithi155.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.3 MB/s | 173 kB 00:00 2024-02-10T08:47:08.820 INFO:teuthology.orchestra.run.smithi077.stdout:(18/32): python3-cherrypy-18.4.0-1.el8.noarch.r 3.2 MB/s | 384 kB 00:00 2024-02-10T08:47:08.846 INFO:teuthology.orchestra.run.smithi077.stdout:(19/32): python3-jaraco-6.2-6.el8.noarch.rpm 102 kB/s | 11 kB 00:00 2024-02-10T08:47:08.850 INFO:teuthology.orchestra.run.smithi155.stdout:(18/32): python3-cherrypy-18.4.0-1.el8.noarch.r 3.4 MB/s | 384 kB 00:00 2024-02-10T08:47:08.871 INFO:teuthology.orchestra.run.smithi077.stdout:(20/32): python3-jaraco-functools-2.0-4.el8.noa 206 kB/s | 18 kB 00:00 2024-02-10T08:47:08.875 INFO:teuthology.orchestra.run.smithi155.stdout:(19/32): python3-jaraco-6.2-6.el8.noarch.rpm 110 kB/s | 11 kB 00:00 2024-02-10T08:47:08.896 INFO:teuthology.orchestra.run.smithi077.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 646 kB/s | 49 kB 00:00 2024-02-10T08:47:08.901 INFO:teuthology.orchestra.run.smithi155.stdout:(20/32): python3-jaraco-functools-2.0-4.el8.noa 174 kB/s | 18 kB 00:00 2024-02-10T08:47:08.922 INFO:teuthology.orchestra.run.smithi077.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 778 kB/s | 59 kB 00:00 2024-02-10T08:47:08.926 INFO:teuthology.orchestra.run.smithi155.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 645 kB/s | 49 kB 00:00 2024-02-10T08:47:08.947 INFO:teuthology.orchestra.run.smithi077.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 310 kB/s | 16 kB 00:00 2024-02-10T08:47:08.951 INFO:teuthology.orchestra.run.smithi155.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 776 kB/s | 59 kB 00:00 2024-02-10T08:47:08.977 INFO:teuthology.orchestra.run.smithi155.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 311 kB/s | 16 kB 00:00 2024-02-10T08:47:08.981 INFO:teuthology.orchestra.run.smithi077.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 318 kB/s | 19 kB 00:00 2024-02-10T08:47:09.006 INFO:teuthology.orchestra.run.smithi077.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 408 kB/s | 24 kB 00:00 2024-02-10T08:47:09.031 INFO:teuthology.orchestra.run.smithi077.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 584 kB/s | 29 kB 00:00 2024-02-10T08:47:09.035 INFO:teuthology.orchestra.run.smithi155.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 224 kB/s | 19 kB 00:00 2024-02-10T08:47:09.057 INFO:teuthology.orchestra.run.smithi077.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 547 kB/s | 27 kB 00:00 2024-02-10T08:47:09.060 INFO:teuthology.orchestra.run.smithi155.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 286 kB/s | 24 kB 00:00 2024-02-10T08:47:09.082 INFO:teuthology.orchestra.run.smithi077.stdout:(28/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.3 MB/s | 283 kB 00:00 2024-02-10T08:47:09.094 INFO:teuthology.orchestra.run.smithi155.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 500 kB/s | 29 kB 00:00 2024-02-10T08:47:09.120 INFO:teuthology.orchestra.run.smithi155.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 469 kB/s | 27 kB 00:00 2024-02-10T08:47:09.130 INFO:teuthology.orchestra.run.smithi077.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 2.5 MB/s | 255 kB 00:00 2024-02-10T08:47:09.145 INFO:teuthology.orchestra.run.smithi155.stdout:(28/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.1 MB/s | 283 kB 00:00 2024-02-10T08:47:09.166 INFO:teuthology.orchestra.run.smithi077.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.0 MB/s | 85 kB 00:00 2024-02-10T08:47:09.182 INFO:teuthology.orchestra.run.smithi077.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 440 kB/s | 23 kB 00:00 2024-02-10T08:47:09.187 INFO:teuthology.orchestra.run.smithi155.stdout:(29/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 3.7 MB/s | 251 kB 00:00 2024-02-10T08:47:09.208 INFO:teuthology.orchestra.run.smithi077.stdout:(32/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.6 MB/s | 251 kB 00:00 2024-02-10T08:47:09.209 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:47:09.209 INFO:teuthology.orchestra.run.smithi077.stdout:Total 18 MB/s | 31 MB 00:01 2024-02-10T08:47:09.212 INFO:teuthology.orchestra.run.smithi155.stdout:(30/32): python3-waitress-1.4.3-1.el8.noarch.rp 2.1 MB/s | 255 kB 00:00 2024-02-10T08:47:09.229 INFO:teuthology.orchestra.run.smithi155.stdout:(31/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.0 MB/s | 85 kB 00:00 2024-02-10T08:47:09.254 INFO:teuthology.orchestra.run.smithi155.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 342 kB/s | 23 kB 00:00 2024-02-10T08:47:09.255 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:47:09.256 INFO:teuthology.orchestra.run.smithi155.stdout:Total 15 MB/s | 31 MB 00:02 2024-02-10T08:47:09.292 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2024-02-10T08:47:09.330 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2024-02-10T08:47:09.331 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2024-02-10T08:47:09.338 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-02-10T08:47:09.376 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-02-10T08:47:09.377 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-02-10T08:47:09.640 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2024-02-10T08:47:09.643 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2024-02-10T08:47:09.702 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-02-10T08:47:09.705 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-02-10T08:47:10.806 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2024-02-10T08:47:10.874 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-02-10T08:47:11.039 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-02-10T08:47:11.117 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-02-10T08:47:11.258 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-02-10T08:47:11.336 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-02-10T08:47:11.517 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-02-10T08:47:11.594 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-02-10T08:47:11.724 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-02-10T08:47:11.826 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-02-10T08:47:11.930 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-02-10T08:47:11.999 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-02-10T08:47:12.110 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-02-10T08:47:12.203 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-02-10T08:47:12.303 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-02-10T08:47:12.406 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-02-10T08:47:12.456 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-02-10T08:47:12.592 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-02-10T08:47:12.688 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-02-10T08:47:12.826 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-02-10T08:47:12.927 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-02-10T08:47:13.046 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-02-10T08:47:13.154 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-02-10T08:47:13.292 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-02-10T08:47:13.428 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-02-10T08:47:13.575 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-02-10T08:47:13.816 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-02-10T08:47:13.963 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-02-10T08:47:14.120 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-02-10T08:47:14.230 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-02-10T08:47:14.283 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-02-10T08:47:14.427 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-02-10T08:47:14.533 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-02-10T08:47:14.712 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-02-10T08:47:14.768 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-10T08:47:14.904 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-10T08:47:15.042 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-02-10T08:47:15.188 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-02-10T08:47:15.264 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-02-10T08:47:15.408 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-02-10T08:47:15.494 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-02-10T08:47:15.621 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-02-10T08:47:15.753 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-02-10T08:47:15.860 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-02-10T08:47:16.016 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-02-10T08:47:16.120 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-02-10T08:47:16.279 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-02-10T08:47:16.377 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-02-10T08:47:16.554 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-02-10T08:47:16.632 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-02-10T08:47:16.951 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-02-10T08:47:16.980 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-02-10T08:47:17.273 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-02-10T08:47:17.291 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-02-10T08:47:17.696 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2024-02-10T08:47:17.715 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2024-02-10T08:47:17.950 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-02-10T08:47:17.952 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-02-10T08:47:19.480 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-02-10T08:47:19.485 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-02-10T08:47:19.708 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-02-10T08:47:19.710 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-02-10T08:47:20.141 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-02-10T08:47:20.144 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-02-10T08:47:20.377 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-02-10T08:47:20.399 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-02-10T08:47:20.601 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-02-10T08:47:20.626 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-02-10T08:47:20.808 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-02-10T08:47:20.827 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-02-10T08:47:20.835 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-02-10T08:47:20.854 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-02-10T08:47:20.974 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2024-02-10T08:47:21.026 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2024-02-10T08:47:21.388 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2024-02-10T08:47:21.389 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2024-02-10T08:47:21.389 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2024-02-10T08:47:21.389 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2024-02-10T08:47:21.389 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2024-02-10T08:47:21.389 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2024-02-10T08:47:21.389 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2024-02-10T08:47:21.389 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-02-10T08:47:21.389 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-02-10T08:47:21.389 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-02-10T08:47:21.389 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-02-10T08:47:21.389 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-02-10T08:47:21.389 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-02-10T08:47:21.389 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-02-10T08:47:21.390 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-02-10T08:47:21.390 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-02-10T08:47:21.390 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-02-10T08:47:21.390 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-10T08:47:21.390 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-02-10T08:47:21.390 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-02-10T08:47:21.390 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-02-10T08:47:21.390 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-02-10T08:47:21.390 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-02-10T08:47:21.390 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-02-10T08:47:21.390 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-02-10T08:47:21.390 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-02-10T08:47:21.390 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-02-10T08:47:21.390 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-02-10T08:47:21.391 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-02-10T08:47:21.391 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-02-10T08:47:21.391 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-02-10T08:47:21.391 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-02-10T08:47:21.433 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2024-02-10T08:47:21.433 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2024-02-10T08:47:21.433 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2024-02-10T08:47:21.433 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2024-02-10T08:47:21.434 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2024-02-10T08:47:21.434 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2024-02-10T08:47:21.434 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2024-02-10T08:47:21.434 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-02-10T08:47:21.434 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-02-10T08:47:21.434 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-02-10T08:47:21.434 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-02-10T08:47:21.434 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-02-10T08:47:21.434 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-02-10T08:47:21.435 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-02-10T08:47:21.435 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-02-10T08:47:21.435 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-02-10T08:47:21.435 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-02-10T08:47:21.435 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-10T08:47:21.435 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-02-10T08:47:21.435 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-02-10T08:47:21.435 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-02-10T08:47:21.435 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-02-10T08:47:21.435 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-02-10T08:47:21.435 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-02-10T08:47:21.435 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-02-10T08:47:21.435 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-02-10T08:47:21.437 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-02-10T08:47:21.437 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-02-10T08:47:21.437 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-02-10T08:47:21.438 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-02-10T08:47:21.438 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-02-10T08:47:21.438 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-02-10T08:47:22.342 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-02-10T08:47:22.342 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:47:22.342 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-02-10T08:47:22.342 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-2:16.2.4-0.el8.x86_64 2024-02-10T08:47:22.342 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2024-02-10T08:47:22.343 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2024-02-10T08:47:22.343 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2024-02-10T08:47:22.343 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2024-02-10T08:47:22.343 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2024-02-10T08:47:22.343 INFO:teuthology.orchestra.run.smithi155.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-02-10T08:47:22.343 INFO:teuthology.orchestra.run.smithi155.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-02-10T08:47:22.343 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-02-10T08:47:22.343 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-02-10T08:47:22.343 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-02-10T08:47:22.343 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-02-10T08:47:22.343 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-6.2-6.el8.noarch 2024-02-10T08:47:22.343 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-02-10T08:47:22.343 INFO:teuthology.orchestra.run.smithi155.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-02-10T08:47:22.343 INFO:teuthology.orchestra.run.smithi155.stdout: python3-mako-1.0.6-14.el8.noarch 2024-02-10T08:47:22.344 INFO:teuthology.orchestra.run.smithi155.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-02-10T08:47:22.344 INFO:teuthology.orchestra.run.smithi155.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-02-10T08:47:22.344 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-02-10T08:47:22.344 INFO:teuthology.orchestra.run.smithi155.stdout: python3-portend-2.6-1.el8.noarch 2024-02-10T08:47:22.344 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-02-10T08:47:22.344 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pycparser-2.14-14.el8.noarch 2024-02-10T08:47:22.344 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pytz-2017.2-11.el8.noarch 2024-02-10T08:47:22.344 INFO:teuthology.orchestra.run.smithi155.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-02-10T08:47:22.344 INFO:teuthology.orchestra.run.smithi155.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-02-10T08:47:22.344 INFO:teuthology.orchestra.run.smithi155.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-02-10T08:47:22.344 INFO:teuthology.orchestra.run.smithi155.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-02-10T08:47:22.344 INFO:teuthology.orchestra.run.smithi155.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-02-10T08:47:22.344 INFO:teuthology.orchestra.run.smithi155.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-02-10T08:47:22.345 INFO:teuthology.orchestra.run.smithi155.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-02-10T08:47:22.345 INFO:teuthology.orchestra.run.smithi155.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-02-10T08:47:22.345 INFO:teuthology.orchestra.run.smithi155.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-02-10T08:47:22.345 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:47:22.345 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-02-10T08:47:22.373 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-02-10T08:47:22.374 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:47:22.374 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2024-02-10T08:47:22.374 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-2:16.2.4-0.el8.x86_64 2024-02-10T08:47:22.374 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2024-02-10T08:47:22.374 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2024-02-10T08:47:22.374 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2024-02-10T08:47:22.374 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2024-02-10T08:47:22.374 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2024-02-10T08:47:22.374 INFO:teuthology.orchestra.run.smithi077.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-02-10T08:47:22.374 INFO:teuthology.orchestra.run.smithi077.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-02-10T08:47:22.374 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-02-10T08:47:22.374 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-02-10T08:47:22.374 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-02-10T08:47:22.375 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-02-10T08:47:22.375 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-6.2-6.el8.noarch 2024-02-10T08:47:22.375 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-02-10T08:47:22.375 INFO:teuthology.orchestra.run.smithi077.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-02-10T08:47:22.375 INFO:teuthology.orchestra.run.smithi077.stdout: python3-mako-1.0.6-14.el8.noarch 2024-02-10T08:47:22.375 INFO:teuthology.orchestra.run.smithi077.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-02-10T08:47:22.375 INFO:teuthology.orchestra.run.smithi077.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-02-10T08:47:22.375 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-02-10T08:47:22.375 INFO:teuthology.orchestra.run.smithi077.stdout: python3-portend-2.6-1.el8.noarch 2024-02-10T08:47:22.375 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-02-10T08:47:22.375 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pycparser-2.14-14.el8.noarch 2024-02-10T08:47:22.375 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pytz-2017.2-11.el8.noarch 2024-02-10T08:47:22.375 INFO:teuthology.orchestra.run.smithi077.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-02-10T08:47:22.375 INFO:teuthology.orchestra.run.smithi077.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-02-10T08:47:22.376 INFO:teuthology.orchestra.run.smithi077.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-02-10T08:47:22.376 INFO:teuthology.orchestra.run.smithi077.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-02-10T08:47:22.376 INFO:teuthology.orchestra.run.smithi077.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-02-10T08:47:22.376 INFO:teuthology.orchestra.run.smithi077.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-02-10T08:47:22.376 INFO:teuthology.orchestra.run.smithi077.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-02-10T08:47:22.376 INFO:teuthology.orchestra.run.smithi077.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-02-10T08:47:22.376 INFO:teuthology.orchestra.run.smithi077.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-02-10T08:47:22.376 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:47:22.376 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-02-10T08:47:22.606 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-base 2024-02-10T08:47:22.680 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph-base 2024-02-10T08:47:23.364 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:12 ago on Sat 10 Feb 2024 08:46:11 AM UTC. 2024-02-10T08:47:23.435 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:01:11 ago on Sat 10 Feb 2024 08:46:12 AM UTC. 2024-02-10T08:47:24.841 INFO:teuthology.orchestra.run.smithi155.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-10T08:47:24.888 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-02-10T08:47:24.889 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-02-10T08:47:24.889 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-02-10T08:47:24.891 INFO:teuthology.orchestra.run.smithi077.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-10T08:47:24.938 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-02-10T08:47:24.939 INFO:teuthology.orchestra.run.smithi077.stdout:Nothing to do. 2024-02-10T08:47:24.939 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-02-10T08:47:25.048 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install cephadm 2024-02-10T08:47:25.239 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install cephadm 2024-02-10T08:47:25.796 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:01:13 ago on Sat 10 Feb 2024 08:46:12 AM UTC. 2024-02-10T08:47:25.998 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:14 ago on Sat 10 Feb 2024 08:46:11 AM UTC. 2024-02-10T08:47:27.310 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-02-10T08:47:27.310 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-10T08:47:27.310 INFO:teuthology.orchestra.run.smithi077.stdout: Package Architecture Version Repository Size 2024-02-10T08:47:27.310 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-10T08:47:27.310 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2024-02-10T08:47:27.310 INFO:teuthology.orchestra.run.smithi077.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2024-02-10T08:47:27.311 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:47:27.311 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2024-02-10T08:47:27.311 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-10T08:47:27.311 INFO:teuthology.orchestra.run.smithi077.stdout:Install 1 Package 2024-02-10T08:47:27.311 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:47:27.311 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 71 k 2024-02-10T08:47:27.311 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 290 k 2024-02-10T08:47:27.312 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2024-02-10T08:47:27.365 INFO:teuthology.orchestra.run.smithi077.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.3 MB/s | 71 kB 00:00 2024-02-10T08:47:27.365 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:47:27.366 INFO:teuthology.orchestra.run.smithi077.stdout:Total 1.3 MB/s | 71 kB 00:00 2024-02-10T08:47:27.366 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2024-02-10T08:47:27.373 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2024-02-10T08:47:27.373 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2024-02-10T08:47:27.374 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2024-02-10T08:47:27.377 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2024-02-10T08:47:27.421 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2024-02-10T08:47:27.515 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-02-10T08:47:27.516 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-02-10T08:47:27.516 INFO:teuthology.orchestra.run.smithi155.stdout: Package Architecture Version Repository Size 2024-02-10T08:47:27.516 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-02-10T08:47:27.516 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-02-10T08:47:27.516 INFO:teuthology.orchestra.run.smithi155.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2024-02-10T08:47:27.516 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:47:27.516 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-02-10T08:47:27.516 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-02-10T08:47:27.516 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-02-10T08:47:27.517 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:47:27.517 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 71 k 2024-02-10T08:47:27.517 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 290 k 2024-02-10T08:47:27.517 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-02-10T08:47:27.572 INFO:teuthology.orchestra.run.smithi155.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.3 MB/s | 71 kB 00:00 2024-02-10T08:47:27.573 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:47:27.573 INFO:teuthology.orchestra.run.smithi155.stdout:Total 1.2 MB/s | 71 kB 00:00 2024-02-10T08:47:27.573 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-02-10T08:47:27.580 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-02-10T08:47:27.580 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-02-10T08:47:27.582 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-02-10T08:47:27.585 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-02-10T08:47:27.653 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-02-10T08:47:27.954 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-10T08:47:28.128 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-10T08:47:28.209 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-10T08:47:28.354 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-10T08:47:28.388 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-10T08:47:28.615 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-10T08:47:28.706 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-10T08:47:28.706 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:47:28.706 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2024-02-10T08:47:28.706 INFO:teuthology.orchestra.run.smithi077.stdout: cephadm-2:16.2.4-0.el8.noarch 2024-02-10T08:47:28.706 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:47:28.706 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-02-10T08:47:28.994 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-10T08:47:28.994 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:47:28.994 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-02-10T08:47:28.994 INFO:teuthology.orchestra.run.smithi155.stdout: cephadm-2:16.2.4-0.el8.noarch 2024-02-10T08:47:28.994 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:47:28.994 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-02-10T08:47:28.997 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph-immutable-object-cache 2024-02-10T08:47:29.199 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-immutable-object-cache 2024-02-10T08:47:29.751 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:01:17 ago on Sat 10 Feb 2024 08:46:12 AM UTC. 2024-02-10T08:47:29.957 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:18 ago on Sat 10 Feb 2024 08:46:11 AM UTC. 2024-02-10T08:47:31.246 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-02-10T08:47:31.247 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-10T08:47:31.247 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repo Size 2024-02-10T08:47:31.247 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-10T08:47:31.247 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2024-02-10T08:47:31.247 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2024-02-10T08:47:31.247 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:47:31.247 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2024-02-10T08:47:31.247 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-10T08:47:31.247 INFO:teuthology.orchestra.run.smithi077.stdout:Install 1 Package 2024-02-10T08:47:31.247 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:47:31.248 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 163 k 2024-02-10T08:47:31.248 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 587 k 2024-02-10T08:47:31.248 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2024-02-10T08:47:31.465 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-02-10T08:47:31.465 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-02-10T08:47:31.465 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-02-10T08:47:31.465 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-02-10T08:47:31.465 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-02-10T08:47:31.465 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2024-02-10T08:47:31.465 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:47:31.466 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-02-10T08:47:31.466 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-02-10T08:47:31.466 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-02-10T08:47:31.466 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:47:31.466 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 163 k 2024-02-10T08:47:31.466 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 587 k 2024-02-10T08:47:31.466 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-02-10T08:47:31.548 INFO:teuthology.orchestra.run.smithi155.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 2.0 MB/s | 163 kB 00:00 2024-02-10T08:47:31.549 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:47:31.549 INFO:teuthology.orchestra.run.smithi155.stdout:Total 1.9 MB/s | 163 kB 00:00 2024-02-10T08:47:31.549 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-02-10T08:47:31.559 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-02-10T08:47:31.560 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-02-10T08:47:31.594 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 471 kB/s | 163 kB 00:00 2024-02-10T08:47:31.595 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:47:31.595 INFO:teuthology.orchestra.run.smithi077.stdout:Total 469 kB/s | 163 kB 00:00 2024-02-10T08:47:31.595 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2024-02-10T08:47:31.606 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2024-02-10T08:47:31.606 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2024-02-10T08:47:31.606 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-02-10T08:47:31.610 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-02-10T08:47:31.653 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2024-02-10T08:47:31.656 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2024-02-10T08:47:31.716 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-02-10T08:47:31.759 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2024-02-10T08:47:31.925 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T08:47:31.973 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T08:47:32.353 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T08:47:32.381 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T08:47:32.772 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T08:47:32.772 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:47:32.772 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2024-02-10T08:47:32.772 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2024-02-10T08:47:32.772 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:47:32.772 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-02-10T08:47:32.784 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T08:47:32.785 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:47:32.785 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-02-10T08:47:32.785 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2024-02-10T08:47:32.785 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:47:32.785 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-02-10T08:47:32.990 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-mgr 2024-02-10T08:47:33.010 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph-mgr 2024-02-10T08:47:33.754 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:22 ago on Sat 10 Feb 2024 08:46:11 AM UTC. 2024-02-10T08:47:33.771 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:01:21 ago on Sat 10 Feb 2024 08:46:12 AM UTC. 2024-02-10T08:47:35.196 INFO:teuthology.orchestra.run.smithi155.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-10T08:47:35.236 INFO:teuthology.orchestra.run.smithi077.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-10T08:47:35.242 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-02-10T08:47:35.243 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-02-10T08:47:35.243 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-02-10T08:47:35.282 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-02-10T08:47:35.283 INFO:teuthology.orchestra.run.smithi077.stdout:Nothing to do. 2024-02-10T08:47:35.283 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-02-10T08:47:35.345 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-mgr-dashboard 2024-02-10T08:47:35.400 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph-mgr-dashboard 2024-02-10T08:47:36.094 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:25 ago on Sat 10 Feb 2024 08:46:11 AM UTC. 2024-02-10T08:47:36.147 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:01:24 ago on Sat 10 Feb 2024 08:46:12 AM UTC. 2024-02-10T08:47:37.613 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-02-10T08:47:37.613 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-02-10T08:47:37.614 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-02-10T08:47:37.614 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-02-10T08:47:37.614 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-02-10T08:47:37.614 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2024-02-10T08:47:37.614 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-02-10T08:47:37.614 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2024-02-10T08:47:37.614 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2024-02-10T08:47:37.614 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-02-10T08:47:37.614 INFO:teuthology.orchestra.run.smithi155.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-02-10T08:47:37.614 INFO:teuthology.orchestra.run.smithi155.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-02-10T08:47:37.614 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:47:37.615 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-02-10T08:47:37.615 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-02-10T08:47:37.615 INFO:teuthology.orchestra.run.smithi155.stdout:Install 6 Packages 2024-02-10T08:47:37.615 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:47:37.615 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 3.0 M 2024-02-10T08:47:37.615 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 15 M 2024-02-10T08:47:37.615 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-02-10T08:47:37.665 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-02-10T08:47:37.666 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-10T08:47:37.666 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repository Size 2024-02-10T08:47:37.667 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-10T08:47:37.667 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2024-02-10T08:47:37.667 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2024-02-10T08:47:37.667 INFO:teuthology.orchestra.run.smithi077.stdout:Installing dependencies: 2024-02-10T08:47:37.667 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2024-02-10T08:47:37.667 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2024-02-10T08:47:37.667 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-02-10T08:47:37.667 INFO:teuthology.orchestra.run.smithi077.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-02-10T08:47:37.667 INFO:teuthology.orchestra.run.smithi077.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-02-10T08:47:37.667 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:47:37.667 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2024-02-10T08:47:37.667 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-10T08:47:37.668 INFO:teuthology.orchestra.run.smithi077.stdout:Install 6 Packages 2024-02-10T08:47:37.668 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:47:37.668 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 3.0 M 2024-02-10T08:47:37.668 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 15 M 2024-02-10T08:47:37.669 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2024-02-10T08:47:37.951 INFO:teuthology.orchestra.run.smithi155.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 346 kB/s | 24 kB 00:00 2024-02-10T08:47:37.976 INFO:teuthology.orchestra.run.smithi155.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 94 kB/s | 8.9 kB 00:00 2024-02-10T08:47:38.032 INFO:teuthology.orchestra.run.smithi077.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 403 kB/s | 24 kB 00:00 2024-02-10T08:47:38.051 INFO:teuthology.orchestra.run.smithi155.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 16 MB/s | 2.7 MB 00:00 2024-02-10T08:47:38.057 INFO:teuthology.orchestra.run.smithi077.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 105 kB/s | 8.9 kB 00:00 2024-02-10T08:47:38.157 INFO:teuthology.orchestra.run.smithi077.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 14 MB/s | 2.7 MB 00:00 2024-02-10T08:47:38.177 INFO:teuthology.orchestra.run.smithi155.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 189 kB/s | 43 kB 00:00 2024-02-10T08:47:38.202 INFO:teuthology.orchestra.run.smithi155.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 149 kB/s | 34 kB 00:00 2024-02-10T08:47:38.224 INFO:teuthology.orchestra.run.smithi077.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 222 kB/s | 43 kB 00:00 2024-02-10T08:47:38.269 INFO:teuthology.orchestra.run.smithi155.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 905 kB/s | 196 kB 00:00 2024-02-10T08:47:38.270 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:47:38.270 INFO:teuthology.orchestra.run.smithi155.stdout:Total 4.5 MB/s | 3.0 MB 00:00 2024-02-10T08:47:38.280 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-02-10T08:47:38.289 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-02-10T08:47:38.290 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-02-10T08:47:38.299 INFO:teuthology.orchestra.run.smithi077.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 139 kB/s | 34 kB 00:00 2024-02-10T08:47:38.391 INFO:teuthology.orchestra.run.smithi077.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 841 kB/s | 196 kB 00:00 2024-02-10T08:47:38.393 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:47:38.393 INFO:teuthology.orchestra.run.smithi077.stdout:Total 4.1 MB/s | 3.0 MB 00:00 2024-02-10T08:47:38.403 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2024-02-10T08:47:38.412 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2024-02-10T08:47:38.412 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2024-02-10T08:47:38.458 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-02-10T08:47:38.461 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-02-10T08:47:38.567 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2024-02-10T08:47:38.570 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2024-02-10T08:47:38.760 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-02-10T08:47:38.835 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2024-02-10T08:47:39.005 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-02-10T08:47:39.069 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-02-10T08:47:39.275 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-02-10T08:47:39.373 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-02-10T08:47:39.501 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-02-10T08:47:39.607 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-02-10T08:47:39.639 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2024-02-10T08:47:39.737 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2024-02-10T08:47:40.680 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2024-02-10T08:47:40.754 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2024-02-10T08:47:40.990 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-02-10T08:47:41.055 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-02-10T08:47:41.264 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-02-10T08:47:41.264 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2024-02-10T08:47:41.264 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2024-02-10T08:47:41.264 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2024-02-10T08:47:41.264 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-02-10T08:47:41.264 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-02-10T08:47:41.339 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-02-10T08:47:41.339 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2024-02-10T08:47:41.340 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2024-02-10T08:47:41.340 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2024-02-10T08:47:41.340 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-02-10T08:47:41.340 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-02-10T08:47:41.657 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-02-10T08:47:41.657 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:47:41.657 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-02-10T08:47:41.657 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2024-02-10T08:47:41.657 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2024-02-10T08:47:41.657 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2024-02-10T08:47:41.658 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-02-10T08:47:41.658 INFO:teuthology.orchestra.run.smithi155.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-02-10T08:47:41.658 INFO:teuthology.orchestra.run.smithi155.stdout: python3-routes-2.4.1-12.el8.noarch 2024-02-10T08:47:41.658 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:47:41.658 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-02-10T08:47:41.738 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-02-10T08:47:41.739 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:47:41.739 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2024-02-10T08:47:41.739 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2024-02-10T08:47:41.739 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2024-02-10T08:47:41.739 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2024-02-10T08:47:41.739 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-02-10T08:47:41.739 INFO:teuthology.orchestra.run.smithi077.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-02-10T08:47:41.739 INFO:teuthology.orchestra.run.smithi077.stdout: python3-routes-2.4.1-12.el8.noarch 2024-02-10T08:47:41.739 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:47:41.739 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-02-10T08:47:41.778 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-10T08:47:41.866 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-10T08:47:42.532 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:31 ago on Sat 10 Feb 2024 08:46:11 AM UTC. 2024-02-10T08:47:42.615 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:01:30 ago on Sat 10 Feb 2024 08:46:12 AM UTC. 2024-02-10T08:47:44.048 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-02-10T08:47:44.049 INFO:teuthology.orchestra.run.smithi155.stdout:============================================================================================ 2024-02-10T08:47:44.049 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-02-10T08:47:44.049 INFO:teuthology.orchestra.run.smithi155.stdout:============================================================================================ 2024-02-10T08:47:44.049 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-02-10T08:47:44.049 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2024-02-10T08:47:44.049 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-02-10T08:47:44.049 INFO:teuthology.orchestra.run.smithi155.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-02-10T08:47:44.050 INFO:teuthology.orchestra.run.smithi155.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-02-10T08:47:44.050 INFO:teuthology.orchestra.run.smithi155.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-02-10T08:47:44.050 INFO:teuthology.orchestra.run.smithi155.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-02-10T08:47:44.050 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-02-10T08:47:44.050 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-02-10T08:47:44.050 INFO:teuthology.orchestra.run.smithi155.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-02-10T08:47:44.050 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:47:44.050 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-02-10T08:47:44.050 INFO:teuthology.orchestra.run.smithi155.stdout:============================================================================================ 2024-02-10T08:47:44.050 INFO:teuthology.orchestra.run.smithi155.stdout:Install 8 Packages 2024-02-10T08:47:44.050 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:47:44.051 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 37 M 2024-02-10T08:47:44.051 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 210 M 2024-02-10T08:47:44.051 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-02-10T08:47:44.115 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-02-10T08:47:44.116 INFO:teuthology.orchestra.run.smithi077.stdout:============================================================================================ 2024-02-10T08:47:44.116 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repository Size 2024-02-10T08:47:44.116 INFO:teuthology.orchestra.run.smithi077.stdout:============================================================================================ 2024-02-10T08:47:44.116 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2024-02-10T08:47:44.117 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2024-02-10T08:47:44.117 INFO:teuthology.orchestra.run.smithi077.stdout:Installing dependencies: 2024-02-10T08:47:44.117 INFO:teuthology.orchestra.run.smithi077.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-02-10T08:47:44.117 INFO:teuthology.orchestra.run.smithi077.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-02-10T08:47:44.117 INFO:teuthology.orchestra.run.smithi077.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-02-10T08:47:44.117 INFO:teuthology.orchestra.run.smithi077.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-02-10T08:47:44.117 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-02-10T08:47:44.117 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-02-10T08:47:44.117 INFO:teuthology.orchestra.run.smithi077.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-02-10T08:47:44.117 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:47:44.117 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2024-02-10T08:47:44.117 INFO:teuthology.orchestra.run.smithi077.stdout:============================================================================================ 2024-02-10T08:47:44.117 INFO:teuthology.orchestra.run.smithi077.stdout:Install 8 Packages 2024-02-10T08:47:44.118 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:47:44.118 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 37 M 2024-02-10T08:47:44.118 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 210 M 2024-02-10T08:47:44.118 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2024-02-10T08:47:44.667 INFO:teuthology.orchestra.run.smithi155.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 47 MB/s | 8.6 MB 00:00 2024-02-10T08:47:44.721 INFO:teuthology.orchestra.run.smithi077.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 59 MB/s | 8.6 MB 00:00 2024-02-10T08:47:45.175 INFO:teuthology.orchestra.run.smithi155.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 7.0 MB/s | 4.9 MB 00:00 2024-02-10T08:47:45.263 INFO:teuthology.orchestra.run.smithi077.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 7.1 MB/s | 4.9 MB 00:00 2024-02-10T08:47:45.333 INFO:teuthology.orchestra.run.smithi155.stdout:(3/8): openblas-0.3.15-4.el8.x86_64.rpm 5.6 MB/s | 4.7 MB 00:00 2024-02-10T08:47:45.346 INFO:teuthology.orchestra.run.smithi077.stdout:(3/8): openblas-0.3.15-4.el8.x86_64.rpm 6.1 MB/s | 4.7 MB 00:00 2024-02-10T08:47:45.358 INFO:teuthology.orchestra.run.smithi155.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.2 MB/s | 225 kB 00:00 2024-02-10T08:47:45.371 INFO:teuthology.orchestra.run.smithi077.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.0 MB/s | 225 kB 00:00 2024-02-10T08:47:45.597 INFO:teuthology.orchestra.run.smithi077.stdout:(5/8): libgfortran-8.5.0-21.el8.x86_64.rpm 2.8 MB/s | 645 kB 00:00 2024-02-10T08:47:45.609 INFO:teuthology.orchestra.run.smithi155.stdout:(5/8): libgfortran-8.5.0-21.el8.x86_64.rpm 2.5 MB/s | 645 kB 00:00 2024-02-10T08:47:45.689 INFO:teuthology.orchestra.run.smithi077.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 1.8 MB/s | 172 kB 00:00 2024-02-10T08:47:45.884 INFO:teuthology.orchestra.run.smithi155.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 627 kB/s | 172 kB 00:00 2024-02-10T08:47:46.743 INFO:teuthology.orchestra.run.smithi155.stdout:(7/8): python3-scipy-1.0.0-21.module_el8+762+77 10 MB/s | 14 MB 00:01 2024-02-10T08:47:47.460 INFO:teuthology.orchestra.run.smithi155.stdout:(8/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 1.3 MB/s | 3.7 MB 00:02 2024-02-10T08:47:47.460 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:47:47.461 INFO:teuthology.orchestra.run.smithi155.stdout:Total 11 MB/s | 37 MB 00:03 2024-02-10T08:47:47.695 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-02-10T08:47:47.714 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-02-10T08:47:47.714 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-02-10T08:47:48.074 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-02-10T08:47:48.077 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-02-10T08:47:48.108 INFO:teuthology.orchestra.run.smithi077.stdout:(7/8): python3-scipy-1.0.0-21.module_el8+762+77 5.1 MB/s | 14 MB 00:02 2024-02-10T08:47:48.700 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-02-10T08:47:48.889 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-10T08:47:49.011 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-10T08:47:49.164 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-10T08:47:49.225 INFO:teuthology.orchestra.run.smithi077.stdout:(8/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 849 kB/s | 3.7 MB 00:04 2024-02-10T08:47:49.225 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:47:49.225 INFO:teuthology.orchestra.run.smithi077.stdout:Total 7.3 MB/s | 37 MB 00:05 2024-02-10T08:47:49.460 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2024-02-10T08:47:49.478 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2024-02-10T08:47:49.478 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2024-02-10T08:47:49.768 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-10T08:47:49.800 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2024-02-10T08:47:49.803 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2024-02-10T08:47:49.997 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-02-10T08:47:50.423 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2024-02-10T08:47:50.587 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-10T08:47:50.614 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-02-10T08:47:50.730 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-10T08:47:50.805 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-10T08:47:50.896 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-10T08:47:51.498 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-10T08:47:51.506 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-10T08:47:51.712 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-02-10T08:47:51.859 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-02-10T08:47:52.139 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-10T08:47:52.316 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-02-10T08:47:52.462 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-10T08:47:53.133 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-10T08:47:53.508 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-02-10T08:47:53.905 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-10T08:47:54.437 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-10T08:47:55.211 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-02-10T08:47:55.379 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-02-10T08:47:55.729 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-02-10T08:47:55.729 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2024-02-10T08:47:55.730 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-02-10T08:47:55.730 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-02-10T08:47:55.730 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-02-10T08:47:55.730 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-02-10T08:47:55.730 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-02-10T08:47:55.730 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-02-10T08:47:56.165 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-02-10T08:47:56.165 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:47:56.165 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-02-10T08:47:56.165 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2024-02-10T08:47:56.166 INFO:teuthology.orchestra.run.smithi155.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-02-10T08:47:56.166 INFO:teuthology.orchestra.run.smithi155.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-02-10T08:47:56.166 INFO:teuthology.orchestra.run.smithi155.stdout: openblas-0.3.15-4.el8.x86_64 2024-02-10T08:47:56.166 INFO:teuthology.orchestra.run.smithi155.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-02-10T08:47:56.166 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-02-10T08:47:56.166 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-02-10T08:47:56.166 INFO:teuthology.orchestra.run.smithi155.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-02-10T08:47:56.166 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:47:56.166 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-02-10T08:47:56.206 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-10T08:47:56.266 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-mgr-rook 2024-02-10T08:47:57.022 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:46 ago on Sat 10 Feb 2024 08:46:11 AM UTC. 2024-02-10T08:47:57.526 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-02-10T08:47:57.703 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-02-10T08:47:58.025 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-02-10T08:47:58.025 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2024-02-10T08:47:58.025 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-02-10T08:47:58.025 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-02-10T08:47:58.025 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-02-10T08:47:58.025 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-02-10T08:47:58.025 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-02-10T08:47:58.025 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-02-10T08:47:58.416 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-02-10T08:47:58.416 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:47:58.416 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2024-02-10T08:47:58.416 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2024-02-10T08:47:58.416 INFO:teuthology.orchestra.run.smithi077.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-02-10T08:47:58.416 INFO:teuthology.orchestra.run.smithi077.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-02-10T08:47:58.416 INFO:teuthology.orchestra.run.smithi077.stdout: openblas-0.3.15-4.el8.x86_64 2024-02-10T08:47:58.416 INFO:teuthology.orchestra.run.smithi077.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-02-10T08:47:58.416 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-02-10T08:47:58.416 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-02-10T08:47:58.417 INFO:teuthology.orchestra.run.smithi077.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-02-10T08:47:58.417 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:47:58.417 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-02-10T08:47:58.531 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-02-10T08:47:58.532 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-02-10T08:47:58.533 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-02-10T08:47:58.533 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-02-10T08:47:58.533 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-02-10T08:47:58.533 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2024-02-10T08:47:58.533 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-02-10T08:47:58.533 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-02-10T08:47:58.533 INFO:teuthology.orchestra.run.smithi155.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-02-10T08:47:58.533 INFO:teuthology.orchestra.run.smithi155.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-02-10T08:47:58.533 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-02-10T08:47:58.533 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-02-10T08:47:58.533 INFO:teuthology.orchestra.run.smithi155.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-02-10T08:47:58.534 INFO:teuthology.orchestra.run.smithi155.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-02-10T08:47:58.534 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-02-10T08:47:58.534 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-02-10T08:47:58.534 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-02-10T08:47:58.534 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-02-10T08:47:58.534 INFO:teuthology.orchestra.run.smithi155.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-02-10T08:47:58.534 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:47:58.534 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-02-10T08:47:58.534 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-02-10T08:47:58.534 INFO:teuthology.orchestra.run.smithi155.stdout:Install 13 Packages 2024-02-10T08:47:58.534 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:47:58.535 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 1.9 M 2024-02-10T08:47:58.535 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 24 M 2024-02-10T08:47:58.535 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-02-10T08:47:58.547 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph-mgr-rook 2024-02-10T08:47:58.687 INFO:teuthology.orchestra.run.smithi155.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 599 kB/s | 27 kB 00:00 2024-02-10T08:47:58.779 INFO:teuthology.orchestra.run.smithi155.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 194 kB/s | 27 kB 00:00 2024-02-10T08:47:58.804 INFO:teuthology.orchestra.run.smithi155.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 123 kB/s | 20 kB 00:00 2024-02-10T08:47:58.880 INFO:teuthology.orchestra.run.smithi155.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 1.1 MB/s | 110 kB 00:00 2024-02-10T08:47:58.905 INFO:teuthology.orchestra.run.smithi155.stdout:(5/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 580 kB/s | 126 kB 00:00 2024-02-10T08:47:59.061 INFO:teuthology.orchestra.run.smithi155.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 236 kB/s | 43 kB 00:00 2024-02-10T08:47:59.098 INFO:teuthology.orchestra.run.smithi155.stdout:(7/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 529 kB/s | 155 kB 00:00 2024-02-10T08:47:59.198 INFO:teuthology.orchestra.run.smithi155.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 113 kB/s | 33 kB 00:00 2024-02-10T08:47:59.240 INFO:teuthology.orchestra.run.smithi155.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 93 kB/s | 16 kB 00:00 2024-02-10T08:47:59.295 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:01:47 ago on Sat 10 Feb 2024 08:46:12 AM UTC. 2024-02-10T08:47:59.340 INFO:teuthology.orchestra.run.smithi155.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 336 kB/s | 81 kB 00:00 2024-02-10T08:47:59.374 INFO:teuthology.orchestra.run.smithi155.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 480 kB/s | 64 kB 00:00 2024-02-10T08:47:59.399 INFO:teuthology.orchestra.run.smithi155.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 1.0 MB/s | 61 kB 00:00 2024-02-10T08:47:59.433 INFO:teuthology.orchestra.run.smithi155.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 5.0 MB/s | 1.2 MB 00:00 2024-02-10T08:47:59.435 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:47:59.435 INFO:teuthology.orchestra.run.smithi155.stdout:Total 2.1 MB/s | 1.9 MB 00:00 2024-02-10T08:47:59.489 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-02-10T08:47:59.504 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-02-10T08:47:59.504 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-02-10T08:47:59.590 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-02-10T08:47:59.595 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-02-10T08:48:00.159 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-02-10T08:48:00.407 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-02-10T08:48:00.651 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-02-10T08:48:00.803 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-02-10T08:48:00.805 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-10T08:48:00.805 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repository Size 2024-02-10T08:48:00.805 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-10T08:48:00.805 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2024-02-10T08:48:00.805 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2024-02-10T08:48:00.805 INFO:teuthology.orchestra.run.smithi077.stdout:Installing dependencies: 2024-02-10T08:48:00.806 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-02-10T08:48:00.806 INFO:teuthology.orchestra.run.smithi077.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-02-10T08:48:00.806 INFO:teuthology.orchestra.run.smithi077.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-02-10T08:48:00.806 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-02-10T08:48:00.806 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-02-10T08:48:00.806 INFO:teuthology.orchestra.run.smithi077.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-02-10T08:48:00.806 INFO:teuthology.orchestra.run.smithi077.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-02-10T08:48:00.806 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-02-10T08:48:00.806 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-02-10T08:48:00.806 INFO:teuthology.orchestra.run.smithi077.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-02-10T08:48:00.806 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-02-10T08:48:00.806 INFO:teuthology.orchestra.run.smithi077.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-02-10T08:48:00.806 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:48:00.806 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2024-02-10T08:48:00.807 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-10T08:48:00.807 INFO:teuthology.orchestra.run.smithi077.stdout:Install 13 Packages 2024-02-10T08:48:00.807 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:48:00.807 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 1.9 M 2024-02-10T08:48:00.807 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 24 M 2024-02-10T08:48:00.807 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2024-02-10T08:48:00.914 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-02-10T08:48:01.168 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-02-10T08:48:01.264 INFO:teuthology.orchestra.run.smithi077.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 455 kB/s | 27 kB 00:00 2024-02-10T08:48:01.364 INFO:teuthology.orchestra.run.smithi077.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 125 kB/s | 20 kB 00:00 2024-02-10T08:48:01.384 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-02-10T08:48:01.389 INFO:teuthology.orchestra.run.smithi077.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 144 kB/s | 27 kB 00:00 2024-02-10T08:48:01.465 INFO:teuthology.orchestra.run.smithi077.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 630 kB/s | 126 kB 00:00 2024-02-10T08:48:01.490 INFO:teuthology.orchestra.run.smithi077.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 876 kB/s | 110 kB 00:00 2024-02-10T08:48:01.501 INFO:teuthology.orchestra.run.smithi077.stdout:[MIRROR] python3-requests-oauthlib-1.0.0-1.el8.noarch.rpm: Status code: 503 for http://mirror.ette.biz/centos/8-stream/BaseOS/x86_64/os/Packages/python3-requests-oauthlib-1.0.0-1.el8.noarch.rpm (IP: 38.105.93.109) 2024-02-10T08:48:01.573 INFO:teuthology.orchestra.run.smithi077.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 843 kB/s | 155 kB 00:00 2024-02-10T08:48:01.599 INFO:teuthology.orchestra.run.smithi077.stdout:(7/13): python3-cachetools-3.1.1-4.el8.noarch.r 303 kB/s | 33 kB 00:00 2024-02-10T08:48:01.643 INFO:teuthology.orchestra.run.smithi077.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 243 kB/s | 16 kB 00:00 2024-02-10T08:48:01.648 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-02-10T08:48:01.666 INFO:teuthology.orchestra.run.smithi077.stdout:(9/13): python3-google-auth-1.1.1-10.el8.noarch 1.2 MB/s | 81 kB 00:00 2024-02-10T08:48:01.700 INFO:teuthology.orchestra.run.smithi077.stdout:(10/13): python3-rsa-4.9-2.el8.noarch.rpm 1.9 MB/s | 64 kB 00:00 2024-02-10T08:48:01.733 INFO:teuthology.orchestra.run.smithi077.stdout:(11/13): python3-kubernetes-11.0.0-6.el8.noarch 13 MB/s | 1.2 MB 00:00 2024-02-10T08:48:01.758 INFO:teuthology.orchestra.run.smithi077.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 1.0 MB/s | 61 kB 00:00 2024-02-10T08:48:01.964 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-02-10T08:48:02.207 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-02-10T08:48:03.226 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-02-10T08:48:03.537 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-02-10T08:48:03.807 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-02-10T08:48:04.002 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-02-10T08:48:04.203 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-02-10T08:48:04.497 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-02-10T08:48:04.497 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2024-02-10T08:48:04.497 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-02-10T08:48:04.497 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-02-10T08:48:04.497 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-02-10T08:48:04.498 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-02-10T08:48:04.498 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-02-10T08:48:04.498 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-02-10T08:48:04.498 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-02-10T08:48:04.498 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-02-10T08:48:04.498 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-02-10T08:48:04.498 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-02-10T08:48:04.498 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-02-10T08:48:05.020 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-02-10T08:48:05.020 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:48:05.020 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-02-10T08:48:05.020 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2024-02-10T08:48:05.020 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-02-10T08:48:05.020 INFO:teuthology.orchestra.run.smithi155.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-02-10T08:48:05.020 INFO:teuthology.orchestra.run.smithi155.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-02-10T08:48:05.020 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-02-10T08:48:05.020 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-02-10T08:48:05.021 INFO:teuthology.orchestra.run.smithi155.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-02-10T08:48:05.021 INFO:teuthology.orchestra.run.smithi155.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-02-10T08:48:05.021 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-02-10T08:48:05.021 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-02-10T08:48:05.021 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-02-10T08:48:05.021 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rsa-4.9-2.el8.noarch 2024-02-10T08:48:05.021 INFO:teuthology.orchestra.run.smithi155.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-02-10T08:48:05.021 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:48:05.021 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-02-10T08:48:05.132 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-mgr-cephadm 2024-02-10T08:48:05.887 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:54 ago on Sat 10 Feb 2024 08:46:11 AM UTC. 2024-02-10T08:48:07.401 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-02-10T08:48:07.402 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-02-10T08:48:07.403 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-02-10T08:48:07.403 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-02-10T08:48:07.403 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-02-10T08:48:07.403 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2024-02-10T08:48:07.403 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-02-10T08:48:07.403 INFO:teuthology.orchestra.run.smithi155.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-02-10T08:48:07.403 INFO:teuthology.orchestra.run.smithi155.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-02-10T08:48:07.403 INFO:teuthology.orchestra.run.smithi155.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-02-10T08:48:07.403 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2024-02-10T08:48:07.403 INFO:teuthology.orchestra.run.smithi155.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-02-10T08:48:07.403 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:48:07.403 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-02-10T08:48:07.403 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-02-10T08:48:07.403 INFO:teuthology.orchestra.run.smithi155.stdout:Install 6 Packages 2024-02-10T08:48:07.404 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:48:07.404 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 5.8 M 2024-02-10T08:48:07.405 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 24 M 2024-02-10T08:48:07.405 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-02-10T08:48:07.684 INFO:teuthology.orchestra.run.smithi155.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 1.4 MB/s | 89 kB 00:00 2024-02-10T08:48:07.800 INFO:teuthology.orchestra.run.smithi155.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 184 kB/s | 21 kB 00:00 2024-02-10T08:48:07.992 INFO:teuthology.orchestra.run.smithi155.stdout:(3/6): python3-execnet-1.7.1-1.el8.noarch.rpm 1.8 MB/s | 355 kB 00:00 2024-02-10T08:48:08.051 INFO:teuthology.orchestra.run.smithi155.stdout:(4/6): python3-remoto-1.2.1-1.el8.noarch.rpm 836 kB/s | 49 kB 00:00 2024-02-10T08:48:08.168 INFO:teuthology.orchestra.run.smithi155.stdout:(5/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 986 kB/s | 538 kB 00:00 2024-02-10T08:48:08.377 INFO:teuthology.orchestra.run.smithi155.stdout:(6/6): python3-babel-2.5.1-7.el8.noarch.rpm 6.3 MB/s | 4.8 MB 00:00 2024-02-10T08:48:08.377 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:48:08.378 INFO:teuthology.orchestra.run.smithi155.stdout:Total 6.0 MB/s | 5.8 MB 00:00 2024-02-10T08:48:08.431 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-02-10T08:48:08.441 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-02-10T08:48:08.441 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-02-10T08:48:08.562 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-02-10T08:48:08.565 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-02-10T08:48:08.839 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-02-10T08:48:09.123 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-02-10T08:48:09.497 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-02-10T08:48:10.470 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-02-10T08:48:10.799 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-02-10T08:48:11.065 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2024-02-10T08:48:11.298 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-02-10T08:48:11.603 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-02-10T08:48:11.604 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2024-02-10T08:48:11.604 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-02-10T08:48:11.604 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2024-02-10T08:48:11.604 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-02-10T08:48:11.604 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-02-10T08:48:11.987 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-02-10T08:48:11.987 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:48:11.987 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-02-10T08:48:11.987 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2024-02-10T08:48:11.988 INFO:teuthology.orchestra.run.smithi155.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2024-02-10T08:48:11.988 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2024-02-10T08:48:11.988 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:48:11.988 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-02-10T08:48:12.114 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-fuse 2024-02-10T08:48:12.869 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:02:01 ago on Sat 10 Feb 2024 08:46:11 AM UTC. 2024-02-10T08:48:14.361 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-02-10T08:48:14.361 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-02-10T08:48:14.361 INFO:teuthology.orchestra.run.smithi155.stdout: Package Architecture Version Repository Size 2024-02-10T08:48:14.361 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-02-10T08:48:14.361 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-02-10T08:48:14.361 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2024-02-10T08:48:14.362 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-02-10T08:48:14.362 INFO:teuthology.orchestra.run.smithi155.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-02-10T08:48:14.362 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:48:14.362 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-02-10T08:48:14.362 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-02-10T08:48:14.362 INFO:teuthology.orchestra.run.smithi155.stdout:Install 2 Packages 2024-02-10T08:48:14.362 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:48:14.363 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 968 k 2024-02-10T08:48:14.363 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 3.2 M 2024-02-10T08:48:14.363 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-02-10T08:48:14.523 INFO:teuthology.orchestra.run.smithi155.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 7.3 MB/s | 885 kB 00:00 2024-02-10T08:48:14.606 INFO:teuthology.orchestra.run.smithi155.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 410 kB/s | 83 kB 00:00 2024-02-10T08:48:14.607 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:48:14.607 INFO:teuthology.orchestra.run.smithi155.stdout:Total 3.9 MB/s | 968 kB 00:00 2024-02-10T08:48:14.617 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-02-10T08:48:14.638 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-02-10T08:48:14.638 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-02-10T08:48:14.725 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-02-10T08:48:14.728 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-02-10T08:48:14.907 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-02-10T08:48:15.169 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-02-10T08:48:15.368 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-02-10T08:48:15.787 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-02-10T08:48:15.788 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2024-02-10T08:48:16.189 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-02-10T08:48:16.189 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:48:16.189 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-02-10T08:48:16.189 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-02-10T08:48:16.189 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:48:16.189 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-02-10T08:48:16.421 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install librados-devel 2024-02-10T08:48:17.185 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:02:06 ago on Sat 10 Feb 2024 08:46:11 AM UTC. 2024-02-10T08:48:18.670 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-02-10T08:48:18.671 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-02-10T08:48:18.671 INFO:teuthology.orchestra.run.smithi155.stdout: Package Architecture Version Repository Size 2024-02-10T08:48:18.671 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-02-10T08:48:18.671 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-02-10T08:48:18.671 INFO:teuthology.orchestra.run.smithi155.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2024-02-10T08:48:18.671 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:48:18.672 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-02-10T08:48:18.672 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-02-10T08:48:18.672 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-02-10T08:48:18.672 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:48:18.672 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 131 k 2024-02-10T08:48:18.673 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 439 k 2024-02-10T08:48:18.673 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-02-10T08:48:18.738 INFO:teuthology.orchestra.run.smithi155.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 1.9 MB/s | 131 kB 00:00 2024-02-10T08:48:18.739 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:48:18.739 INFO:teuthology.orchestra.run.smithi155.stdout:Total 1.9 MB/s | 131 kB 00:00 2024-02-10T08:48:18.739 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-02-10T08:48:18.745 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-02-10T08:48:18.745 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-02-10T08:48:18.796 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-02-10T08:48:18.799 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-02-10T08:48:18.906 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-02-10T08:48:19.158 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T08:48:19.456 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T08:48:20.111 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T08:48:20.111 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:48:20.111 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-02-10T08:48:20.111 INFO:teuthology.orchestra.run.smithi155.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2024-02-10T08:48:20.111 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:48:20.111 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-02-10T08:48:20.336 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install libcephfs2 2024-02-10T08:48:21.109 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:02:10 ago on Sat 10 Feb 2024 08:46:11 AM UTC. 2024-02-10T08:48:22.584 INFO:teuthology.orchestra.run.smithi155.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-10T08:48:22.631 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-02-10T08:48:22.632 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-02-10T08:48:22.632 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-02-10T08:48:22.727 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install libcephfs-devel 2024-02-10T08:48:23.477 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:02:12 ago on Sat 10 Feb 2024 08:46:11 AM UTC. 2024-02-10T08:48:24.986 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-02-10T08:48:24.987 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-02-10T08:48:24.987 INFO:teuthology.orchestra.run.smithi155.stdout: Package Architecture Version Repository Size 2024-02-10T08:48:24.987 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-02-10T08:48:24.987 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-02-10T08:48:24.987 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2024-02-10T08:48:24.987 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:48:24.987 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-02-10T08:48:24.988 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-02-10T08:48:24.988 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-02-10T08:48:24.988 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:48:24.988 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 24 k 2024-02-10T08:48:24.988 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 94 k 2024-02-10T08:48:24.988 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-02-10T08:48:25.043 INFO:teuthology.orchestra.run.smithi155.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 443 kB/s | 24 kB 00:00 2024-02-10T08:48:25.043 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:48:25.043 INFO:teuthology.orchestra.run.smithi155.stdout:Total 433 kB/s | 24 kB 00:00 2024-02-10T08:48:25.044 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-02-10T08:48:25.047 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-02-10T08:48:25.047 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-02-10T08:48:25.050 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-02-10T08:48:25.053 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-02-10T08:48:25.372 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-02-10T08:48:26.581 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T08:48:27.607 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T08:48:28.032 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T08:48:28.032 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:48:28.032 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-02-10T08:48:28.032 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2024-02-10T08:48:28.032 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:48:28.032 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-02-10T08:48:28.135 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install librados2 2024-02-10T08:48:28.890 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:02:17 ago on Sat 10 Feb 2024 08:46:11 AM UTC. 2024-02-10T08:48:30.366 INFO:teuthology.orchestra.run.smithi155.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-10T08:48:30.413 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-02-10T08:48:30.414 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-02-10T08:48:30.414 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-02-10T08:48:30.510 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install librbd1 2024-02-10T08:48:31.263 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:02:20 ago on Sat 10 Feb 2024 08:46:11 AM UTC. 2024-02-10T08:48:32.734 INFO:teuthology.orchestra.run.smithi155.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-10T08:48:32.780 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-02-10T08:48:32.781 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-02-10T08:48:32.781 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-02-10T08:48:33.057 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install python3-rados 2024-02-10T08:48:33.815 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:02:22 ago on Sat 10 Feb 2024 08:46:11 AM UTC. 2024-02-10T08:48:35.274 INFO:teuthology.orchestra.run.smithi155.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-10T08:48:35.320 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-02-10T08:48:35.322 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-02-10T08:48:35.322 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-02-10T08:48:35.414 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install python3-rgw 2024-02-10T08:48:36.164 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:02:25 ago on Sat 10 Feb 2024 08:46:11 AM UTC. 2024-02-10T08:48:37.626 INFO:teuthology.orchestra.run.smithi155.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-10T08:48:37.671 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-02-10T08:48:37.672 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-02-10T08:48:37.672 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-02-10T08:48:37.774 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install python3-cephfs 2024-02-10T08:48:38.528 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:02:27 ago on Sat 10 Feb 2024 08:46:11 AM UTC. 2024-02-10T08:48:39.977 INFO:teuthology.orchestra.run.smithi155.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-10T08:48:40.023 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-02-10T08:48:40.024 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-02-10T08:48:40.024 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-02-10T08:48:40.119 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install python3-rbd 2024-02-10T08:48:40.871 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:02:29 ago on Sat 10 Feb 2024 08:46:11 AM UTC. 2024-02-10T08:48:42.341 INFO:teuthology.orchestra.run.smithi155.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-10T08:48:42.387 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-02-10T08:48:42.388 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-02-10T08:48:42.388 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-02-10T08:48:42.494 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install rbd-fuse 2024-02-10T08:48:43.252 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:02:32 ago on Sat 10 Feb 2024 08:46:11 AM UTC. 2024-02-10T08:48:44.775 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-02-10T08:48:44.775 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-02-10T08:48:44.776 INFO:teuthology.orchestra.run.smithi155.stdout: Package Architecture Version Repository Size 2024-02-10T08:48:44.776 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-02-10T08:48:44.776 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-02-10T08:48:44.776 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2024-02-10T08:48:44.776 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:48:44.776 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-02-10T08:48:44.776 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-02-10T08:48:44.776 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-02-10T08:48:44.776 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:48:44.776 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 89 k 2024-02-10T08:48:44.776 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 298 k 2024-02-10T08:48:44.777 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-02-10T08:48:44.830 INFO:teuthology.orchestra.run.smithi155.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 1.6 MB/s | 89 kB 00:00 2024-02-10T08:48:44.830 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:48:44.830 INFO:teuthology.orchestra.run.smithi155.stdout:Total 1.6 MB/s | 89 kB 00:00 2024-02-10T08:48:44.831 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-02-10T08:48:44.840 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-02-10T08:48:44.840 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-02-10T08:48:44.890 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-02-10T08:48:44.893 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-02-10T08:48:45.022 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-02-10T08:48:45.221 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T08:48:45.558 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T08:48:45.999 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T08:48:45.999 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:48:46.000 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-02-10T08:48:46.000 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2024-02-10T08:48:46.000 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:48:46.000 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-02-10T08:48:46.266 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install rbd-mirror 2024-02-10T08:48:47.030 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:02:36 ago on Sat 10 Feb 2024 08:46:11 AM UTC. 2024-02-10T08:48:48.573 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-02-10T08:48:48.574 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-02-10T08:48:48.574 INFO:teuthology.orchestra.run.smithi155.stdout: Package Architecture Version Repository Size 2024-02-10T08:48:48.574 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-02-10T08:48:48.574 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-02-10T08:48:48.574 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-02-10T08:48:48.574 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:48:48.574 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-02-10T08:48:48.574 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-02-10T08:48:48.574 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-02-10T08:48:48.575 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:48:48.575 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 3.7 M 2024-02-10T08:48:48.575 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 18 M 2024-02-10T08:48:48.575 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-02-10T08:48:48.676 INFO:teuthology.orchestra.run.smithi155.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 37 MB/s | 3.7 MB 00:00 2024-02-10T08:48:48.676 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:48:48.677 INFO:teuthology.orchestra.run.smithi155.stdout:Total 36 MB/s | 3.7 MB 00:00 2024-02-10T08:48:48.677 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-02-10T08:48:48.689 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-02-10T08:48:48.690 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-02-10T08:48:48.752 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-02-10T08:48:48.755 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-02-10T08:48:49.222 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-02-10T08:48:49.447 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T08:48:49.867 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T08:48:50.228 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T08:48:50.228 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:48:50.228 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-02-10T08:48:50.228 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2024-02-10T08:48:50.229 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:48:50.229 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-02-10T08:48:50.440 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install rbd-nbd 2024-02-10T08:48:51.206 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:02:40 ago on Sat 10 Feb 2024 08:46:11 AM UTC. 2024-02-10T08:48:52.718 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-02-10T08:48:52.719 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-02-10T08:48:52.719 INFO:teuthology.orchestra.run.smithi155.stdout: Package Architecture Version Repository Size 2024-02-10T08:48:52.719 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-02-10T08:48:52.719 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-02-10T08:48:52.719 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2024-02-10T08:48:52.719 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:48:52.719 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-02-10T08:48:52.719 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-02-10T08:48:52.719 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-02-10T08:48:52.719 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:48:52.720 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 173 k 2024-02-10T08:48:52.720 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 649 k 2024-02-10T08:48:52.720 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-02-10T08:48:52.772 INFO:teuthology.orchestra.run.smithi155.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 3.3 MB/s | 173 kB 00:00 2024-02-10T08:48:52.772 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:48:52.772 INFO:teuthology.orchestra.run.smithi155.stdout:Total 3.2 MB/s | 173 kB 00:00 2024-02-10T08:48:52.773 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-02-10T08:48:52.783 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-02-10T08:48:52.783 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-02-10T08:48:52.834 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-02-10T08:48:52.837 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-02-10T08:48:52.946 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-02-10T08:48:53.186 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T08:48:53.518 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T08:48:53.978 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T08:48:53.979 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:48:53.979 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-02-10T08:48:53.979 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2024-02-10T08:48:53.979 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:48:53.979 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-02-10T08:48:54.205 DEBUG:teuthology.parallel:result is None 2024-02-10T08:52:10.247 INFO:teuthology.orchestra.run.smithi077.stdout:[MIRROR] python3-requests-oauthlib-1.0.0-1.el8.noarch.rpm: Curl error (56): Failure when receiving data from the peer for http://mirrors.cmich.edu/centos/8-stream/BaseOS/x86_64/os/Packages/python3-requests-oauthlib-1.0.0-1.el8.noarch.rpm [Recv failure: Connection reset by peer] 2024-02-10T08:52:10.596 INFO:teuthology.orchestra.run.smithi077.stdout:(13/13): python3-requests-oauthlib-1.0.0-1.el8. 175 B/s | 43 kB 04:09 2024-02-10T08:52:10.596 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:52:10.596 INFO:teuthology.orchestra.run.smithi077.stdout:Total 7.8 kB/s | 1.9 MB 04:09 2024-02-10T08:52:10.648 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2024-02-10T08:52:10.664 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2024-02-10T08:52:10.665 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2024-02-10T08:52:10.753 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2024-02-10T08:52:10.756 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2024-02-10T08:52:11.301 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2024-02-10T08:52:11.590 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-02-10T08:52:11.797 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-02-10T08:52:12.048 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-02-10T08:52:12.303 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-02-10T08:52:12.502 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-02-10T08:52:12.757 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-02-10T08:52:13.102 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-02-10T08:52:13.393 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-02-10T08:52:14.465 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-02-10T08:52:14.777 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-02-10T08:52:15.048 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-02-10T08:52:15.228 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-02-10T08:52:15.437 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-02-10T08:52:15.735 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-02-10T08:52:15.735 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2024-02-10T08:52:15.736 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-02-10T08:52:15.736 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-02-10T08:52:15.736 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-02-10T08:52:15.736 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-02-10T08:52:15.736 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-02-10T08:52:15.736 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-02-10T08:52:15.736 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-02-10T08:52:15.736 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-02-10T08:52:15.736 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-02-10T08:52:15.736 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-02-10T08:52:15.736 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-02-10T08:52:16.182 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-02-10T08:52:16.182 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:52:16.183 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2024-02-10T08:52:16.183 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2024-02-10T08:52:16.183 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-02-10T08:52:16.183 INFO:teuthology.orchestra.run.smithi077.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-02-10T08:52:16.183 INFO:teuthology.orchestra.run.smithi077.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-02-10T08:52:16.184 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-02-10T08:52:16.184 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-02-10T08:52:16.184 INFO:teuthology.orchestra.run.smithi077.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-02-10T08:52:16.184 INFO:teuthology.orchestra.run.smithi077.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-02-10T08:52:16.184 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-02-10T08:52:16.185 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-02-10T08:52:16.185 INFO:teuthology.orchestra.run.smithi077.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-02-10T08:52:16.185 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rsa-4.9-2.el8.noarch 2024-02-10T08:52:16.185 INFO:teuthology.orchestra.run.smithi077.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-02-10T08:52:16.185 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:52:16.185 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-02-10T08:52:16.300 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph-mgr-cephadm 2024-02-10T08:52:17.052 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:06:05 ago on Sat 10 Feb 2024 08:46:12 AM UTC. 2024-02-10T08:52:18.555 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-02-10T08:52:18.556 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-10T08:52:18.556 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repository Size 2024-02-10T08:52:18.556 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-10T08:52:18.556 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2024-02-10T08:52:18.556 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2024-02-10T08:52:18.556 INFO:teuthology.orchestra.run.smithi077.stdout:Installing dependencies: 2024-02-10T08:52:18.556 INFO:teuthology.orchestra.run.smithi077.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-02-10T08:52:18.556 INFO:teuthology.orchestra.run.smithi077.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-02-10T08:52:18.556 INFO:teuthology.orchestra.run.smithi077.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-02-10T08:52:18.556 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2024-02-10T08:52:18.556 INFO:teuthology.orchestra.run.smithi077.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-02-10T08:52:18.556 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:52:18.557 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2024-02-10T08:52:18.557 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-10T08:52:18.557 INFO:teuthology.orchestra.run.smithi077.stdout:Install 6 Packages 2024-02-10T08:52:18.557 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:52:18.557 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 5.8 M 2024-02-10T08:52:18.558 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 24 M 2024-02-10T08:52:18.558 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2024-02-10T08:52:19.111 INFO:teuthology.orchestra.run.smithi077.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 1.8 MB/s | 89 kB 00:00 2024-02-10T08:52:19.353 INFO:teuthology.orchestra.run.smithi077.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 88 kB/s | 21 kB 00:00 2024-02-10T08:52:19.487 INFO:teuthology.orchestra.run.smithi077.stdout:(3/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.2 MB/s | 538 kB 00:00 2024-02-10T08:52:19.579 INFO:teuthology.orchestra.run.smithi077.stdout:(4/6): python3-execnet-1.7.1-1.el8.noarch.rpm 1.5 MB/s | 355 kB 00:00 2024-02-10T08:52:19.739 INFO:teuthology.orchestra.run.smithi077.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 194 kB/s | 49 kB 00:00 2024-02-10T08:52:20.013 INFO:teuthology.orchestra.run.smithi077.stdout:(6/6): python3-babel-2.5.1-7.el8.noarch.rpm 5.0 MB/s | 4.8 MB 00:00 2024-02-10T08:52:20.013 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:52:20.013 INFO:teuthology.orchestra.run.smithi077.stdout:Total 4.0 MB/s | 5.8 MB 00:01 2024-02-10T08:52:20.066 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2024-02-10T08:52:20.076 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2024-02-10T08:52:20.076 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2024-02-10T08:52:20.181 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2024-02-10T08:52:20.184 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2024-02-10T08:52:20.470 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2024-02-10T08:52:20.760 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-02-10T08:52:21.047 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-02-10T08:52:21.991 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-02-10T08:52:22.406 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-02-10T08:52:22.699 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2024-02-10T08:52:22.961 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-02-10T08:52:23.255 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-02-10T08:52:23.255 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2024-02-10T08:52:23.255 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-02-10T08:52:23.256 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2024-02-10T08:52:23.256 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-02-10T08:52:23.256 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-02-10T08:52:23.643 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-02-10T08:52:23.643 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:52:23.643 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2024-02-10T08:52:23.643 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2024-02-10T08:52:23.644 INFO:teuthology.orchestra.run.smithi077.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2024-02-10T08:52:23.644 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2024-02-10T08:52:23.644 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:52:23.644 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-02-10T08:52:23.783 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph-fuse 2024-02-10T08:52:24.531 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:06:12 ago on Sat 10 Feb 2024 08:46:12 AM UTC. 2024-02-10T08:52:26.048 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-02-10T08:52:26.049 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-10T08:52:26.049 INFO:teuthology.orchestra.run.smithi077.stdout: Package Architecture Version Repository Size 2024-02-10T08:52:26.049 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-10T08:52:26.049 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2024-02-10T08:52:26.049 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2024-02-10T08:52:26.049 INFO:teuthology.orchestra.run.smithi077.stdout:Installing dependencies: 2024-02-10T08:52:26.049 INFO:teuthology.orchestra.run.smithi077.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-02-10T08:52:26.049 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:52:26.049 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2024-02-10T08:52:26.049 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-10T08:52:26.050 INFO:teuthology.orchestra.run.smithi077.stdout:Install 2 Packages 2024-02-10T08:52:26.050 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:52:26.050 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 968 k 2024-02-10T08:52:26.050 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 3.2 M 2024-02-10T08:52:26.050 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2024-02-10T08:52:26.146 INFO:teuthology.orchestra.run.smithi077.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 16 MB/s | 885 kB 00:00 2024-02-10T08:52:26.147 INFO:teuthology.orchestra.run.smithi077.stdout:[MIRROR] fuse-2.9.7-17.el8.x86_64.rpm: Status code: 503 for http://mirror.ette.biz/centos/8-stream/BaseOS/x86_64/os/Packages/fuse-2.9.7-17.el8.x86_64.rpm (IP: 38.105.93.109) 2024-02-10T08:56:32.572 INFO:teuthology.orchestra.run.smithi077.stdout:[MIRROR] fuse-2.9.7-17.el8.x86_64.rpm: Curl error (56): Failure when receiving data from the peer for http://mirrors.cmich.edu/centos/8-stream/BaseOS/x86_64/os/Packages/fuse-2.9.7-17.el8.x86_64.rpm [Recv failure: Connection reset by peer] 2024-02-10T08:56:32.950 INFO:teuthology.orchestra.run.smithi077.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 344 B/s | 83 kB 04:06 2024-02-10T08:56:32.950 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:56:32.950 INFO:teuthology.orchestra.run.smithi077.stdout:Total 3.9 kB/s | 968 kB 04:06 2024-02-10T08:56:32.961 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2024-02-10T08:56:32.983 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2024-02-10T08:56:32.983 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2024-02-10T08:56:33.077 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2024-02-10T08:56:33.080 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2024-02-10T08:56:33.297 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2024-02-10T08:56:33.554 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-02-10T08:56:33.755 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-02-10T08:56:34.150 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-02-10T08:56:34.150 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2024-02-10T08:56:34.556 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-02-10T08:56:34.556 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:56:34.556 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2024-02-10T08:56:34.556 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-02-10T08:56:34.556 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:56:34.556 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-02-10T08:56:34.741 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install librados-devel 2024-02-10T08:56:35.500 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:10:23 ago on Sat 10 Feb 2024 08:46:12 AM UTC. 2024-02-10T08:56:37.023 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-02-10T08:56:37.024 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-10T08:56:37.024 INFO:teuthology.orchestra.run.smithi077.stdout: Package Architecture Version Repository Size 2024-02-10T08:56:37.024 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-10T08:56:37.024 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2024-02-10T08:56:37.024 INFO:teuthology.orchestra.run.smithi077.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2024-02-10T08:56:37.024 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:56:37.024 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2024-02-10T08:56:37.024 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-10T08:56:37.024 INFO:teuthology.orchestra.run.smithi077.stdout:Install 1 Package 2024-02-10T08:56:37.024 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:56:37.025 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 131 k 2024-02-10T08:56:37.025 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 439 k 2024-02-10T08:56:37.025 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2024-02-10T08:56:37.087 INFO:teuthology.orchestra.run.smithi077.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 2.1 MB/s | 131 kB 00:00 2024-02-10T08:56:37.088 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:56:37.088 INFO:teuthology.orchestra.run.smithi077.stdout:Total 2.0 MB/s | 131 kB 00:00 2024-02-10T08:56:37.088 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2024-02-10T08:56:37.093 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2024-02-10T08:56:37.093 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2024-02-10T08:56:37.143 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2024-02-10T08:56:37.146 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2024-02-10T08:56:37.254 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2024-02-10T08:56:37.504 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T08:56:37.777 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T08:56:38.180 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T08:56:38.180 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:56:38.180 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2024-02-10T08:56:38.181 INFO:teuthology.orchestra.run.smithi077.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2024-02-10T08:56:38.181 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:56:38.181 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-02-10T08:56:38.409 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install libcephfs2 2024-02-10T08:56:39.175 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:10:27 ago on Sat 10 Feb 2024 08:46:12 AM UTC. 2024-02-10T08:56:40.642 INFO:teuthology.orchestra.run.smithi077.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-10T08:56:40.688 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-02-10T08:56:40.689 INFO:teuthology.orchestra.run.smithi077.stdout:Nothing to do. 2024-02-10T08:56:40.689 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-02-10T08:56:40.815 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install libcephfs-devel 2024-02-10T08:56:41.568 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:10:29 ago on Sat 10 Feb 2024 08:46:12 AM UTC. 2024-02-10T08:56:43.050 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-02-10T08:56:43.051 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-10T08:56:43.051 INFO:teuthology.orchestra.run.smithi077.stdout: Package Architecture Version Repository Size 2024-02-10T08:56:43.051 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-10T08:56:43.052 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2024-02-10T08:56:43.052 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2024-02-10T08:56:43.052 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:56:43.052 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2024-02-10T08:56:43.052 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-10T08:56:43.052 INFO:teuthology.orchestra.run.smithi077.stdout:Install 1 Package 2024-02-10T08:56:43.052 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:56:43.052 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 24 k 2024-02-10T08:56:43.053 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 94 k 2024-02-10T08:56:43.053 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2024-02-10T08:56:43.140 INFO:teuthology.orchestra.run.smithi077.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 276 kB/s | 24 kB 00:00 2024-02-10T08:56:43.140 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:56:43.141 INFO:teuthology.orchestra.run.smithi077.stdout:Total 272 kB/s | 24 kB 00:00 2024-02-10T08:56:43.141 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2024-02-10T08:56:43.144 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2024-02-10T08:56:43.144 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2024-02-10T08:56:43.147 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2024-02-10T08:56:43.150 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2024-02-10T08:56:43.196 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2024-02-10T08:56:43.427 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T08:56:43.717 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T08:56:44.063 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T08:56:44.063 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:56:44.063 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2024-02-10T08:56:44.063 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2024-02-10T08:56:44.063 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:56:44.063 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-02-10T08:56:44.181 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install librados2 2024-02-10T08:56:44.932 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:10:32 ago on Sat 10 Feb 2024 08:46:12 AM UTC. 2024-02-10T08:56:46.391 INFO:teuthology.orchestra.run.smithi077.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-10T08:56:46.437 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-02-10T08:56:46.438 INFO:teuthology.orchestra.run.smithi077.stdout:Nothing to do. 2024-02-10T08:56:46.438 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-02-10T08:56:46.539 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install librbd1 2024-02-10T08:56:47.289 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:10:35 ago on Sat 10 Feb 2024 08:46:12 AM UTC. 2024-02-10T08:56:48.738 INFO:teuthology.orchestra.run.smithi077.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-10T08:56:48.783 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-02-10T08:56:48.784 INFO:teuthology.orchestra.run.smithi077.stdout:Nothing to do. 2024-02-10T08:56:48.784 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-02-10T08:56:48.899 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install python3-rados 2024-02-10T08:56:49.645 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:10:37 ago on Sat 10 Feb 2024 08:46:12 AM UTC. 2024-02-10T08:56:51.088 INFO:teuthology.orchestra.run.smithi077.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-10T08:56:51.134 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-02-10T08:56:51.135 INFO:teuthology.orchestra.run.smithi077.stdout:Nothing to do. 2024-02-10T08:56:51.135 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-02-10T08:56:51.240 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install python3-rgw 2024-02-10T08:56:51.989 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:10:39 ago on Sat 10 Feb 2024 08:46:12 AM UTC. 2024-02-10T08:56:53.453 INFO:teuthology.orchestra.run.smithi077.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-10T08:56:53.498 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-02-10T08:56:53.499 INFO:teuthology.orchestra.run.smithi077.stdout:Nothing to do. 2024-02-10T08:56:53.500 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-02-10T08:56:53.606 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install python3-cephfs 2024-02-10T08:56:54.381 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:10:42 ago on Sat 10 Feb 2024 08:46:12 AM UTC. 2024-02-10T08:56:55.829 INFO:teuthology.orchestra.run.smithi077.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-10T08:56:55.876 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-02-10T08:56:55.877 INFO:teuthology.orchestra.run.smithi077.stdout:Nothing to do. 2024-02-10T08:56:55.877 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-02-10T08:56:55.994 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install python3-rbd 2024-02-10T08:56:56.740 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:10:44 ago on Sat 10 Feb 2024 08:46:12 AM UTC. 2024-02-10T08:56:58.225 INFO:teuthology.orchestra.run.smithi077.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-10T08:56:58.270 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-02-10T08:56:58.271 INFO:teuthology.orchestra.run.smithi077.stdout:Nothing to do. 2024-02-10T08:56:58.271 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-02-10T08:56:58.382 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install rbd-fuse 2024-02-10T08:56:59.132 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:10:47 ago on Sat 10 Feb 2024 08:46:12 AM UTC. 2024-02-10T08:57:00.629 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-02-10T08:57:00.629 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-10T08:57:00.629 INFO:teuthology.orchestra.run.smithi077.stdout: Package Architecture Version Repository Size 2024-02-10T08:57:00.629 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-10T08:57:00.629 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2024-02-10T08:57:00.629 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2024-02-10T08:57:00.630 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:57:00.630 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2024-02-10T08:57:00.630 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-10T08:57:00.630 INFO:teuthology.orchestra.run.smithi077.stdout:Install 1 Package 2024-02-10T08:57:00.630 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:57:00.630 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 89 k 2024-02-10T08:57:00.630 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 298 k 2024-02-10T08:57:00.631 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2024-02-10T08:57:00.675 INFO:teuthology.orchestra.run.smithi077.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 1.9 MB/s | 89 kB 00:00 2024-02-10T08:57:00.676 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:57:00.676 INFO:teuthology.orchestra.run.smithi077.stdout:Total 1.9 MB/s | 89 kB 00:00 2024-02-10T08:57:00.676 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2024-02-10T08:57:00.686 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2024-02-10T08:57:00.686 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2024-02-10T08:57:00.734 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2024-02-10T08:57:00.737 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2024-02-10T08:57:00.846 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2024-02-10T08:57:01.084 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T08:57:01.356 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T08:57:01.767 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T08:57:01.779 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:57:01.779 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2024-02-10T08:57:01.780 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2024-02-10T08:57:01.780 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:57:01.780 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-02-10T08:57:02.015 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install rbd-mirror 2024-02-10T08:57:02.770 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:10:50 ago on Sat 10 Feb 2024 08:46:12 AM UTC. 2024-02-10T08:57:04.270 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-02-10T08:57:04.270 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-10T08:57:04.270 INFO:teuthology.orchestra.run.smithi077.stdout: Package Architecture Version Repository Size 2024-02-10T08:57:04.270 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-10T08:57:04.271 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2024-02-10T08:57:04.271 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-02-10T08:57:04.271 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:57:04.271 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2024-02-10T08:57:04.271 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-10T08:57:04.271 INFO:teuthology.orchestra.run.smithi077.stdout:Install 1 Package 2024-02-10T08:57:04.271 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:57:04.271 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 3.7 M 2024-02-10T08:57:04.272 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 18 M 2024-02-10T08:57:04.272 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2024-02-10T08:57:04.353 INFO:teuthology.orchestra.run.smithi077.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 46 MB/s | 3.7 MB 00:00 2024-02-10T08:57:04.354 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:57:04.354 INFO:teuthology.orchestra.run.smithi077.stdout:Total 45 MB/s | 3.7 MB 00:00 2024-02-10T08:57:04.354 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2024-02-10T08:57:04.366 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2024-02-10T08:57:04.366 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2024-02-10T08:57:04.429 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2024-02-10T08:57:04.432 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2024-02-10T08:57:04.922 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2024-02-10T08:57:05.126 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T08:57:05.542 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T08:57:05.950 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T08:57:05.950 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:57:05.950 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2024-02-10T08:57:05.950 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2024-02-10T08:57:05.950 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:57:05.950 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-02-10T08:57:06.187 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install rbd-nbd 2024-02-10T08:57:06.946 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:10:54 ago on Sat 10 Feb 2024 08:46:12 AM UTC. 2024-02-10T08:57:08.449 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-02-10T08:57:08.449 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-10T08:57:08.449 INFO:teuthology.orchestra.run.smithi077.stdout: Package Architecture Version Repository Size 2024-02-10T08:57:08.450 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-10T08:57:08.450 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2024-02-10T08:57:08.450 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2024-02-10T08:57:08.450 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:57:08.450 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2024-02-10T08:57:08.450 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-10T08:57:08.450 INFO:teuthology.orchestra.run.smithi077.stdout:Install 1 Package 2024-02-10T08:57:08.450 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:57:08.450 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 173 k 2024-02-10T08:57:08.450 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 649 k 2024-02-10T08:57:08.450 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2024-02-10T08:57:08.498 INFO:teuthology.orchestra.run.smithi077.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 3.6 MB/s | 173 kB 00:00 2024-02-10T08:57:08.499 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2024-02-10T08:57:08.499 INFO:teuthology.orchestra.run.smithi077.stdout:Total 3.5 MB/s | 173 kB 00:00 2024-02-10T08:57:08.499 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2024-02-10T08:57:08.509 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2024-02-10T08:57:08.509 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2024-02-10T08:57:08.560 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2024-02-10T08:57:08.563 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2024-02-10T08:57:08.689 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2024-02-10T08:57:08.883 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T08:57:09.206 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T08:57:09.642 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-10T08:57:09.642 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:57:09.642 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2024-02-10T08:57:09.643 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2024-02-10T08:57:09.643 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-10T08:57:09.643 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-02-10T08:57:09.901 DEBUG:teuthology.parallel:result is None 2024-02-10T08:57:09.902 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-10T08:57:09.902 INFO:teuthology.packaging:ref: None 2024-02-10T08:57:09.902 INFO:teuthology.packaging:tag: v16.2.4 2024-02-10T08:57:09.902 INFO:teuthology.packaging:branch: None 2024-02-10T08:57:09.902 INFO:teuthology.packaging:sha1: 0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-10T08:57:09.903 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-10T08:57:10.057 DEBUG:teuthology.orchestra.run.smithi077:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-10T08:57:10.079 INFO:teuthology.orchestra.run.smithi077.stdout:16.2.4-0.el8 2024-02-10T08:57:10.080 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2024-02-10T08:57:10.080 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2024-02-10T08:57:10.081 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-10T08:57:10.082 INFO:teuthology.packaging:ref: None 2024-02-10T08:57:10.082 INFO:teuthology.packaging:tag: v16.2.4 2024-02-10T08:57:10.082 INFO:teuthology.packaging:branch: None 2024-02-10T08:57:10.082 INFO:teuthology.packaging:sha1: 0e714d9a4bd2a821113e6318adb87bd06cf81ec1 2024-02-10T08:57:10.082 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-10T08:57:10.238 DEBUG:teuthology.orchestra.run.smithi155:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-10T08:57:10.269 INFO:teuthology.orchestra.run.smithi155.stdout:16.2.4-0.el8 2024-02-10T08:57:10.269 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2024-02-10T08:57:10.269 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2024-02-10T08:57:10.271 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-02-10T08:57:10.271 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-02-10T08:57:10.271 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-10T08:57:10.298 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-02-10T08:57:10.298 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-10T08:57:10.342 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-02-10T08:57:10.343 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-02-10T08:57:10.343 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/usr/bin/daemon-helper 2024-02-10T08:57:10.369 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-10T08:57:10.435 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-02-10T08:57:10.435 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/usr/bin/daemon-helper 2024-02-10T08:57:10.465 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-10T08:57:10.535 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-02-10T08:57:10.535 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-02-10T08:57:10.535 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-10T08:57:10.561 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-10T08:57:10.628 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-02-10T08:57:10.628 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-10T08:57:10.657 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-10T08:57:10.725 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-02-10T08:57:10.726 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-02-10T08:57:10.726 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/usr/bin/stdin-killer 2024-02-10T08:57:10.752 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-10T08:57:10.819 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-02-10T08:57:10.819 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/usr/bin/stdin-killer 2024-02-10T08:57:10.847 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-10T08:57:10.915 INFO:teuthology.run_tasks:Running task print... 2024-02-10T08:57:10.926 INFO:teuthology.task.print:**** done install task... 2024-02-10T08:57:10.926 INFO:teuthology.run_tasks:Running task cephadm... 2024-02-10T08:57:11.018 INFO:tasks.cephadm:Config: {'avoid_pacific_features': True, 'cephadm_branch': 'v16.2.4', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'docker.io/ceph/ceph:v16.2.4', 'roleless': True, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(PG_DEGRADED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '0e714d9a4bd2a821113e6318adb87bd06cf81ec1'} 2024-02-10T08:57:11.018 INFO:tasks.cephadm:Cluster image is docker.io/ceph/ceph:v16.2.4 2024-02-10T08:57:11.019 INFO:tasks.cephadm:Cluster fsid is 6081b54a-c7f2-11ee-95b8-87774f69a715 2024-02-10T08:57:11.019 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-02-10T08:57:11.019 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-02-10T08:57:11.019 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi077': '172.21.15.77', 'mon.smithi155': '172.21.15.155'} 2024-02-10T08:57:11.019 INFO:tasks.cephadm:Normalizing hostnames... 2024-02-10T08:57:11.019 DEBUG:teuthology.orchestra.run.smithi077:> sudo hostname $(hostname -s) 2024-02-10T08:57:11.047 DEBUG:teuthology.orchestra.run.smithi155:> sudo hostname $(hostname -s) 2024-02-10T08:57:11.074 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v16.2.4)... 2024-02-10T08:57:11.074 DEBUG:teuthology.orchestra.run.smithi077:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-10T08:57:11.281 INFO:teuthology.orchestra.run.smithi077.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Feb 10 08:57 /home/ubuntu/cephtest/cephadm 2024-02-10T08:57:11.282 DEBUG:teuthology.orchestra.run.smithi155:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-10T08:57:11.479 INFO:teuthology.orchestra.run.smithi155.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Feb 10 08:57 /home/ubuntu/cephtest/cephadm 2024-02-10T08:57:11.479 DEBUG:teuthology.orchestra.run.smithi077:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-10T08:57:11.498 DEBUG:teuthology.orchestra.run.smithi155:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-10T08:57:11.527 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /etc/ceph 2024-02-10T08:57:11.565 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /etc/ceph 2024-02-10T08:57:11.592 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 777 /etc/ceph 2024-02-10T08:57:11.631 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod 777 /etc/ceph 2024-02-10T08:57:11.658 INFO:tasks.cephadm:Writing seed config... 2024-02-10T08:57:11.659 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-02-10T08:57:11.659 INFO:tasks.cephadm: override: [client] debug client = 20 2024-02-10T08:57:11.659 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-02-10T08:57:11.660 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-02-10T08:57:11.660 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-02-10T08:57:11.660 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-02-10T08:57:11.660 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-02-10T08:57:11.660 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-02-10T08:57:11.660 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-02-10T08:57:11.660 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-02-10T08:57:11.660 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-02-10T08:57:11.660 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-02-10T08:57:11.660 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-02-10T08:57:11.660 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-02-10T08:57:11.660 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-02-10T08:57:11.660 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-02-10T08:57:11.660 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-02-10T08:57:11.660 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-02-10T08:57:11.661 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-02-10T08:57:11.661 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-02-10T08:57:11.661 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-02-10T08:57:11.661 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-02-10T08:57:11.661 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-02-10T08:57:11.661 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-02-10T08:57:11.661 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-02-10T08:57:11.661 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-02-10T08:57:11.661 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-02-10T08:57:11.661 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-02-10T08:57:11.661 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-02-10T08:57:11.661 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-02-10T08:57:11.661 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-02-10T08:57:11.661 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-02-10T08:57:11.661 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-02-10T08:57:11.661 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-02-10T08:57:11.662 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-02-10T08:57:11.662 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-02-10T08:57:11.662 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-02-10T08:57:11.662 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-02-10T08:57:11.662 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-02-10T08:57:11.662 DEBUG:teuthology.orchestra.run.smithi077:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-02-10T08:57:11.689 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 6081b54a-c7f2-11ee-95b8-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-02-10T08:57:11.689 DEBUG:teuthology.orchestra.run.smithi077:mon.smithi077> sudo journalctl -f -n 0 -u ceph-6081b54a-c7f2-11ee-95b8-87774f69a715@mon.smithi077.service 2024-02-10T08:57:11.732 INFO:tasks.cephadm:Bootstrapping... 2024-02-10T08:57:11.732 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 -v bootstrap --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.77 && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-02-10T08:57:11.758 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:-- Logs begin at Sat 2024-02-10 08:38:58 UTC. -- 2024-02-10T08:57:12.001 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-02-10T08:57:12.999 INFO:teuthology.orchestra.run.smithi077.stderr:/bin/podman: 4.8.3 2024-02-10T08:57:13.017 INFO:teuthology.orchestra.run.smithi077.stderr:Verifying podman|docker is present... 2024-02-10T08:57:13.017 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-02-10T08:57:13.040 INFO:teuthology.orchestra.run.smithi077.stderr:/bin/podman: 4.8.3 2024-02-10T08:57:13.051 INFO:teuthology.orchestra.run.smithi077.stderr:Verifying lvm2 is present... 2024-02-10T08:57:13.052 INFO:teuthology.orchestra.run.smithi077.stderr:Verifying time synchronization is in place... 2024-02-10T08:57:13.052 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: systemctl is-enabled chrony.service 2024-02-10T08:57:13.059 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-02-10T08:57:13.059 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: systemctl is-active chrony.service 2024-02-10T08:57:13.065 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: inactive 2024-02-10T08:57:13.066 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: systemctl is-enabled chronyd.service 2024-02-10T08:57:13.072 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: enabled 2024-02-10T08:57:13.072 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: systemctl is-active chronyd.service 2024-02-10T08:57:13.078 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: active 2024-02-10T08:57:13.079 INFO:teuthology.orchestra.run.smithi077.stderr:Unit chronyd.service is enabled and running 2024-02-10T08:57:13.079 INFO:teuthology.orchestra.run.smithi077.stderr:Repeating the final host check... 2024-02-10T08:57:13.079 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-02-10T08:57:13.099 INFO:teuthology.orchestra.run.smithi077.stderr:/bin/podman: 4.8.3 2024-02-10T08:57:13.111 INFO:teuthology.orchestra.run.smithi077.stderr:podman|docker (/bin/podman) is present 2024-02-10T08:57:13.111 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl is present 2024-02-10T08:57:13.111 INFO:teuthology.orchestra.run.smithi077.stderr:lvcreate is present 2024-02-10T08:57:13.111 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: systemctl is-enabled chrony.service 2024-02-10T08:57:13.116 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-02-10T08:57:13.117 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: systemctl is-active chrony.service 2024-02-10T08:57:13.122 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: inactive 2024-02-10T08:57:13.123 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: systemctl is-enabled chronyd.service 2024-02-10T08:57:13.128 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: enabled 2024-02-10T08:57:13.129 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: systemctl is-active chronyd.service 2024-02-10T08:57:13.134 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: active 2024-02-10T08:57:13.135 INFO:teuthology.orchestra.run.smithi077.stderr:Unit chronyd.service is enabled and running 2024-02-10T08:57:13.135 INFO:teuthology.orchestra.run.smithi077.stderr:Host looks OK 2024-02-10T08:57:13.135 INFO:teuthology.orchestra.run.smithi077.stderr:Cluster fsid: 6081b54a-c7f2-11ee-95b8-87774f69a715 2024-02-10T08:57:13.135 INFO:teuthology.orchestra.run.smithi077.stderr:Acquiring lock 139803068135912 on /run/cephadm/6081b54a-c7f2-11ee-95b8-87774f69a715.lock 2024-02-10T08:57:13.136 INFO:teuthology.orchestra.run.smithi077.stderr:Lock 139803068135912 acquired on /run/cephadm/6081b54a-c7f2-11ee-95b8-87774f69a715.lock 2024-02-10T08:57:13.136 INFO:teuthology.orchestra.run.smithi077.stderr:Verifying IP 172.21.15.77 port 3300 ... 2024-02-10T08:57:13.136 INFO:teuthology.orchestra.run.smithi077.stderr:Verifying IP 172.21.15.77 port 6789 ... 2024-02-10T08:57:13.136 INFO:teuthology.orchestra.run.smithi077.stderr:Base mon IP is 172.21.15.77, final addrv is [v2:172.21.15.77:3300,v1:172.21.15.77:6789] 2024-02-10T08:57:13.136 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /sbin/ip route ls 2024-02-10T08:57:13.138 INFO:teuthology.orchestra.run.smithi077.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.77 metric 100 2024-02-10T08:57:13.138 INFO:teuthology.orchestra.run.smithi077.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.77 metric 100 2024-02-10T08:57:13.139 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /sbin/ip -6 route ls 2024-02-10T08:57:13.145 INFO:teuthology.orchestra.run.smithi077.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-02-10T08:57:13.145 INFO:teuthology.orchestra.run.smithi077.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-02-10T08:57:13.145 INFO:teuthology.orchestra.run.smithi077.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1778sec hoplimit 64 pref medium 2024-02-10T08:57:13.145 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /sbin/ip -6 addr ls 2024-02-10T08:57:13.145 INFO:teuthology.orchestra.run.smithi077.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-02-10T08:57:13.146 INFO:teuthology.orchestra.run.smithi077.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-02-10T08:57:13.146 INFO:teuthology.orchestra.run.smithi077.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-02-10T08:57:13.146 INFO:teuthology.orchestra.run.smithi077.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-02-10T08:57:13.146 INFO:teuthology.orchestra.run.smithi077.stderr:/sbin/ip: inet6 fe80::ec4:7aff:fe8f:d3d3/64 scope link 2024-02-10T08:57:13.146 INFO:teuthology.orchestra.run.smithi077.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-02-10T08:57:13.147 INFO:teuthology.orchestra.run.smithi077.stderr:Mon IP 172.21.15.77 is in CIDR network 172.21.0.0/20 2024-02-10T08:57:13.147 INFO:teuthology.orchestra.run.smithi077.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-02-10T08:57:13.147 INFO:teuthology.orchestra.run.smithi077.stderr:Pulling container image docker.io/ceph/ceph:v16.2.4... 2024-02-10T08:57:13.147 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /bin/podman pull docker.io/ceph/ceph:v16.2.4 2024-02-10T08:57:13.244 INFO:teuthology.orchestra.run.smithi077.stderr:/bin/podman: Trying to pull docker.io/ceph/ceph:v16.2.4... 2024-02-10T08:57:19.917 INFO:teuthology.orchestra.run.smithi077.stderr:/bin/podman: Getting image source signatures 2024-02-10T08:57:19.930 INFO:teuthology.orchestra.run.smithi077.stderr:/bin/podman: Copying blob sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621 2024-02-10T08:57:19.930 INFO:teuthology.orchestra.run.smithi077.stderr:/bin/podman: Copying blob sha256:551101690791055467b1840be133b2e6b029d263771bc6c46a96bb7acdacac39 2024-02-10T08:57:31.964 INFO:teuthology.orchestra.run.smithi077.stderr:/bin/podman: Copying config sha256:8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2024-02-10T08:57:32.131 INFO:teuthology.orchestra.run.smithi077.stderr:/bin/podman: Writing manifest to image destination 2024-02-10T08:57:32.433 INFO:teuthology.orchestra.run.smithi077.stderr:/bin/podman: 8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2024-02-10T08:57:32.444 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --version 2024-02-10T08:57:33.057 INFO:teuthology.orchestra.run.smithi077.stderr:ceph: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2024-02-10T08:57:34.084 INFO:teuthology.orchestra.run.smithi077.stderr:Ceph version: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2024-02-10T08:57:34.084 INFO:teuthology.orchestra.run.smithi077.stderr:Extracting ceph user uid/gid from container image... 2024-02-10T08:57:34.084 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 -c %u %g /var/lib/ceph 2024-02-10T08:57:34.710 INFO:teuthology.orchestra.run.smithi077.stderr:stat: 167 167 2024-02-10T08:57:35.367 INFO:teuthology.orchestra.run.smithi077.stderr:Creating initial keys... 2024-02-10T08:57:35.367 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-02-10T08:57:35.907 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-authtool: AQB/OsdlCcTTMBAA+wG5ONKaCTjdiYhirJ0/6w== 2024-02-10T08:57:36.674 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-02-10T08:57:37.205 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-authtool: AQCBOsdlEmWJBhAAyodY01Ng+jEpReZqJ2R81w== 2024-02-10T08:57:37.855 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-02-10T08:57:38.419 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-authtool: AQCCOsdl0JoYERAAH66KNOEjVcSsQZaTMkIwWQ== 2024-02-10T08:57:38.976 INFO:teuthology.orchestra.run.smithi077.stderr:Creating initial monmap... 2024-02-10T08:57:38.976 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/monmaptool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 -v /tmp/ceph-tmpkka2p5j7:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --create --clobber --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 --addv smithi077 [v2:172.21.15.77:3300,v1:172.21.15.77:6789] /tmp/monmap 2024-02-10T08:57:39.542 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-10T08:57:39.542 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 6081b54a-c7f2-11ee-95b8-87774f69a715 2024-02-10T08:57:39.542 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-10T08:57:40.296 INFO:teuthology.orchestra.run.smithi077.stderr:monmaptool for smithi077 [v2:172.21.15.77:3300,v1:172.21.15.77:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-10T08:57:40.296 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/monmaptool: set fsid to 6081b54a-c7f2-11ee-95b8-87774f69a715 2024-02-10T08:57:40.296 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-10T08:57:40.297 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-02-10T08:57:40.297 INFO:teuthology.orchestra.run.smithi077.stderr:Creating mon... 2024-02-10T08:57:40.297 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-mon --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/mon.smithi077:/var/lib/ceph/mon/ceph-smithi077:z -v /tmp/ceph-tmpiqr9vov7:/tmp/keyring:z -v /tmp/ceph-tmpkka2p5j7:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --mkfs -i smithi077 --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -c /dev/null --monmap /tmp/monmap --keyring /tmp/keyring --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true 2024-02-10T08:57:40.933 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.809+0000 7f8e4659c700 0 set uid:gid to 167:167 (ceph:ceph) 2024-02-10T08:57:40.933 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.809+0000 7f8e4659c700 1 imported monmap: 2024-02-10T08:57:40.933 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: epoch 0 2024-02-10T08:57:40.933 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 2024-02-10T08:57:40.934 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: last_changed 2024-02-10T08:57:39.474770+0000 2024-02-10T08:57:40.934 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: created 2024-02-10T08:57:39.474770+0000 2024-02-10T08:57:40.934 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2024-02-10T08:57:40.934 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: election_strategy: 1 2024-02-10T08:57:40.934 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon.smithi077 2024-02-10T08:57:40.934 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: 2024-02-10T08:57:40.934 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.809+0000 7f8e4659c700 0 /usr/bin/ceph-mon: set fsid to 6081b54a-c7f2-11ee-95b8-87774f69a715 2024-02-10T08:57:40.934 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: RocksDB version: 6.8.1 2024-02-10T08:57:40.934 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: 2024-02-10T08:57:40.934 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-10T08:57:40.934 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Compile date May 13 2021 2024-02-10T08:57:40.934 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: DB SUMMARY 2024-02-10T08:57:40.934 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: 2024-02-10T08:57:40.934 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi077/store.db dir, Total Num: 0, files: 2024-02-10T08:57:40.935 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: 2024-02-10T08:57:40.935 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi077/store.db: 2024-02-10T08:57:40.935 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: 2024-02-10T08:57:40.935 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.error_if_exists: 0 2024-02-10T08:57:40.935 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.create_if_missing: 1 2024-02-10T08:57:40.935 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.paranoid_checks: 1 2024-02-10T08:57:40.935 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.env: 0x55dc9dab91c0 2024-02-10T08:57:40.935 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.fs: Posix File System 2024-02-10T08:57:40.935 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.info_log: 0x55dc9eaacec0 2024-02-10T08:57:40.935 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.max_file_opening_threads: 16 2024-02-10T08:57:40.935 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.statistics: (nil) 2024-02-10T08:57:40.935 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.use_fsync: 0 2024-02-10T08:57:40.935 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.max_log_file_size: 0 2024-02-10T08:57:40.935 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-10T08:57:40.936 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.log_file_time_to_roll: 0 2024-02-10T08:57:40.936 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.keep_log_file_num: 1000 2024-02-10T08:57:40.936 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.recycle_log_file_num: 0 2024-02-10T08:57:40.936 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.allow_fallocate: 1 2024-02-10T08:57:40.936 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.allow_mmap_reads: 0 2024-02-10T08:57:40.936 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.allow_mmap_writes: 0 2024-02-10T08:57:40.936 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.use_direct_reads: 0 2024-02-10T08:57:40.936 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-10T08:57:40.936 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.create_missing_column_families: 0 2024-02-10T08:57:40.936 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.db_log_dir: 2024-02-10T08:57:40.936 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi077/store.db 2024-02-10T08:57:40.936 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.table_cache_numshardbits: 6 2024-02-10T08:57:40.936 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.max_subcompactions: 1 2024-02-10T08:57:40.937 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.max_background_flushes: -1 2024-02-10T08:57:40.937 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-02-10T08:57:40.937 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-02-10T08:57:40.937 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-10T08:57:40.937 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-10T08:57:40.937 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-02-10T08:57:40.937 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.advise_random_on_open: 1 2024-02-10T08:57:40.937 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.db_write_buffer_size: 0 2024-02-10T08:57:40.937 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.write_buffer_manager: 0x55dc9eab3530 2024-02-10T08:57:40.937 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-10T08:57:40.937 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-10T08:57:40.937 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-10T08:57:40.937 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.use_adaptive_mutex: 0 2024-02-10T08:57:40.937 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.rate_limiter: (nil) 2024-02-10T08:57:40.938 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-10T08:57:40.938 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.wal_recovery_mode: 2 2024-02-10T08:57:40.938 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.enable_thread_tracking: 0 2024-02-10T08:57:40.938 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.enable_pipelined_write: 0 2024-02-10T08:57:40.938 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.unordered_write: 0 2024-02-10T08:57:40.938 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-10T08:57:40.938 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-10T08:57:40.938 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-10T08:57:40.938 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-10T08:57:40.938 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.row_cache: None 2024-02-10T08:57:40.938 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.wal_filter: None 2024-02-10T08:57:40.938 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-10T08:57:40.938 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.allow_ingest_behind: 0 2024-02-10T08:57:40.938 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.preserve_deletes: 0 2024-02-10T08:57:40.939 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.two_write_queues: 0 2024-02-10T08:57:40.939 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.manual_wal_flush: 0 2024-02-10T08:57:40.939 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.atomic_flush: 0 2024-02-10T08:57:40.939 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-10T08:57:40.939 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.persist_stats_to_disk: 0 2024-02-10T08:57:40.939 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-02-10T08:57:40.939 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.log_readahead_size: 0 2024-02-10T08:57:40.939 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-02-10T08:57:40.939 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.max_background_jobs: 2 2024-02-10T08:57:40.939 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.max_background_compactions: -1 2024-02-10T08:57:40.939 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-10T08:57:40.939 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-10T08:57:40.939 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.delayed_write_rate : 16777216 2024-02-10T08:57:40.939 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.max_total_wal_size: 0 2024-02-10T08:57:40.940 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-10T08:57:40.940 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.stats_dump_period_sec: 600 2024-02-10T08:57:40.940 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.stats_persist_period_sec: 600 2024-02-10T08:57:40.940 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-10T08:57:40.940 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.max_open_files: -1 2024-02-10T08:57:40.940 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.bytes_per_sync: 0 2024-02-10T08:57:40.940 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-02-10T08:57:40.940 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-02-10T08:57:40.940 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Options.compaction_readahead_size: 0 2024-02-10T08:57:40.940 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Compression algorithms supported: 2024-02-10T08:57:40.940 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-10T08:57:40.940 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: kZSTD supported: 0 2024-02-10T08:57:40.940 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: kXpressCompression supported: 0 2024-02-10T08:57:40.940 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: kLZ4HCCompression supported: 1 2024-02-10T08:57:40.941 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: kLZ4Compression supported: 1 2024-02-10T08:57:40.941 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: kBZip2Compression supported: 0 2024-02-10T08:57:40.941 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: kZlibCompression supported: 1 2024-02-10T08:57:40.941 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: kSnappyCompression supported: 1 2024-02-10T08:57:40.941 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-02-10T08:57:40.941 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.810+0000 7f8e4659c700 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2024-02-10T08:57:40.941 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: 2024-02-10T08:57:40.941 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi077/store.db/MANIFEST-000001 2024-02-10T08:57:40.941 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: 2024-02-10T08:57:40.941 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-02-10T08:57:40.941 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: 2024-02-10T08:57:40.941 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-10T08:57:40.941 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.merge_operator: 2024-02-10T08:57:40.941 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.compaction_filter: None 2024-02-10T08:57:40.941 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.compaction_filter_factory: None 2024-02-10T08:57:40.942 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-02-10T08:57:40.942 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.table_factory: BlockBasedTable 2024-02-10T08:57:40.942 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55dc9e9c5cb8) 2024-02-10T08:57:40.942 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2024-02-10T08:57:40.942 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-10T08:57:40.942 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-10T08:57:40.942 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2024-02-10T08:57:40.942 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: index_type: 0 2024-02-10T08:57:40.942 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2024-02-10T08:57:40.942 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: index_shortening: 1 2024-02-10T08:57:40.942 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2024-02-10T08:57:40.942 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2024-02-10T08:57:40.942 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: checksum: 1 2024-02-10T08:57:40.942 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2024-02-10T08:57:40.942 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: block_cache: 0x55dc9e9fad10 2024-02-10T08:57:40.942 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2024-02-10T08:57:40.943 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: block_cache_options: 2024-02-10T08:57:40.943 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: capacity : 536870912 2024-02-10T08:57:40.943 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2024-02-10T08:57:40.943 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2024-02-10T08:57:40.943 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2024-02-10T08:57:40.943 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2024-02-10T08:57:40.943 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2024-02-10T08:57:40.943 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: block_size: 4096 2024-02-10T08:57:40.943 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2024-02-10T08:57:40.943 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2024-02-10T08:57:40.943 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2024-02-10T08:57:40.943 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2024-02-10T08:57:40.943 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: partition_filters: 0 2024-02-10T08:57:40.943 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2024-02-10T08:57:40.943 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-10T08:57:40.943 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2024-02-10T08:57:40.943 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: verify_compression: 0 2024-02-10T08:57:40.944 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2024-02-10T08:57:40.944 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: format_version: 2 2024-02-10T08:57:40.944 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2024-02-10T08:57:40.944 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: block_align: 0 2024-02-10T08:57:40.944 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: 2024-02-10T08:57:40.944 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.write_buffer_size: 33554432 2024-02-10T08:57:40.944 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.max_write_buffer_number: 2 2024-02-10T08:57:40.944 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.compression: NoCompression 2024-02-10T08:57:40.944 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.bottommost_compression: Disabled 2024-02-10T08:57:40.944 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.prefix_extractor: nullptr 2024-02-10T08:57:40.944 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-10T08:57:40.944 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.num_levels: 7 2024-02-10T08:57:40.944 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-10T08:57:40.944 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-10T08:57:40.945 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-10T08:57:40.945 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-10T08:57:40.945 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-10T08:57:40.945 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-10T08:57:40.945 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-10T08:57:40.945 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-10T08:57:40.945 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-10T08:57:40.945 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.compression_opts.window_bits: -14 2024-02-10T08:57:40.945 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.compression_opts.level: 32767 2024-02-10T08:57:40.945 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.compression_opts.strategy: 0 2024-02-10T08:57:40.945 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-10T08:57:40.945 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-10T08:57:40.945 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.compression_opts.enabled: false 2024-02-10T08:57:40.945 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-10T08:57:40.945 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-10T08:57:40.945 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-10T08:57:40.946 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.target_file_size_base: 67108864 2024-02-10T08:57:40.946 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.target_file_size_multiplier: 1 2024-02-10T08:57:40.946 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-10T08:57:40.946 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-10T08:57:40.946 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-10T08:57:40.946 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-10T08:57:40.946 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-10T08:57:40.946 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-10T08:57:40.946 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-10T08:57:40.946 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-10T08:57:40.946 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-10T08:57:40.946 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-10T08:57:40.946 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-10T08:57:40.946 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-10T08:57:40.946 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.arena_block_size: 4194304 2024-02-10T08:57:40.946 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-10T08:57:40.947 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-10T08:57:40.947 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-10T08:57:40.947 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.disable_auto_compactions: 0 2024-02-10T08:57:40.947 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-10T08:57:40.947 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-10T08:57:40.947 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-10T08:57:40.947 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-10T08:57:40.947 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-10T08:57:40.947 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-10T08:57:40.947 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-10T08:57:40.947 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-10T08:57:40.947 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-10T08:57:40.947 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-10T08:57:40.947 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.table_properties_collectors: 2024-02-10T08:57:40.947 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.inplace_update_support: 0 2024-02-10T08:57:40.948 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-02-10T08:57:40.948 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-10T08:57:40.948 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-10T08:57:40.948 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.memtable_huge_page_size: 0 2024-02-10T08:57:40.948 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.bloom_locality: 0 2024-02-10T08:57:40.948 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.max_successive_merges: 0 2024-02-10T08:57:40.948 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-02-10T08:57:40.948 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.paranoid_file_checks: 0 2024-02-10T08:57:40.948 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.force_consistency_checks: 0 2024-02-10T08:57:40.948 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.report_bg_io_stats: 0 2024-02-10T08:57:40.948 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.ttl: 2592000 2024-02-10T08:57:40.948 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-02-10T08:57:40.948 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi077/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-10T08:57:40.948 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: 2024-02-10T08:57:40.949 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.811+0000 7f8e4659c700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2024-02-10T08:57:40.949 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: 2024-02-10T08:57:40.949 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.812+0000 7f8e4659c700 4 rocksdb: DB pointer 0x55dc9eabf800 2024-02-10T08:57:40.949 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.812+0000 7f8e2f435700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-02-10T08:57:40.949 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.812+0000 7f8e2f435700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-02-10T08:57:40.949 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2024-02-10T08:57:40.949 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T08:57:40.949 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-10T08:57:40.949 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-10T08:57:40.949 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T08:57:40.949 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-10T08:57:40.949 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-10T08:57:40.949 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T08:57:40.949 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: 2024-02-10T08:57:40.950 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-10T08:57:40.950 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T08:57:40.950 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T08:57:40.950 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-10T08:57:40.950 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-10T08:57:40.950 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: 2024-02-10T08:57:40.950 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-10T08:57:40.950 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T08:57:40.950 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T08:57:40.950 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T08:57:40.950 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-02-10T08:57:40.950 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T08:57:40.950 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T08:57:40.950 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T08:57:40.951 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-02-10T08:57:40.951 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T08:57:40.951 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T08:57:40.951 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T08:57:40.951 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: 2024-02-10T08:57:40.951 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-02-10T08:57:40.951 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: 2024-02-10T08:57:40.951 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-10T08:57:40.951 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T08:57:40.951 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T08:57:40.951 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-10T08:57:40.951 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-10T08:57:40.951 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: 2024-02-10T08:57:40.951 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-10T08:57:40.951 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T08:57:40.951 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T08:57:40.952 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T08:57:40.952 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-02-10T08:57:40.952 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T08:57:40.952 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T08:57:40.952 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T08:57:40.952 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-02-10T08:57:40.952 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T08:57:40.952 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T08:57:40.952 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T08:57:40.952 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: 2024-02-10T08:57:40.952 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-02-10T08:57:40.952 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: 2024-02-10T08:57:40.952 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.814+0000 7f8e4659c700 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2024-02-10T08:57:40.952 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.814+0000 7f8e4659c700 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2024-02-10T08:57:40.953 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph-mon: debug 2024-02-10T08:57:40.814+0000 7f8e4659c700 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-smithi077 for mon.smithi077 2024-02-10T08:57:41.561 INFO:teuthology.orchestra.run.smithi077.stderr:create mon.smithi077 on 2024-02-10T08:57:41.561 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: install -d -m0770 -o 167 -g 167 /var/run/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715 2024-02-10T08:57:41.567 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: systemctl disable ceph.target 2024-02-10T08:57:41.574 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-02-10T08:57:41.658 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: systemctl enable ceph.target 2024-02-10T08:57:41.665 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-02-10T08:57:41.745 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: systemctl start ceph.target 2024-02-10T08:57:41.752 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: systemctl enable ceph-6081b54a-c7f2-11ee-95b8-87774f69a715.target 2024-02-10T08:57:41.758 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-6081b54a-c7f2-11ee-95b8-87774f69a715.target → /etc/systemd/system/ceph-6081b54a-c7f2-11ee-95b8-87774f69a715.target. 2024-02-10T08:57:41.758 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-6081b54a-c7f2-11ee-95b8-87774f69a715.target → /etc/systemd/system/ceph-6081b54a-c7f2-11ee-95b8-87774f69a715.target. 2024-02-10T08:57:41.838 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: systemctl start ceph-6081b54a-c7f2-11ee-95b8-87774f69a715.target 2024-02-10T08:57:41.845 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: systemctl daemon-reload 2024-02-10T08:57:41.928 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: systemctl stop ceph-6081b54a-c7f2-11ee-95b8-87774f69a715@mon.smithi077 2024-02-10T08:57:41.951 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: systemctl reset-failed ceph-6081b54a-c7f2-11ee-95b8-87774f69a715@mon.smithi077 2024-02-10T08:57:41.956 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: Failed to reset failed state of unit ceph-6081b54a-c7f2-11ee-95b8-87774f69a715@mon.smithi077.service: Unit ceph-6081b54a-c7f2-11ee-95b8-87774f69a715@mon.smithi077.service not loaded. 2024-02-10T08:57:41.957 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: systemctl enable ceph-6081b54a-c7f2-11ee-95b8-87774f69a715@mon.smithi077 2024-02-10T08:57:41.963 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: Created symlink /etc/systemd/system/ceph-6081b54a-c7f2-11ee-95b8-87774f69a715.target.wants/ceph-6081b54a-c7f2-11ee-95b8-87774f69a715@mon.smithi077.service → /etc/systemd/system/ceph-6081b54a-c7f2-11ee-95b8-87774f69a715@.service. 2024-02-10T08:57:42.047 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: systemctl start ceph-6081b54a-c7f2-11ee-95b8-87774f69a715@mon.smithi077 2024-02-10T08:57:42.617 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: systemctl is-enabled firewalld.service 2024-02-10T08:57:42.626 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: disabled 2024-02-10T08:57:42.626 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: systemctl is-active firewalld.service 2024-02-10T08:57:42.633 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: inactive 2024-02-10T08:57:42.634 INFO:teuthology.orchestra.run.smithi077.stderr:firewalld.service is not enabled 2024-02-10T08:57:42.634 INFO:teuthology.orchestra.run.smithi077.stderr:Not possible to enable service . firewalld.service is not available 2024-02-10T08:57:42.634 INFO:teuthology.orchestra.run.smithi077.stderr:Not possible to open ports <[]>. firewalld.service is not available 2024-02-10T08:57:42.635 INFO:teuthology.orchestra.run.smithi077.stderr:Waiting for mon to start... 2024-02-10T08:57:42.635 INFO:teuthology.orchestra.run.smithi077.stderr:Waiting for mon... 2024-02-10T08:57:42.635 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/mon.smithi077:/var/lib/ceph/mon/ceph-smithi077:z -v /tmp/ceph-tmpzzyygg7o:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpx5m65_qh:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status 2024-02-10T08:57:42.859 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:42 smithi077 bash[39019]: 044c4b370d1aa886934f30b57498852696020d30c64a340b054a78be8ccb7cdb 2024-02-10T08:57:42.859 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:42 smithi077 systemd[1]: Started Ceph mon.smithi077 for 6081b54a-c7f2-11ee-95b8-87774f69a715. 2024-02-10T08:57:43.172 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:43.152+0000 7fe0b6b3c700 1 Processor -- start 2024-02-10T08:57:43.172 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:43.152+0000 7fe0b6b3c700 1 -- start start 2024-02-10T08:57:43.172 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:43.153+0000 7fe0b6b3c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe0b0070430 0x7fe0b00fea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-10T08:57:43.172 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:43.153+0000 7fe0b6b3c700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0b00fef40 con 0x7fe0b0070430 2024-02-10T08:57:43.176 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:43.153+0000 7fe0b48d8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe0b0070430 0x7fe0b00fea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:57:43.176 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:43.153+0000 7fe0b48d8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe0b0070430 0x7fe0b00fea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:39776/0 (socket says 172.21.15.77:39776) 2024-02-10T08:57:43.176 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:43.153+0000 7fe0b48d8700 1 -- 172.21.15.77:0/2990312796 learned_addr learned my addr 172.21.15.77:0/2990312796 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-10T08:57:43.176 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:43.153+0000 7fe0b48d8700 1 -- 172.21.15.77:0/2990312796 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0b00ff080 con 0x7fe0b0070430 2024-02-10T08:57:43.176 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:43.153+0000 7fe0b48d8700 1 --2- 172.21.15.77:0/2990312796 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe0b0070430 0x7fe0b00fea50 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fe0a400b5f0 tx=0x7fe0a4006ee0).ready entity=mon.0 client_cookie=3e7f9f5d5f71acef server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:57:43.176 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:43.154+0000 7fe0af7fe700 1 -- 172.21.15.77:0/2990312796 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe0a400d600 con 0x7fe0b0070430 2024-02-10T08:57:43.176 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:43.154+0000 7fe0af7fe700 1 -- 172.21.15.77:0/2990312796 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0a400d760 con 0x7fe0b0070430 2024-02-10T08:57:43.176 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:43.154+0000 7fe0af7fe700 1 -- 172.21.15.77:0/2990312796 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe0a400da30 con 0x7fe0b0070430 2024-02-10T08:57:43.176 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:43.154+0000 7fe0b6b3c700 1 -- 172.21.15.77:0/2990312796 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe0b0070430 msgr2=0x7fe0b00fea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:57:43.176 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:43.154+0000 7fe0b6b3c700 1 --2- 172.21.15.77:0/2990312796 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe0b0070430 0x7fe0b00fea50 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fe0a400b5f0 tx=0x7fe0a4006ee0).stop 2024-02-10T08:57:43.176 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:43.154+0000 7fe0b6b3c700 1 -- 172.21.15.77:0/2990312796 shutdown_connections 2024-02-10T08:57:43.177 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:43.154+0000 7fe0b6b3c700 1 --2- 172.21.15.77:0/2990312796 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe0b0070430 0x7fe0b00fea50 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:57:43.177 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:43.154+0000 7fe0b6b3c700 1 -- 172.21.15.77:0/2990312796 >> 172.21.15.77:0/2990312796 conn(0x7fe0b00f6530 msgr2=0x7fe0b00f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:57:43.177 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:43.154+0000 7fe0b6b3c700 1 -- 172.21.15.77:0/2990312796 shutdown_connections 2024-02-10T08:57:43.177 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:43.154+0000 7fe0b6b3c700 1 -- 172.21.15.77:0/2990312796 wait complete. 2024-02-10T08:57:43.177 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:43.155+0000 7fe0b6b3c700 1 Processor -- start 2024-02-10T08:57:43.177 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:43.155+0000 7fe0b6b3c700 1 -- start start 2024-02-10T08:57:43.177 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:43.155+0000 7fe0b6b3c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe0b010a420 0x7fe0b010a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:57:43.177 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:43.155+0000 7fe0b6b3c700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0b010ad30 con 0x7fe0b010a420 2024-02-10T08:57:43.177 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:43.155+0000 7fe0b48d8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe0b010a420 0x7fe0b010a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:57:43.177 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:43.155+0000 7fe0b48d8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe0b010a420 0x7fe0b010a840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:39778/0 (socket says 172.21.15.77:39778) 2024-02-10T08:57:43.177 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:43.155+0000 7fe0b48d8700 1 -- 172.21.15.77:0/754691975 learned_addr learned my addr 172.21.15.77:0/754691975 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-10T08:57:43.177 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:43.155+0000 7fe0b48d8700 1 -- 172.21.15.77:0/754691975 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0a400b200 con 0x7fe0b010a420 2024-02-10T08:57:43.177 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:43.155+0000 7fe0b48d8700 1 --2- 172.21.15.77:0/754691975 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe0b010a420 0x7fe0b010a840 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fe0a4006f20 tx=0x7fe0a4016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:57:43.177 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:43.155+0000 7fe0ad7fa700 1 -- 172.21.15.77:0/754691975 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe0a4011b00 con 0x7fe0b010a420 2024-02-10T08:57:43.178 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:43.155+0000 7fe0ad7fa700 1 -- 172.21.15.77:0/754691975 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0a4011c80 con 0x7fe0b010a420 2024-02-10T08:57:43.178 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:43.155+0000 7fe0b6b3c700 1 -- 172.21.15.77:0/754691975 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0b010af30 con 0x7fe0b010a420 2024-02-10T08:57:43.178 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:43.155+0000 7fe0ad7fa700 1 -- 172.21.15.77:0/754691975 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe0a401d430 con 0x7fe0b010a420 2024-02-10T08:57:43.178 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:43.156+0000 7fe0b6b3c700 1 -- 172.21.15.77:0/754691975 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0b010ba80 con 0x7fe0b010a420 2024-02-10T08:57:43.178 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:43.156+0000 7fe0ad7fa700 1 -- 172.21.15.77:0/754691975 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fe0a400f070 con 0x7fe0b010a420 2024-02-10T08:57:43.178 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:43.156+0000 7fe0ad7fa700 1 -- 172.21.15.77:0/754691975 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fe0a4027460 con 0x7fe0b010a420 2024-02-10T08:57:43.178 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:43.156+0000 7fe0b6b3c700 1 -- 172.21.15.77:0/754691975 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0b0059100 con 0x7fe0b010a420 2024-02-10T08:57:43.178 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:43.159+0000 7fe0ad7fa700 1 -- 172.21.15.77:0/754691975 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fe0a4016070 con 0x7fe0b010a420 2024-02-10T08:57:43.231 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:43.228+0000 7fe0b6b3c700 1 -- 172.21.15.77:0/754691975 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fe0b0045ad0 con 0x7fe0b010a420 2024-02-10T08:57:43.231 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:43.228+0000 7fe0ad7fa700 1 -- 172.21.15.77:0/754691975 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7fe0a4036070 con 0x7fe0b010a420 2024-02-10T08:57:43.231 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: cluster: 2024-02-10T08:57:43.231 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: id: 6081b54a-c7f2-11ee-95b8-87774f69a715 2024-02-10T08:57:43.231 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-02-10T08:57:43.232 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:43.232 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: services: 2024-02-10T08:57:43.232 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi077 (age 0.655028s) 2024-02-10T08:57:43.232 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: mgr: no daemons active 2024-02-10T08:57:43.232 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-02-10T08:57:43.232 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:43.232 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: data: 2024-02-10T08:57:43.232 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-02-10T08:57:43.232 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-02-10T08:57:43.232 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-02-10T08:57:43.232 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: pgs: 2024-02-10T08:57:43.232 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:43.232 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:43.229+0000 7fe0b6b3c700 1 -- 172.21.15.77:0/754691975 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe0b010a420 msgr2=0x7fe0b010a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:57:43.232 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:43.229+0000 7fe0b6b3c700 1 --2- 172.21.15.77:0/754691975 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe0b010a420 0x7fe0b010a840 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fe0a4006f20 tx=0x7fe0a4016000).stop 2024-02-10T08:57:43.233 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:43.229+0000 7fe0b6b3c700 1 -- 172.21.15.77:0/754691975 shutdown_connections 2024-02-10T08:57:43.233 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:43.229+0000 7fe0b6b3c700 1 --2- 172.21.15.77:0/754691975 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe0b010a420 0x7fe0b010a840 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:57:43.233 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:43.229+0000 7fe0b6b3c700 1 -- 172.21.15.77:0/754691975 >> 172.21.15.77:0/754691975 conn(0x7fe0b00f6530 msgr2=0x7fe0b00f8280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:57:43.233 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:43.229+0000 7fe0b6b3c700 1 -- 172.21.15.77:0/754691975 shutdown_connections 2024-02-10T08:57:43.233 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:43.229+0000 7fe0b6b3c700 1 -- 172.21.15.77:0/754691975 wait complete. 2024-02-10T08:57:43.899 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:43 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39033]: cluster 2024-02-10T08:57:42.574639+0000 mon.smithi077 (mon.0) 2 : cluster [INF] mon.smithi077 is new leader, mons smithi077 in quorum (ranks 0) 2024-02-10T08:57:43.899 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:43 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39033]: cluster 2024-02-10T08:57:42.574815+0000 mon.smithi077 (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {smithi077=[v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0]} 2024-02-10T08:57:43.899 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:43 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39033]: cluster 2024-02-10T08:57:42.578038+0000 mon.smithi077 (mon.0) 4 : cluster [DBG] fsmap 2024-02-10T08:57:43.899 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:43 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39033]: cluster 2024-02-10T08:57:42.581838+0000 mon.smithi077 (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-02-10T08:57:43.899 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:43 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39033]: cluster 2024-02-10T08:57:42.582309+0000 mon.smithi077 (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2024-02-10T08:57:43.899 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:43 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39033]: audit 2024-02-10T08:57:43.229631+0000 mon.smithi077 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.77:0/754691975' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-10T08:57:43.925 INFO:teuthology.orchestra.run.smithi077.stderr:mon is available 2024-02-10T08:57:43.925 INFO:teuthology.orchestra.run.smithi077.stderr:Assimilating anything we can from ceph.conf... 2024-02-10T08:57:43.925 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpzzyygg7o:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpx5m65_qh:/etc/ceph/ceph.conf:z -v /var/lib/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/mon.smithi077:/var/lib/ceph/mon/ceph-smithi077:z docker.io/ceph/ceph:v16.2.4 config assimilate-conf -i /var/lib/ceph/mon/ceph-smithi077/config 2024-02-10T08:57:44.512 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:44.509+0000 7f3905ab5700 1 Processor -- start 2024-02-10T08:57:44.512 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:44.509+0000 7f3905ab5700 1 -- start start 2024-02-10T08:57:44.512 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:44.509+0000 7f3905ab5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39000fad90 0x7f39000fb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-10T08:57:44.512 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:44.509+0000 7f3905ab5700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39000fb6a0 con 0x7f39000fad90 2024-02-10T08:57:44.512 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:44.509+0000 7f38ff7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39000fad90 0x7f39000fb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:57:44.512 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:44.509+0000 7f38ff7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39000fad90 0x7f39000fb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:39782/0 (socket says 172.21.15.77:39782) 2024-02-10T08:57:44.513 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:44.509+0000 7f38ff7fe700 1 -- 172.21.15.77:0/3696029606 learned_addr learned my addr 172.21.15.77:0/3696029606 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-10T08:57:44.513 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:44.509+0000 7f38ff7fe700 1 -- 172.21.15.77:0/3696029606 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39000fb7e0 con 0x7f39000fad90 2024-02-10T08:57:44.513 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:44.510+0000 7f38ff7fe700 1 --2- 172.21.15.77:0/3696029606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39000fad90 0x7f39000fb1b0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f38f400b5f0 tx=0x7f38f4006ee0).ready entity=mon.0 client_cookie=93a2e2ec0736bf33 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:57:44.513 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:44.510+0000 7f38fe7fc700 1 -- 172.21.15.77:0/3696029606 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f38f400d600 con 0x7f39000fad90 2024-02-10T08:57:44.513 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:44.510+0000 7f38fe7fc700 1 -- 172.21.15.77:0/3696029606 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38f400d760 con 0x7f39000fad90 2024-02-10T08:57:44.513 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:44.510+0000 7f38fe7fc700 1 -- 172.21.15.77:0/3696029606 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f38f400da30 con 0x7f39000fad90 2024-02-10T08:57:44.513 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:44.510+0000 7f3905ab5700 1 -- 172.21.15.77:0/3696029606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39000fad90 msgr2=0x7f39000fb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:57:44.513 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:44.510+0000 7f3905ab5700 1 --2- 172.21.15.77:0/3696029606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39000fad90 0x7f39000fb1b0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f38f400b5f0 tx=0x7f38f4006ee0).stop 2024-02-10T08:57:44.513 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:44.510+0000 7f3905ab5700 1 -- 172.21.15.77:0/3696029606 shutdown_connections 2024-02-10T08:57:44.513 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:44.511+0000 7f3905ab5700 1 --2- 172.21.15.77:0/3696029606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39000fad90 0x7f39000fb1b0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:57:44.513 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:44.511+0000 7f3905ab5700 1 -- 172.21.15.77:0/3696029606 >> 172.21.15.77:0/3696029606 conn(0x7f39000f6530 msgr2=0x7f39000f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:57:44.513 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:44.511+0000 7f3905ab5700 1 -- 172.21.15.77:0/3696029606 shutdown_connections 2024-02-10T08:57:44.513 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:44.511+0000 7f3905ab5700 1 -- 172.21.15.77:0/3696029606 wait complete. 2024-02-10T08:57:44.515 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:44.511+0000 7f3905ab5700 1 Processor -- start 2024-02-10T08:57:44.515 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:44.512+0000 7f3905ab5700 1 -- start start 2024-02-10T08:57:44.516 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:44.512+0000 7f3905ab5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39000fad90 0x7f390010f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:57:44.516 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:44.512+0000 7f3905ab5700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f390010ae80 con 0x7f39000fad90 2024-02-10T08:57:44.516 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:44.512+0000 7f38ff7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39000fad90 0x7f390010f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:57:44.516 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:44.512+0000 7f38ff7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39000fad90 0x7f390010f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:39798/0 (socket says 172.21.15.77:39798) 2024-02-10T08:57:44.516 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:44.512+0000 7f38ff7fe700 1 -- 172.21.15.77:0/4235596185 learned_addr learned my addr 172.21.15.77:0/4235596185 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-10T08:57:44.516 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:44.512+0000 7f38ff7fe700 1 -- 172.21.15.77:0/4235596185 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38f400b200 con 0x7f39000fad90 2024-02-10T08:57:44.516 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:44.512+0000 7f38ff7fe700 1 --2- 172.21.15.77:0/4235596185 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39000fad90 0x7f390010f490 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f38f4006c70 tx=0x7f38f4016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:57:44.516 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:44.512+0000 7f38e7fff700 1 -- 172.21.15.77:0/4235596185 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f38f4011b40 con 0x7f39000fad90 2024-02-10T08:57:44.516 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:44.512+0000 7f38e7fff700 1 -- 172.21.15.77:0/4235596185 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38f4011ca0 con 0x7f39000fad90 2024-02-10T08:57:44.516 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:44.512+0000 7f3905ab5700 1 -- 172.21.15.77:0/4235596185 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3900070450 con 0x7f39000fad90 2024-02-10T08:57:44.516 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:44.512+0000 7f38e7fff700 1 -- 172.21.15.77:0/4235596185 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f38f401d430 con 0x7f39000fad90 2024-02-10T08:57:44.516 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:44.513+0000 7f3905ab5700 1 -- 172.21.15.77:0/4235596185 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f390010f980 con 0x7f39000fad90 2024-02-10T08:57:44.516 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:44.513+0000 7f38e7fff700 1 -- 172.21.15.77:0/4235596185 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f38f400f070 con 0x7f39000fad90 2024-02-10T08:57:44.517 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:44.513+0000 7f38e7fff700 1 -- 172.21.15.77:0/4235596185 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f38f4027460 con 0x7f39000fad90 2024-02-10T08:57:44.517 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:44.514+0000 7f3905ab5700 1 -- 172.21.15.77:0/4235596185 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3900045ad0 con 0x7f39000fad90 2024-02-10T08:57:44.517 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:44.516+0000 7f38e7fff700 1 -- 172.21.15.77:0/4235596185 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f38f4016070 con 0x7f39000fad90 2024-02-10T08:57:44.581 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:44.580+0000 7f3905ab5700 1 -- 172.21.15.77:0/4235596185 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f39000fc050 con 0x7f39000fad90 2024-02-10T08:57:44.587 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:44.584+0000 7f38e7fff700 1 -- 172.21.15.77:0/4235596185 <== mon.0 v2:172.21.15.77:3300/0 7 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f38f401d620 con 0x7f39000fad90 2024-02-10T08:57:44.588 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:44.584+0000 7f38e7fff700 1 -- 172.21.15.77:0/4235596185 <== mon.0 v2:172.21.15.77:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+489 (secure 0 0 0) 0x7f38f4030990 con 0x7f39000fad90 2024-02-10T08:57:44.588 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:44.588 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: [global] 2024-02-10T08:57:44.588 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: fsid = 6081b54a-c7f2-11ee-95b8-87774f69a715 2024-02-10T08:57:44.588 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: log_to_journald = false 2024-02-10T08:57:44.588 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.77:3300,v1:172.21.15.77:6789] 2024-02-10T08:57:44.588 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-02-10T08:57:44.588 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-02-10T08:57:44.588 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-02-10T08:57:44.588 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-02-10T08:57:44.588 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:44.588 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: [mgr] 2024-02-10T08:57:44.588 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-02-10T08:57:44.589 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:44.589 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: [osd] 2024-02-10T08:57:44.589 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: mon_osd_backfillfull_ratio = 0.85 2024-02-10T08:57:44.589 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: mon_osd_full_ratio = 0.9 2024-02-10T08:57:44.589 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: mon_osd_nearfull_ratio = 0.8 2024-02-10T08:57:44.589 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-02-10T08:57:44.589 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: osd_memory_target_autotune = true 2024-02-10T08:57:44.589 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-02-10T08:57:44.589 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:44.585+0000 7f3905ab5700 1 -- 172.21.15.77:0/4235596185 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39000fad90 msgr2=0x7f390010f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:57:44.589 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:44.585+0000 7f3905ab5700 1 --2- 172.21.15.77:0/4235596185 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39000fad90 0x7f390010f490 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f38f4006c70 tx=0x7f38f4016000).stop 2024-02-10T08:57:44.589 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:44.585+0000 7f3905ab5700 1 -- 172.21.15.77:0/4235596185 shutdown_connections 2024-02-10T08:57:44.589 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:44.585+0000 7f3905ab5700 1 --2- 172.21.15.77:0/4235596185 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f39000fad90 0x7f390010f490 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:57:44.589 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:44.585+0000 7f3905ab5700 1 -- 172.21.15.77:0/4235596185 >> 172.21.15.77:0/4235596185 conn(0x7f39000f6530 msgr2=0x7f39000fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:57:44.589 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:44.585+0000 7f3905ab5700 1 -- 172.21.15.77:0/4235596185 shutdown_connections 2024-02-10T08:57:44.589 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:44.585+0000 7f3905ab5700 1 -- 172.21.15.77:0/4235596185 wait complete. 2024-02-10T08:57:44.899 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:44 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39033]: audit 2024-02-10T08:57:44.582448+0000 mon.smithi077 (mon.0) 8 : audit [INF] from='client.? 172.21.15.77:0/4235596185' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-10T08:57:44.899 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:44 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39033]: audit 2024-02-10T08:57:44.585496+0000 mon.smithi077 (mon.0) 9 : audit [INF] from='client.? 172.21.15.77:0/4235596185' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-02-10T08:57:45.304 INFO:teuthology.orchestra.run.smithi077.stderr:Generating new minimal ceph.conf... 2024-02-10T08:57:45.304 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpzzyygg7o:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpx5m65_qh:/etc/ceph/ceph.conf:z -v /var/lib/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/mon.smithi077:/var/lib/ceph/mon/ceph-smithi077:z docker.io/ceph/ceph:v16.2.4 config generate-minimal-conf -o /var/lib/ceph/mon/ceph-smithi077/config 2024-02-10T08:57:45.784 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:45.779+0000 7fd8eb831700 1 Processor -- start 2024-02-10T08:57:45.784 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:45.779+0000 7fd8eb831700 1 -- start start 2024-02-10T08:57:45.785 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:45.780+0000 7fd8eb831700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8e4070430 0x7fd8e40fea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-10T08:57:45.785 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:45.780+0000 7fd8eb831700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e40fef40 con 0x7fd8e4070430 2024-02-10T08:57:45.785 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:45.780+0000 7fd8e95cd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8e4070430 0x7fd8e40fea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:57:45.785 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:45.780+0000 7fd8e95cd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8e4070430 0x7fd8e40fea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:39812/0 (socket says 172.21.15.77:39812) 2024-02-10T08:57:45.785 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:45.780+0000 7fd8e95cd700 1 -- 172.21.15.77:0/1515671543 learned_addr learned my addr 172.21.15.77:0/1515671543 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-10T08:57:45.785 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:45.780+0000 7fd8e95cd700 1 -- 172.21.15.77:0/1515671543 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8e40ff080 con 0x7fd8e4070430 2024-02-10T08:57:45.785 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:45.780+0000 7fd8e95cd700 1 --2- 172.21.15.77:0/1515671543 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8e4070430 0x7fd8e40fea50 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fd8d400b5f0 tx=0x7fd8d4006ee0).ready entity=mon.0 client_cookie=27ad5271294a53a9 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:57:45.785 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:45.781+0000 7fd8d3fff700 1 -- 172.21.15.77:0/1515671543 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8d400d600 con 0x7fd8e4070430 2024-02-10T08:57:45.785 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:45.781+0000 7fd8d3fff700 1 -- 172.21.15.77:0/1515671543 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fd8d400dbf0 con 0x7fd8e4070430 2024-02-10T08:57:45.785 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:45.781+0000 7fd8d3fff700 1 -- 172.21.15.77:0/1515671543 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8d4011cb0 con 0x7fd8e4070430 2024-02-10T08:57:45.785 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:45.781+0000 7fd8eb831700 1 -- 172.21.15.77:0/1515671543 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8e4070430 msgr2=0x7fd8e40fea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:57:45.785 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:45.781+0000 7fd8eb831700 1 --2- 172.21.15.77:0/1515671543 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8e4070430 0x7fd8e40fea50 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fd8d400b5f0 tx=0x7fd8d4006ee0).stop 2024-02-10T08:57:45.785 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:45.781+0000 7fd8eb831700 1 -- 172.21.15.77:0/1515671543 shutdown_connections 2024-02-10T08:57:45.785 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:45.781+0000 7fd8eb831700 1 --2- 172.21.15.77:0/1515671543 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8e4070430 0x7fd8e40fea50 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:57:45.785 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:45.781+0000 7fd8eb831700 1 -- 172.21.15.77:0/1515671543 >> 172.21.15.77:0/1515671543 conn(0x7fd8e40f6530 msgr2=0x7fd8e40f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:57:45.786 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:45.781+0000 7fd8eb831700 1 -- 172.21.15.77:0/1515671543 shutdown_connections 2024-02-10T08:57:45.786 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:45.781+0000 7fd8eb831700 1 -- 172.21.15.77:0/1515671543 wait complete. 2024-02-10T08:57:45.786 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:45.782+0000 7fd8eb831700 1 Processor -- start 2024-02-10T08:57:45.786 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:45.782+0000 7fd8eb831700 1 -- start start 2024-02-10T08:57:45.786 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:45.782+0000 7fd8eb831700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8e4070430 0x7fd8e410ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:57:45.786 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:45.782+0000 7fd8eb831700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e410b030 con 0x7fd8e4070430 2024-02-10T08:57:45.786 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:45.782+0000 7fd8e95cd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8e4070430 0x7fd8e410ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:57:45.786 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:45.782+0000 7fd8e95cd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8e4070430 0x7fd8e410ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:39818/0 (socket says 172.21.15.77:39818) 2024-02-10T08:57:45.786 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:45.782+0000 7fd8e95cd700 1 -- 172.21.15.77:0/1306520 learned_addr learned my addr 172.21.15.77:0/1306520 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-10T08:57:45.786 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:45.782+0000 7fd8e95cd700 1 -- 172.21.15.77:0/1306520 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8d400b200 con 0x7fd8e4070430 2024-02-10T08:57:45.786 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:45.782+0000 7fd8e95cd700 1 --2- 172.21.15.77:0/1306520 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8e4070430 0x7fd8e410ab40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fd8d4006800 tx=0x7fd8d4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:57:45.786 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:45.783+0000 7fd8d1ffb700 1 -- 172.21.15.77:0/1306520 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8d401c800 con 0x7fd8e4070430 2024-02-10T08:57:45.786 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:45.783+0000 7fd8d1ffb700 1 -- 172.21.15.77:0/1306520 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fd8d401cdf0 con 0x7fd8e4070430 2024-02-10T08:57:45.786 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:45.783+0000 7fd8eb831700 1 -- 172.21.15.77:0/1306520 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8e410b230 con 0x7fd8e4070430 2024-02-10T08:57:45.786 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:45.783+0000 7fd8d1ffb700 1 -- 172.21.15.77:0/1306520 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8d4024bd0 con 0x7fd8e4070430 2024-02-10T08:57:45.787 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:45.783+0000 7fd8eb831700 1 -- 172.21.15.77:0/1306520 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8e410b6c0 con 0x7fd8e4070430 2024-02-10T08:57:45.787 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:45.783+0000 7fd8d1ffb700 1 -- 172.21.15.77:0/1306520 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fd8d400f070 con 0x7fd8e4070430 2024-02-10T08:57:45.787 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:45.783+0000 7fd8d1ffb700 1 -- 172.21.15.77:0/1306520 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fd8d4025c70 con 0x7fd8e4070430 2024-02-10T08:57:45.787 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:45.784+0000 7fd8eb831700 1 -- 172.21.15.77:0/1306520 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8e4045ad0 con 0x7fd8e4070430 2024-02-10T08:57:45.787 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:45.786+0000 7fd8d1ffb700 1 -- 172.21.15.77:0/1306520 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fd8d4016070 con 0x7fd8e4070430 2024-02-10T08:57:45.852 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:45.851+0000 7fd8eb831700 1 -- 172.21.15.77:0/1306520 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fd8e40fff10 con 0x7fd8e4070430 2024-02-10T08:57:45.852 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:45.851+0000 7fd8d1ffb700 1 -- 172.21.15.77:0/1306520 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7fd8d4033090 con 0x7fd8e4070430 2024-02-10T08:57:45.854 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:45.852+0000 7fd8eb831700 1 -- 172.21.15.77:0/1306520 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8e4070430 msgr2=0x7fd8e410ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:57:45.854 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:45.852+0000 7fd8eb831700 1 --2- 172.21.15.77:0/1306520 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8e4070430 0x7fd8e410ab40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fd8d4006800 tx=0x7fd8d4006ee0).stop 2024-02-10T08:57:45.854 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:45.852+0000 7fd8eb831700 1 -- 172.21.15.77:0/1306520 shutdown_connections 2024-02-10T08:57:45.854 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:45.852+0000 7fd8eb831700 1 --2- 172.21.15.77:0/1306520 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8e4070430 0x7fd8e410ab40 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:57:45.854 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:45.852+0000 7fd8eb831700 1 -- 172.21.15.77:0/1306520 >> 172.21.15.77:0/1306520 conn(0x7fd8e40f6530 msgr2=0x7fd8e40f7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:57:45.854 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:45.852+0000 7fd8eb831700 1 -- 172.21.15.77:0/1306520 shutdown_connections 2024-02-10T08:57:45.854 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:45.852+0000 7fd8eb831700 1 -- 172.21.15.77:0/1306520 wait complete. 2024-02-10T08:57:46.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:45 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39033]: audit 2024-02-10T08:57:45.852937+0000 mon.smithi077 (mon.0) 10 : audit [DBG] from='client.? 172.21.15.77:0/1306520' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:57:46.411 INFO:teuthology.orchestra.run.smithi077.stderr:Restarting the monitor... 2024-02-10T08:57:46.411 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: systemctl restart ceph-6081b54a-c7f2-11ee-95b8-87774f69a715@mon.smithi077 2024-02-10T08:57:46.870 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:46 smithi077 systemd[1]: Stopping Ceph mon.smithi077 for 6081b54a-c7f2-11ee-95b8-87774f69a715... 2024-02-10T08:57:46.870 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:46 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39033]: debug 2024-02-10T08:57:46.541+0000 7f3c29b46700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi077 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-02-10T08:57:46.870 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:46 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39033]: debug 2024-02-10T08:57:46.541+0000 7f3c29b46700 -1 mon.smithi077@0(leader) e1 *** Got Signal Terminated *** 2024-02-10T08:57:47.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:46 smithi077 podman[39357]: ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077 2024-02-10T08:57:47.519 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 systemd[1]: ceph-6081b54a-c7f2-11ee-95b8-87774f69a715@mon.smithi077.service: Succeeded. 2024-02-10T08:57:47.519 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 systemd[1]: Stopped Ceph mon.smithi077 for 6081b54a-c7f2-11ee-95b8-87774f69a715. 2024-02-10T08:57:47.519 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 systemd[1]: Starting Ceph mon.smithi077 for 6081b54a-c7f2-11ee-95b8-87774f69a715... 2024-02-10T08:57:47.717 INFO:teuthology.orchestra.run.smithi077.stderr:Setting mon public_network to 172.21.0.0/20 2024-02-10T08:57:47.718 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpzzyygg7o:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpx5m65_qh:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mon public_network 172.21.0.0/20 2024-02-10T08:57:47.942 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.652+0000 7f3f7735e700 0 set uid:gid to 167:167 (ceph:ceph) 2024-02-10T08:57:47.942 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.652+0000 7f3f7735e700 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-mon, pid 7 2024-02-10T08:57:47.943 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.652+0000 7f3f7735e700 0 pidfile_write: ignore empty --pid-file 2024-02-10T08:57:47.943 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.656+0000 7f3f7735e700 0 load: jerasure load: lrc load: isa 2024-02-10T08:57:47.943 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: RocksDB version: 6.8.1 2024-02-10T08:57:47.943 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: 2024-02-10T08:57:47.943 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-10T08:57:47.943 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Compile date May 13 2021 2024-02-10T08:57:47.943 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: DB SUMMARY 2024-02-10T08:57:47.943 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: 2024-02-10T08:57:47.943 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: CURRENT file: CURRENT 2024-02-10T08:57:47.943 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: 2024-02-10T08:57:47.943 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: IDENTITY file: IDENTITY 2024-02-10T08:57:47.943 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: 2024-02-10T08:57:47.943 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2024-02-10T08:57:47.944 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: 2024-02-10T08:57:47.944 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi077/store.db dir, Total Num: 1, files: 000004.sst 2024-02-10T08:57:47.944 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: 2024-02-10T08:57:47.944 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi077/store.db: 000006.log size: 98698 ; 2024-02-10T08:57:47.944 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: 2024-02-10T08:57:47.944 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.error_if_exists: 0 2024-02-10T08:57:47.944 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.create_if_missing: 0 2024-02-10T08:57:47.944 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.paranoid_checks: 1 2024-02-10T08:57:47.944 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.env: 0x5637ef1d91c0 2024-02-10T08:57:47.944 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.fs: Posix File System 2024-02-10T08:57:47.944 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.info_log: 0x5637f00bb960 2024-02-10T08:57:47.944 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.max_file_opening_threads: 16 2024-02-10T08:57:47.944 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.statistics: (nil) 2024-02-10T08:57:47.944 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.use_fsync: 0 2024-02-10T08:57:47.945 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.max_log_file_size: 0 2024-02-10T08:57:47.945 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-10T08:57:47.945 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.log_file_time_to_roll: 0 2024-02-10T08:57:47.945 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.keep_log_file_num: 1000 2024-02-10T08:57:47.945 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.recycle_log_file_num: 0 2024-02-10T08:57:47.945 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.allow_fallocate: 1 2024-02-10T08:57:47.945 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.allow_mmap_reads: 0 2024-02-10T08:57:47.945 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.allow_mmap_writes: 0 2024-02-10T08:57:47.945 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.use_direct_reads: 0 2024-02-10T08:57:47.945 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-10T08:57:47.945 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.create_missing_column_families: 0 2024-02-10T08:57:47.945 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.db_log_dir: 2024-02-10T08:57:47.945 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi077/store.db 2024-02-10T08:57:47.945 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.table_cache_numshardbits: 6 2024-02-10T08:57:47.945 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.max_subcompactions: 1 2024-02-10T08:57:47.946 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.max_background_flushes: -1 2024-02-10T08:57:47.946 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-02-10T08:57:47.946 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-02-10T08:57:47.946 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-10T08:57:47.946 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-10T08:57:47.946 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-02-10T08:57:47.946 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.advise_random_on_open: 1 2024-02-10T08:57:47.946 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.db_write_buffer_size: 0 2024-02-10T08:57:47.946 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.write_buffer_manager: 0x5637f0d30030 2024-02-10T08:57:47.946 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-10T08:57:47.946 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-10T08:57:47.946 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-10T08:57:47.946 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.use_adaptive_mutex: 0 2024-02-10T08:57:47.946 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.rate_limiter: (nil) 2024-02-10T08:57:47.947 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-10T08:57:47.947 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.wal_recovery_mode: 2 2024-02-10T08:57:47.947 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.enable_thread_tracking: 0 2024-02-10T08:57:47.947 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.enable_pipelined_write: 0 2024-02-10T08:57:47.947 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.unordered_write: 0 2024-02-10T08:57:47.947 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-10T08:57:47.947 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-10T08:57:47.947 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-10T08:57:47.947 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-10T08:57:47.947 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.row_cache: None 2024-02-10T08:57:47.947 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.wal_filter: None 2024-02-10T08:57:47.947 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-10T08:57:47.947 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.allow_ingest_behind: 0 2024-02-10T08:57:47.947 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.preserve_deletes: 0 2024-02-10T08:57:47.948 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.two_write_queues: 0 2024-02-10T08:57:47.948 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.manual_wal_flush: 0 2024-02-10T08:57:47.948 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.atomic_flush: 0 2024-02-10T08:57:47.948 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-10T08:57:47.948 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.persist_stats_to_disk: 0 2024-02-10T08:57:47.948 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-02-10T08:57:47.948 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.log_readahead_size: 0 2024-02-10T08:57:47.948 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-02-10T08:57:47.948 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.max_background_jobs: 2 2024-02-10T08:57:47.948 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.max_background_compactions: -1 2024-02-10T08:57:47.948 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-10T08:57:47.948 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-10T08:57:47.948 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.delayed_write_rate : 16777216 2024-02-10T08:57:47.948 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.max_total_wal_size: 0 2024-02-10T08:57:47.949 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-10T08:57:47.949 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.stats_dump_period_sec: 600 2024-02-10T08:57:47.949 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.stats_persist_period_sec: 600 2024-02-10T08:57:47.949 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-10T08:57:47.949 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.max_open_files: -1 2024-02-10T08:57:47.949 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.bytes_per_sync: 0 2024-02-10T08:57:47.949 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-02-10T08:57:47.949 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-02-10T08:57:47.949 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.compaction_readahead_size: 0 2024-02-10T08:57:47.949 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Compression algorithms supported: 2024-02-10T08:57:47.949 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-10T08:57:47.949 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: kZSTD supported: 0 2024-02-10T08:57:47.949 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: kXpressCompression supported: 0 2024-02-10T08:57:47.949 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: kLZ4HCCompression supported: 1 2024-02-10T08:57:47.949 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: kLZ4Compression supported: 1 2024-02-10T08:57:47.950 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: kBZip2Compression supported: 0 2024-02-10T08:57:47.950 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: kZlibCompression supported: 1 2024-02-10T08:57:47.950 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: kSnappyCompression supported: 1 2024-02-10T08:57:47.950 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-02-10T08:57:47.951 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi077/store.db/MANIFEST-000005 2024-02-10T08:57:47.951 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: 2024-02-10T08:57:47.951 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-02-10T08:57:47.952 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: 2024-02-10T08:57:47.952 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-10T08:57:47.952 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.merge_operator: 2024-02-10T08:57:47.952 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.compaction_filter: None 2024-02-10T08:57:47.952 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.compaction_filter_factory: None 2024-02-10T08:57:47.952 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-02-10T08:57:47.952 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.table_factory: BlockBasedTable 2024-02-10T08:57:47.952 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5637effd3cb0) 2024-02-10T08:57:47.952 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cache_index_and_filter_blocks: 1 2024-02-10T08:57:47.952 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-10T08:57:47.952 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-10T08:57:47.952 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: pin_top_level_index_and_filter: 1 2024-02-10T08:57:47.952 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: index_type: 0 2024-02-10T08:57:47.952 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: data_block_index_type: 0 2024-02-10T08:57:47.953 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: index_shortening: 1 2024-02-10T08:57:47.953 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: data_block_hash_table_util_ratio: 0.750000 2024-02-10T08:57:47.953 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: hash_index_allow_collision: 1 2024-02-10T08:57:47.953 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: checksum: 1 2024-02-10T08:57:47.953 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: no_block_cache: 0 2024-02-10T08:57:47.953 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: block_cache: 0x5637f0008f10 2024-02-10T08:57:47.953 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: block_cache_name: BinnedLRUCache 2024-02-10T08:57:47.953 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: block_cache_options: 2024-02-10T08:57:47.953 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: capacity : 536870912 2024-02-10T08:57:47.953 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: num_shard_bits : 4 2024-02-10T08:57:47.953 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: strict_capacity_limit : 0 2024-02-10T08:57:47.953 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: high_pri_pool_ratio: 0.000 2024-02-10T08:57:47.953 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: block_cache_compressed: (nil) 2024-02-10T08:57:47.953 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: persistent_cache: (nil) 2024-02-10T08:57:47.954 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: block_size: 4096 2024-02-10T08:57:47.954 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: block_size_deviation: 10 2024-02-10T08:57:47.954 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: block_restart_interval: 16 2024-02-10T08:57:47.954 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: index_block_restart_interval: 1 2024-02-10T08:57:47.954 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: metadata_block_size: 4096 2024-02-10T08:57:47.954 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: partition_filters: 0 2024-02-10T08:57:47.954 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: use_delta_encoding: 1 2024-02-10T08:57:47.954 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-10T08:57:47.954 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: whole_key_filtering: 1 2024-02-10T08:57:47.954 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: verify_compression: 0 2024-02-10T08:57:47.954 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: read_amp_bytes_per_bit: 0 2024-02-10T08:57:47.954 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: format_version: 2 2024-02-10T08:57:47.954 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: enable_index_compression: 1 2024-02-10T08:57:47.954 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: block_align: 0 2024-02-10T08:57:47.955 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: 2024-02-10T08:57:47.955 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.write_buffer_size: 33554432 2024-02-10T08:57:47.955 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.max_write_buffer_number: 2 2024-02-10T08:57:47.955 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.compression: NoCompression 2024-02-10T08:57:47.955 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.bottommost_compression: Disabled 2024-02-10T08:57:47.955 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.prefix_extractor: nullptr 2024-02-10T08:57:47.955 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-10T08:57:47.955 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.num_levels: 7 2024-02-10T08:57:47.955 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-10T08:57:47.955 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-10T08:57:47.955 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-10T08:57:47.955 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-10T08:57:47.955 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-10T08:57:47.955 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-10T08:57:47.955 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-10T08:57:47.956 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-10T08:57:47.956 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-10T08:57:47.956 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.compression_opts.window_bits: -14 2024-02-10T08:57:47.956 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.compression_opts.level: 32767 2024-02-10T08:57:47.956 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.compression_opts.strategy: 0 2024-02-10T08:57:47.956 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-10T08:57:47.956 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-10T08:57:47.956 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.compression_opts.enabled: false 2024-02-10T08:57:47.956 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-10T08:57:47.956 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-10T08:57:47.956 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-10T08:57:47.956 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.target_file_size_base: 67108864 2024-02-10T08:57:47.956 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.target_file_size_multiplier: 1 2024-02-10T08:57:47.956 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-10T08:57:47.957 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-10T08:57:47.957 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-10T08:57:47.957 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-10T08:57:47.957 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-10T08:57:47.957 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-10T08:57:47.957 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-10T08:57:47.957 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-10T08:57:47.957 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-10T08:57:47.957 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-10T08:57:47.957 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-10T08:57:47.957 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-10T08:57:47.957 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.arena_block_size: 4194304 2024-02-10T08:57:47.957 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-10T08:57:47.957 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-10T08:57:47.958 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-10T08:57:47.958 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.disable_auto_compactions: 0 2024-02-10T08:57:47.958 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-10T08:57:47.958 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-10T08:57:47.958 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-10T08:57:47.958 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-10T08:57:47.958 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-10T08:57:47.958 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-10T08:57:47.958 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-10T08:57:47.958 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-10T08:57:47.958 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-10T08:57:47.958 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-10T08:57:47.958 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.table_properties_collectors: 2024-02-10T08:57:47.958 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.inplace_update_support: 0 2024-02-10T08:57:47.958 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-02-10T08:57:47.959 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-10T08:57:47.959 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-10T08:57:47.959 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.memtable_huge_page_size: 0 2024-02-10T08:57:47.959 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.bloom_locality: 0 2024-02-10T08:57:47.959 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.max_successive_merges: 0 2024-02-10T08:57:47.959 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-02-10T08:57:47.959 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.paranoid_file_checks: 0 2024-02-10T08:57:47.959 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.force_consistency_checks: 0 2024-02-10T08:57:47.959 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.report_bg_io_stats: 0 2024-02-10T08:57:47.959 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.ttl: 2592000 2024-02-10T08:57:47.959 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.657+0000 7f3f7735e700 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-02-10T08:57:47.959 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.658+0000 7f3f7735e700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi077/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-10T08:57:47.959 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: 2024-02-10T08:57:47.959 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.658+0000 7f3f7735e700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2024-02-10T08:57:47.960 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: 2024-02-10T08:57:47.960 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.658+0000 7f3f7735e700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707555467659664, "job": 1, "event": "recovery_started", "log_files": [6]} 2024-02-10T08:57:47.960 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.658+0000 7f3f7735e700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2024-02-10T08:57:47.960 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.658+0000 7f3f7735e700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-02-10T08:57:47.960 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.659+0000 7f3f7735e700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707555467660499, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 94272, "table_properties": {"data_size": 92360, "index_size": 389, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 709, "raw_key_size": 12563, "raw_average_key_size": 49, "raw_value_size": 85391, "raw_average_value_size": 336, "num_data_blocks": 12, "num_entries": 254, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1707555467, "oldest_key_time": 3, "file_creation_time": 0}} 2024-02-10T08:57:47.960 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.659+0000 7f3f7735e700 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2024-02-10T08:57:47.960 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: 2024-02-10T08:57:47.960 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.659+0000 7f3f7735e700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1707555467660912, "job": 1, "event": "recovery_finished"} 2024-02-10T08:57:47.960 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.660+0000 7f3f7735e700 4 rocksdb: DB pointer 0x5637f00cd800 2024-02-10T08:57:47.960 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.660+0000 7f3f5e176700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-02-10T08:57:47.960 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.660+0000 7f3f5e176700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-02-10T08:57:47.960 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: ** DB Stats ** 2024-02-10T08:57:47.960 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T08:57:47.960 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-10T08:57:47.960 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-10T08:57:47.961 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T08:57:47.961 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-10T08:57:47.961 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-10T08:57:47.961 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T08:57:47.961 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: 2024-02-10T08:57:47.961 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: ** Compaction Stats [default] ** 2024-02-10T08:57:47.961 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T08:57:47.961 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T08:57:47.961 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: L0 2/0 93.75 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 156.9 0.00 0.00 1 0.001 0 0 2024-02-10T08:57:47.961 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: Sum 2/0 93.75 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 156.9 0.00 0.00 1 0.001 0 0 2024-02-10T08:57:47.961 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 156.9 0.00 0.00 1 0.001 0 0 2024-02-10T08:57:47.961 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: 2024-02-10T08:57:47.961 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: ** Compaction Stats [default] ** 2024-02-10T08:57:47.961 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T08:57:47.962 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T08:57:47.962 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 156.9 0.00 0.00 1 0.001 0 0 2024-02-10T08:57:47.962 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T08:57:47.962 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-10T08:57:47.962 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T08:57:47.962 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T08:57:47.962 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T08:57:47.962 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: AddFile(Keys): cumulative 0, interval 0 2024-02-10T08:57:47.962 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: Cumulative compaction: 0.00 GB write, 33.26 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T08:57:47.962 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: Interval compaction: 0.00 GB write, 33.26 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T08:57:47.962 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T08:57:47.962 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: 2024-02-10T08:57:47.962 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: ** File Read Latency Histogram By Level [default] ** 2024-02-10T08:57:47.962 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: 2024-02-10T08:57:47.962 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: ** Compaction Stats [default] ** 2024-02-10T08:57:47.963 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T08:57:47.963 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T08:57:47.963 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: L0 2/0 93.75 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 156.9 0.00 0.00 1 0.001 0 0 2024-02-10T08:57:47.963 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: Sum 2/0 93.75 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 156.9 0.00 0.00 1 0.001 0 0 2024-02-10T08:57:47.963 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-10T08:57:47.963 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: 2024-02-10T08:57:47.963 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: ** Compaction Stats [default] ** 2024-02-10T08:57:47.963 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-10T08:57:47.963 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T08:57:47.963 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 156.9 0.00 0.00 1 0.001 0 0 2024-02-10T08:57:47.963 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T08:57:47.963 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-10T08:57:47.963 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T08:57:47.963 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T08:57:47.964 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T08:57:47.964 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: AddFile(Keys): cumulative 0, interval 0 2024-02-10T08:57:47.964 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: Cumulative compaction: 0.00 GB write, 32.90 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T08:57:47.964 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T08:57:47.964 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T08:57:47.964 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: 2024-02-10T08:57:47.964 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: ** File Read Latency Histogram By Level [default] ** 2024-02-10T08:57:47.964 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: 2024-02-10T08:57:47.964 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.660+0000 7f3f7735e700 0 starting mon.smithi077 rank 0 at public addrs [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] at bind addrs [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi077 fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 2024-02-10T08:57:47.964 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.661+0000 7f3f7735e700 1 mon.smithi077@-1(???) e1 preinit fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 2024-02-10T08:57:47.964 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.662+0000 7f3f7735e700 0 mon.smithi077@-1(???).mds e1 new map 2024-02-10T08:57:47.964 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.662+0000 7f3f7735e700 0 mon.smithi077@-1(???).mds e1 print_map 2024-02-10T08:57:47.964 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: e1 2024-02-10T08:57:47.964 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: enable_multiple, ever_enabled_multiple: 1,1 2024-02-10T08:57:47.965 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-10T08:57:47.965 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: legacy client fscid: -1 2024-02-10T08:57:47.965 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: 2024-02-10T08:57:47.965 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: No filesystems configured 2024-02-10T08:57:47.965 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: 2024-02-10T08:57:47.965 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.662+0000 7f3f7735e700 0 mon.smithi077@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-10T08:57:47.965 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.662+0000 7f3f7735e700 0 mon.smithi077@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T08:57:47.965 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.662+0000 7f3f7735e700 0 mon.smithi077@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T08:57:47.965 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.662+0000 7f3f7735e700 0 mon.smithi077@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T08:57:47.965 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:57:45.852937+0000 mon.smithi077 (mon.0) 10 : audit [DBG] from='client.? 172.21.15.77:0/1306520' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T08:57:47.965 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.662+0000 7f3f7735e700 1 mon.smithi077@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-02-10T08:57:47.965 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.663+0000 7f3f7735e700 20 expand_channel_meta expand map: {default=false} 2024-02-10T08:57:47.965 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.663+0000 7f3f7735e700 20 expand_channel_meta from 'false' to 'false' 2024-02-10T08:57:47.965 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.663+0000 7f3f7735e700 20 expand_channel_meta expanded map: {default=false} 2024-02-10T08:57:47.965 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.663+0000 7f3f7735e700 20 expand_channel_meta expand map: {default=info} 2024-02-10T08:57:47.966 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.663+0000 7f3f7735e700 20 expand_channel_meta from 'info' to 'info' 2024-02-10T08:57:47.966 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.663+0000 7f3f7735e700 20 expand_channel_meta expanded map: {default=info} 2024-02-10T08:57:47.966 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.663+0000 7f3f7735e700 20 expand_channel_meta expand map: {default=daemon} 2024-02-10T08:57:47.966 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.663+0000 7f3f7735e700 20 expand_channel_meta from 'daemon' to 'daemon' 2024-02-10T08:57:47.966 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.663+0000 7f3f7735e700 20 expand_channel_meta expanded map: {default=daemon} 2024-02-10T08:57:47.966 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.663+0000 7f3f7735e700 20 expand_channel_meta expand map: {default=debug} 2024-02-10T08:57:47.966 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.663+0000 7f3f7735e700 20 expand_channel_meta from 'debug' to 'debug' 2024-02-10T08:57:47.966 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T08:57:47.663+0000 7f3f7735e700 20 expand_channel_meta expanded map: {default=debug} 2024-02-10T08:57:47.966 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 bash[39429]: a440be6a0c7fd4281a307f19882011393885db87a8c4747370782adaeeb344ca 2024-02-10T08:57:47.966 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 systemd[1]: Started Ceph mon.smithi077 for 6081b54a-c7f2-11ee-95b8-87774f69a715. 2024-02-10T08:57:47.966 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T08:57:47.679019+0000 mon.smithi077 (mon.0) 1 : cluster [INF] mon.smithi077 is new leader, mons smithi077 in quorum (ranks 0) 2024-02-10T08:57:47.966 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T08:57:47.679062+0000 mon.smithi077 (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {smithi077=[v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0]} 2024-02-10T08:57:47.966 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T08:57:47.679110+0000 mon.smithi077 (mon.0) 3 : cluster [DBG] fsmap 2024-02-10T08:57:47.966 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T08:57:47.679143+0000 mon.smithi077 (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-02-10T08:57:47.967 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T08:57:47.679494+0000 mon.smithi077 (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2024-02-10T08:57:48.267 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:48.261+0000 7fb68245a700 1 Processor -- start 2024-02-10T08:57:48.267 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:48.261+0000 7fb68245a700 1 -- start start 2024-02-10T08:57:48.267 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:48.261+0000 7fb68245a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb67c070430 0x7fb67c0fc840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-10T08:57:48.267 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:48.261+0000 7fb68245a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb67c0fcd30 con 0x7fb67c070430 2024-02-10T08:57:48.267 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:48.262+0000 7fb67bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb67c070430 0x7fb67c0fc840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:57:48.267 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:48.262+0000 7fb67bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb67c070430 0x7fb67c0fc840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43394/0 (socket says 172.21.15.77:43394) 2024-02-10T08:57:48.267 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:48.262+0000 7fb67bfff700 1 -- 172.21.15.77:0/2241916787 learned_addr learned my addr 172.21.15.77:0/2241916787 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-10T08:57:48.267 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:48.262+0000 7fb67bfff700 1 -- 172.21.15.77:0/2241916787 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb67c0fce70 con 0x7fb67c070430 2024-02-10T08:57:48.267 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:48.262+0000 7fb67bfff700 1 --2- 172.21.15.77:0/2241916787 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb67c070430 0x7fb67c0fc840 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fb66400b5f0 tx=0x7fb664006ee0).ready entity=mon.0 client_cookie=b070dd30c1d2c557 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:57:48.267 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:48.262+0000 7fb67affd700 1 -- 172.21.15.77:0/2241916787 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb66400d600 con 0x7fb67c070430 2024-02-10T08:57:48.267 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:48.262+0000 7fb67affd700 1 -- 172.21.15.77:0/2241916787 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fb66400dbf0 con 0x7fb67c070430 2024-02-10T08:57:48.268 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:48.263+0000 7fb67affd700 1 -- 172.21.15.77:0/2241916787 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb664011cb0 con 0x7fb67c070430 2024-02-10T08:57:48.268 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:48.263+0000 7fb68245a700 1 -- 172.21.15.77:0/2241916787 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb67c070430 msgr2=0x7fb67c0fc840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:57:48.268 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:48.263+0000 7fb68245a700 1 --2- 172.21.15.77:0/2241916787 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb67c070430 0x7fb67c0fc840 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fb66400b5f0 tx=0x7fb664006ee0).stop 2024-02-10T08:57:48.268 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:48.263+0000 7fb68245a700 1 -- 172.21.15.77:0/2241916787 shutdown_connections 2024-02-10T08:57:48.268 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:48.263+0000 7fb68245a700 1 --2- 172.21.15.77:0/2241916787 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb67c070430 0x7fb67c0fc840 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:57:48.268 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:48.263+0000 7fb68245a700 1 -- 172.21.15.77:0/2241916787 >> 172.21.15.77:0/2241916787 conn(0x7fb67c0f6530 msgr2=0x7fb67c0f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:57:48.268 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:48.263+0000 7fb68245a700 1 -- 172.21.15.77:0/2241916787 shutdown_connections 2024-02-10T08:57:48.268 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:48.263+0000 7fb68245a700 1 -- 172.21.15.77:0/2241916787 wait complete. 2024-02-10T08:57:48.268 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:48.264+0000 7fb68245a700 1 Processor -- start 2024-02-10T08:57:48.268 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:48.264+0000 7fb68245a700 1 -- start start 2024-02-10T08:57:48.268 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:48.264+0000 7fb68245a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb67c070430 0x7fb67c1111a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:57:48.268 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:48.264+0000 7fb68245a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb67c111690 con 0x7fb67c070430 2024-02-10T08:57:48.268 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:48.264+0000 7fb67bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb67c070430 0x7fb67c1111a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:57:48.269 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:48.264+0000 7fb67bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb67c070430 0x7fb67c1111a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43408/0 (socket says 172.21.15.77:43408) 2024-02-10T08:57:48.269 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:48.264+0000 7fb67bfff700 1 -- 172.21.15.77:0/2816936036 learned_addr learned my addr 172.21.15.77:0/2816936036 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-10T08:57:48.269 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:48.265+0000 7fb67bfff700 1 -- 172.21.15.77:0/2816936036 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb66400b200 con 0x7fb67c070430 2024-02-10T08:57:48.269 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:48.265+0000 7fb67bfff700 1 --2- 172.21.15.77:0/2816936036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb67c070430 0x7fb67c1111a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fb66400df90 tx=0x7fb664006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:57:48.269 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:48.265+0000 7fb678ff9700 1 -- 172.21.15.77:0/2816936036 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb66401b410 con 0x7fb67c070430 2024-02-10T08:57:48.269 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:48.265+0000 7fb678ff9700 1 -- 172.21.15.77:0/2816936036 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fb66401ba00 con 0x7fb67c070430 2024-02-10T08:57:48.269 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:48.265+0000 7fb68245a700 1 -- 172.21.15.77:0/2816936036 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb67c111890 con 0x7fb67c070430 2024-02-10T08:57:48.269 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:48.265+0000 7fb678ff9700 1 -- 172.21.15.77:0/2816936036 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb66401aa30 con 0x7fb67c070430 2024-02-10T08:57:48.269 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:48.265+0000 7fb68245a700 1 -- 172.21.15.77:0/2816936036 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb67c111d20 con 0x7fb67c070430 2024-02-10T08:57:48.269 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:48.265+0000 7fb678ff9700 1 -- 172.21.15.77:0/2816936036 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fb66400f070 con 0x7fb67c070430 2024-02-10T08:57:48.269 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:48.265+0000 7fb678ff9700 1 -- 172.21.15.77:0/2816936036 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fb664025c80 con 0x7fb67c070430 2024-02-10T08:57:48.270 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:48.266+0000 7fb68245a700 1 -- 172.21.15.77:0/2816936036 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb67c045ad0 con 0x7fb67c070430 2024-02-10T08:57:48.270 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:48.268+0000 7fb678ff9700 1 -- 172.21.15.77:0/2816936036 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fb664016070 con 0x7fb67c070430 2024-02-10T08:57:48.333 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:48.332+0000 7fb68245a700 1 -- 172.21.15.77:0/2816936036 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7fb67c0fdbb0 con 0x7fb67c070430 2024-02-10T08:57:48.335 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:48.334+0000 7fb678ff9700 1 -- 172.21.15.77:0/2816936036 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7fb664025460 con 0x7fb67c070430 2024-02-10T08:57:48.336 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:48.335+0000 7fb68245a700 1 -- 172.21.15.77:0/2816936036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb67c070430 msgr2=0x7fb67c1111a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:57:48.336 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:48.335+0000 7fb68245a700 1 --2- 172.21.15.77:0/2816936036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb67c070430 0x7fb67c1111a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fb66400df90 tx=0x7fb664006ee0).stop 2024-02-10T08:57:48.336 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:48.335+0000 7fb68245a700 1 -- 172.21.15.77:0/2816936036 shutdown_connections 2024-02-10T08:57:48.337 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:48.335+0000 7fb68245a700 1 --2- 172.21.15.77:0/2816936036 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb67c070430 0x7fb67c1111a0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:57:48.337 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:48.335+0000 7fb68245a700 1 -- 172.21.15.77:0/2816936036 >> 172.21.15.77:0/2816936036 conn(0x7fb67c0f6530 msgr2=0x7fb67c0f7f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:57:48.337 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:48.335+0000 7fb68245a700 1 -- 172.21.15.77:0/2816936036 shutdown_connections 2024-02-10T08:57:48.337 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:48.335+0000 7fb68245a700 1 -- 172.21.15.77:0/2816936036 wait complete. 2024-02-10T08:57:49.024 INFO:teuthology.orchestra.run.smithi077.stderr:Wrote config to /etc/ceph/ceph.conf 2024-02-10T08:57:49.025 INFO:teuthology.orchestra.run.smithi077.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-02-10T08:57:49.025 INFO:teuthology.orchestra.run.smithi077.stderr:Creating mgr... 2024-02-10T08:57:49.025 INFO:teuthology.orchestra.run.smithi077.stderr:Verifying port 9283 ... 2024-02-10T08:57:49.027 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: systemctl daemon-reload 2024-02-10T08:57:49.126 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: systemctl stop ceph-6081b54a-c7f2-11ee-95b8-87774f69a715@mgr.smithi077.dxntvw 2024-02-10T08:57:49.149 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: systemctl reset-failed ceph-6081b54a-c7f2-11ee-95b8-87774f69a715@mgr.smithi077.dxntvw 2024-02-10T08:57:49.154 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: Failed to reset failed state of unit ceph-6081b54a-c7f2-11ee-95b8-87774f69a715@mgr.smithi077.dxntvw.service: Unit ceph-6081b54a-c7f2-11ee-95b8-87774f69a715@mgr.smithi077.dxntvw.service not loaded. 2024-02-10T08:57:49.155 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: systemctl enable ceph-6081b54a-c7f2-11ee-95b8-87774f69a715@mgr.smithi077.dxntvw 2024-02-10T08:57:49.161 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: Created symlink /etc/systemd/system/ceph-6081b54a-c7f2-11ee-95b8-87774f69a715.target.wants/ceph-6081b54a-c7f2-11ee-95b8-87774f69a715@mgr.smithi077.dxntvw.service → /etc/systemd/system/ceph-6081b54a-c7f2-11ee-95b8-87774f69a715@.service. 2024-02-10T08:57:49.245 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: systemctl start ceph-6081b54a-c7f2-11ee-95b8-87774f69a715@mgr.smithi077.dxntvw 2024-02-10T08:57:49.587 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:49 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:57:48.335669+0000 mon.smithi077 (mon.0) 6 : audit [INF] from='client.? 172.21.15.77:0/2816936036' entity='client.admin' 2024-02-10T08:57:49.862 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: systemctl is-enabled firewalld.service 2024-02-10T08:57:49.870 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: disabled 2024-02-10T08:57:49.871 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: systemctl is-active firewalld.service 2024-02-10T08:57:49.879 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: inactive 2024-02-10T08:57:49.880 INFO:teuthology.orchestra.run.smithi077.stderr:firewalld.service is not enabled 2024-02-10T08:57:49.880 INFO:teuthology.orchestra.run.smithi077.stderr:Not possible to enable service . firewalld.service is not available 2024-02-10T08:57:49.880 INFO:teuthology.orchestra.run.smithi077.stderr:Not possible to open ports <[]>. firewalld.service is not available 2024-02-10T08:57:49.880 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: systemctl is-enabled firewalld.service 2024-02-10T08:57:49.888 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: disabled 2024-02-10T08:57:49.889 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: systemctl is-active firewalld.service 2024-02-10T08:57:49.896 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: inactive 2024-02-10T08:57:49.898 INFO:teuthology.orchestra.run.smithi077.stderr:firewalld.service is not enabled 2024-02-10T08:57:49.898 INFO:teuthology.orchestra.run.smithi077.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-02-10T08:57:49.898 INFO:teuthology.orchestra.run.smithi077.stderr:Waiting for mgr to start... 2024-02-10T08:57:49.898 INFO:teuthology.orchestra.run.smithi077.stderr:Waiting for mgr... 2024-02-10T08:57:49.898 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpzzyygg7o:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpx5m65_qh:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-02-10T08:57:50.537 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:50.534+0000 7fe4de680700 1 Processor -- start 2024-02-10T08:57:50.537 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:50.534+0000 7fe4de680700 1 -- start start 2024-02-10T08:57:50.537 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:50.534+0000 7fe4de680700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4d80703b0 0x7fe4d8100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-10T08:57:50.537 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:50.534+0000 7fe4de680700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d80fc8c0 con 0x7fe4d80703b0 2024-02-10T08:57:50.537 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:50.534+0000 7fe4d7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4d80703b0 0x7fe4d8100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:57:50.537 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:50.534+0000 7fe4d7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4d80703b0 0x7fe4d8100ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43428/0 (socket says 172.21.15.77:43428) 2024-02-10T08:57:50.537 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:50.534+0000 7fe4d7fff700 1 -- 172.21.15.77:0/515907511 learned_addr learned my addr 172.21.15.77:0/515907511 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-10T08:57:50.538 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:50.535+0000 7fe4d7fff700 1 -- 172.21.15.77:0/515907511 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4d81013c0 con 0x7fe4d80703b0 2024-02-10T08:57:50.538 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:50.535+0000 7fe4d7fff700 1 --2- 172.21.15.77:0/515907511 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4d80703b0 0x7fe4d8100ed0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fe4c800b5f0 tx=0x7fe4c8006ee0).ready entity=mon.0 client_cookie=e6aacdf794c8eec9 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:57:50.538 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:50.535+0000 7fe4d6ffd700 1 -- 172.21.15.77:0/515907511 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe4c800d600 con 0x7fe4d80703b0 2024-02-10T08:57:50.538 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:50.535+0000 7fe4d6ffd700 1 -- 172.21.15.77:0/515907511 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fe4c800dbf0 con 0x7fe4d80703b0 2024-02-10T08:57:50.538 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:50.535+0000 7fe4d6ffd700 1 -- 172.21.15.77:0/515907511 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe4c8011cb0 con 0x7fe4d80703b0 2024-02-10T08:57:50.539 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:50.536+0000 7fe4de680700 1 -- 172.21.15.77:0/515907511 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4d80703b0 msgr2=0x7fe4d8100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:57:50.539 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:50.536+0000 7fe4de680700 1 --2- 172.21.15.77:0/515907511 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4d80703b0 0x7fe4d8100ed0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fe4c800b5f0 tx=0x7fe4c8006ee0).stop 2024-02-10T08:57:50.540 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:50.536+0000 7fe4de680700 1 -- 172.21.15.77:0/515907511 shutdown_connections 2024-02-10T08:57:50.540 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:50.536+0000 7fe4de680700 1 --2- 172.21.15.77:0/515907511 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4d80703b0 0x7fe4d8100ed0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:57:50.540 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:50.536+0000 7fe4de680700 1 -- 172.21.15.77:0/515907511 >> 172.21.15.77:0/515907511 conn(0x7fe4d80f6530 msgr2=0x7fe4d80f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:57:50.540 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:50.537+0000 7fe4de680700 1 -- 172.21.15.77:0/515907511 shutdown_connections 2024-02-10T08:57:50.540 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:50.537+0000 7fe4de680700 1 -- 172.21.15.77:0/515907511 wait complete. 2024-02-10T08:57:50.540 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:50.537+0000 7fe4de680700 1 Processor -- start 2024-02-10T08:57:50.540 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:50.538+0000 7fe4de680700 1 -- start start 2024-02-10T08:57:50.540 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:50.538+0000 7fe4de680700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4d80703b0 0x7fe4d810cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:57:50.540 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:50.538+0000 7fe4de680700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d810d270 con 0x7fe4d80703b0 2024-02-10T08:57:50.540 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:50.538+0000 7fe4d7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4d80703b0 0x7fe4d810cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:57:50.540 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:50.538+0000 7fe4d7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4d80703b0 0x7fe4d810cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43430/0 (socket says 172.21.15.77:43430) 2024-02-10T08:57:50.541 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:50.538+0000 7fe4d7fff700 1 -- 172.21.15.77:0/1107971487 learned_addr learned my addr 172.21.15.77:0/1107971487 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-10T08:57:50.541 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:50.538+0000 7fe4d7fff700 1 -- 172.21.15.77:0/1107971487 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4c800b200 con 0x7fe4d80703b0 2024-02-10T08:57:50.541 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:50.538+0000 7fe4d7fff700 1 --2- 172.21.15.77:0/1107971487 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4d80703b0 0x7fe4d810cd80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fe4c8006800 tx=0x7fe4c8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:57:50.541 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:50.539+0000 7fe4d4ff9700 1 -- 172.21.15.77:0/1107971487 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe4c801b570 con 0x7fe4d80703b0 2024-02-10T08:57:50.541 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:50.539+0000 7fe4d4ff9700 1 -- 172.21.15.77:0/1107971487 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fe4c801bb60 con 0x7fe4d80703b0 2024-02-10T08:57:50.541 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:50.539+0000 7fe4de680700 1 -- 172.21.15.77:0/1107971487 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4d810d470 con 0x7fe4d80703b0 2024-02-10T08:57:50.542 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:50.539+0000 7fe4d4ff9700 1 -- 172.21.15.77:0/1107971487 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe4c801a9f0 con 0x7fe4d80703b0 2024-02-10T08:57:50.542 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:50.539+0000 7fe4de680700 1 -- 172.21.15.77:0/1107971487 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4d810e080 con 0x7fe4d80703b0 2024-02-10T08:57:50.542 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:50.539+0000 7fe4d4ff9700 1 -- 172.21.15.77:0/1107971487 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fe4c800f070 con 0x7fe4d80703b0 2024-02-10T08:57:50.542 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:50.539+0000 7fe4d4ff9700 1 -- 172.21.15.77:0/1107971487 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fe4c8025cb0 con 0x7fe4d80703b0 2024-02-10T08:57:50.542 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:50.540+0000 7fe4de680700 1 -- 172.21.15.77:0/1107971487 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4d8059100 con 0x7fe4d80703b0 2024-02-10T08:57:50.544 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:50.543+0000 7fe4d4ff9700 1 -- 172.21.15.77:0/1107971487 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fe4c8016070 con 0x7fe4d80703b0 2024-02-10T08:57:50.619 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:50.612+0000 7fe4de680700 1 -- 172.21.15.77:0/1107971487 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fe4d8102040 con 0x7fe4d80703b0 2024-02-10T08:57:50.619 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:50.612+0000 7fe4d4ff9700 1 -- 172.21.15.77:0/1107971487 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7fe4c8025460 con 0x7fe4d80703b0 2024-02-10T08:57:50.619 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:50.619 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: { 2024-02-10T08:57:50.619 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "fsid": "6081b54a-c7f2-11ee-95b8-87774f69a715", 2024-02-10T08:57:50.620 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "health": { 2024-02-10T08:57:50.620 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-10T08:57:50.620 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "checks": {}, 2024-02-10T08:57:50.620 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "mutes": [] 2024-02-10T08:57:50.620 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: }, 2024-02-10T08:57:50.620 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-10T08:57:50.620 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "quorum": [ 2024-02-10T08:57:50.620 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 0 2024-02-10T08:57:50.620 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: ], 2024-02-10T08:57:50.620 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-10T08:57:50.620 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "smithi077" 2024-02-10T08:57:50.620 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: ], 2024-02-10T08:57:50.620 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "quorum_age": 2, 2024-02-10T08:57:50.620 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "monmap": { 2024-02-10T08:57:50.621 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T08:57:50.621 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-02-10T08:57:50.621 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-10T08:57:50.621 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: }, 2024-02-10T08:57:50.621 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "osdmap": { 2024-02-10T08:57:50.621 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T08:57:50.621 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-10T08:57:50.621 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-10T08:57:50.621 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-10T08:57:50.621 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-10T08:57:50.621 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-10T08:57:50.621 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-10T08:57:50.621 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: }, 2024-02-10T08:57:50.621 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "pgmap": { 2024-02-10T08:57:50.622 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-10T08:57:50.622 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-10T08:57:50.622 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-10T08:57:50.622 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-10T08:57:50.622 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-10T08:57:50.622 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-10T08:57:50.622 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-10T08:57:50.622 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-10T08:57:50.622 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: }, 2024-02-10T08:57:50.622 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "fsmap": { 2024-02-10T08:57:50.622 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T08:57:50.622 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-10T08:57:50.622 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-10T08:57:50.622 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: }, 2024-02-10T08:57:50.622 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-10T08:57:50.623 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "available": false, 2024-02-10T08:57:50.623 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-10T08:57:50.623 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "modules": [ 2024-02-10T08:57:50.623 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "iostat", 2024-02-10T08:57:50.623 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "restful" 2024-02-10T08:57:50.623 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: ], 2024-02-10T08:57:50.623 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "services": {} 2024-02-10T08:57:50.623 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: }, 2024-02-10T08:57:50.623 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "servicemap": { 2024-02-10T08:57:50.623 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T08:57:50.623 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "modified": "2024-02-10T08:57:42.576518+0000", 2024-02-10T08:57:50.623 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "services": {} 2024-02-10T08:57:50.623 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: }, 2024-02-10T08:57:50.623 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-10T08:57:50.623 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: } 2024-02-10T08:57:50.624 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:50.613+0000 7fe4de680700 1 -- 172.21.15.77:0/1107971487 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4d80703b0 msgr2=0x7fe4d810cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:57:50.624 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:50.613+0000 7fe4de680700 1 --2- 172.21.15.77:0/1107971487 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4d80703b0 0x7fe4d810cd80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fe4c8006800 tx=0x7fe4c8006ee0).stop 2024-02-10T08:57:50.624 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:50.613+0000 7fe4de680700 1 -- 172.21.15.77:0/1107971487 shutdown_connections 2024-02-10T08:57:50.624 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:50.613+0000 7fe4de680700 1 --2- 172.21.15.77:0/1107971487 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4d80703b0 0x7fe4d810cd80 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:57:50.624 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:50.613+0000 7fe4de680700 1 -- 172.21.15.77:0/1107971487 >> 172.21.15.77:0/1107971487 conn(0x7fe4d80f6530 msgr2=0x7fe4d80f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:57:50.624 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:50.613+0000 7fe4de680700 1 -- 172.21.15.77:0/1107971487 shutdown_connections 2024-02-10T08:57:50.624 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:50.613+0000 7fe4de680700 1 -- 172.21.15.77:0/1107971487 wait complete. 2024-02-10T08:57:50.898 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:50 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:57:50.613602+0000 mon.smithi077 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.77:0/1107971487' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-10T08:57:51.258 INFO:teuthology.orchestra.run.smithi077.stderr:mgr not available, waiting (1/15)... 2024-02-10T08:57:53.261 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpzzyygg7o:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpx5m65_qh:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-02-10T08:57:53.854 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:53.840+0000 7f6bc958c700 1 Processor -- start 2024-02-10T08:57:53.854 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:53.840+0000 7f6bc958c700 1 -- start start 2024-02-10T08:57:53.854 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:53.840+0000 7f6bc958c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bc40fbea0 0x7f6bc40fc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-10T08:57:53.854 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:53.840+0000 7f6bc958c700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc40fc7b0 con 0x7f6bc40fbea0 2024-02-10T08:57:53.855 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:53.840+0000 7f6bc2ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bc40fbea0 0x7f6bc40fc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:57:53.855 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:53.840+0000 7f6bc2ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bc40fbea0 0x7f6bc40fc2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43432/0 (socket says 172.21.15.77:43432) 2024-02-10T08:57:53.855 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:53.840+0000 7f6bc2ffd700 1 -- 172.21.15.77:0/2078175317 learned_addr learned my addr 172.21.15.77:0/2078175317 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-10T08:57:53.855 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:53.840+0000 7f6bc2ffd700 1 -- 172.21.15.77:0/2078175317 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bc40fc8f0 con 0x7f6bc40fbea0 2024-02-10T08:57:53.855 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:53.840+0000 7f6bc2ffd700 1 --2- 172.21.15.77:0/2078175317 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bc40fbea0 0x7f6bc40fc2c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f6bb400b5f0 tx=0x7f6bb4006ee0).ready entity=mon.0 client_cookie=49d811c7698f8c83 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:57:53.855 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:53.841+0000 7f6bc1ffb700 1 -- 172.21.15.77:0/2078175317 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6bb400d600 con 0x7f6bc40fbea0 2024-02-10T08:57:53.855 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:53.841+0000 7f6bc1ffb700 1 -- 172.21.15.77:0/2078175317 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f6bb400dbf0 con 0x7f6bc40fbea0 2024-02-10T08:57:53.855 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:53.841+0000 7f6bc1ffb700 1 -- 172.21.15.77:0/2078175317 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6bb4011cb0 con 0x7f6bc40fbea0 2024-02-10T08:57:53.855 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:53.841+0000 7f6bc958c700 1 -- 172.21.15.77:0/2078175317 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bc40fbea0 msgr2=0x7f6bc40fc2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:57:53.855 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:53.841+0000 7f6bc958c700 1 --2- 172.21.15.77:0/2078175317 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bc40fbea0 0x7f6bc40fc2c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f6bb400b5f0 tx=0x7f6bb4006ee0).stop 2024-02-10T08:57:53.855 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:53.841+0000 7f6bc958c700 1 -- 172.21.15.77:0/2078175317 shutdown_connections 2024-02-10T08:57:53.855 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:53.841+0000 7f6bc958c700 1 --2- 172.21.15.77:0/2078175317 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bc40fbea0 0x7f6bc40fc2c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:57:53.855 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:53.841+0000 7f6bc958c700 1 -- 172.21.15.77:0/2078175317 >> 172.21.15.77:0/2078175317 conn(0x7f6bc40f76a0 msgr2=0x7f6bc40f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:57:53.856 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:53.841+0000 7f6bc958c700 1 -- 172.21.15.77:0/2078175317 shutdown_connections 2024-02-10T08:57:53.856 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:53.841+0000 7f6bc958c700 1 -- 172.21.15.77:0/2078175317 wait complete. 2024-02-10T08:57:53.856 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:53.842+0000 7f6bc958c700 1 Processor -- start 2024-02-10T08:57:53.856 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:53.842+0000 7f6bc958c700 1 -- start start 2024-02-10T08:57:53.856 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:53.842+0000 7f6bc958c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bc40fbea0 0x7f6bc410abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:57:53.856 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:53.842+0000 7f6bc958c700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc410b0b0 con 0x7f6bc40fbea0 2024-02-10T08:57:53.856 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:53.842+0000 7f6bc2ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bc40fbea0 0x7f6bc410abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:57:53.856 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:53.842+0000 7f6bc2ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bc40fbea0 0x7f6bc410abc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43446/0 (socket says 172.21.15.77:43446) 2024-02-10T08:57:53.856 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:53.842+0000 7f6bc2ffd700 1 -- 172.21.15.77:0/3533620951 learned_addr learned my addr 172.21.15.77:0/3533620951 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-10T08:57:53.856 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:53.842+0000 7f6bc2ffd700 1 -- 172.21.15.77:0/3533620951 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bb400b200 con 0x7f6bc40fbea0 2024-02-10T08:57:53.856 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:53.843+0000 7f6bc2ffd700 1 --2- 172.21.15.77:0/3533620951 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bc40fbea0 0x7f6bc410abc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f6bb4002f40 tx=0x7f6bb4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:57:53.856 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:53.843+0000 7f6bab7fe700 1 -- 172.21.15.77:0/3533620951 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6bb401b410 con 0x7f6bc40fbea0 2024-02-10T08:57:53.856 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:53.843+0000 7f6bc958c700 1 -- 172.21.15.77:0/3533620951 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bc410b2b0 con 0x7f6bc40fbea0 2024-02-10T08:57:53.856 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:53.843+0000 7f6bab7fe700 1 -- 172.21.15.77:0/3533620951 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f6bb401ba00 con 0x7f6bc40fbea0 2024-02-10T08:57:53.857 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:53.843+0000 7f6bab7fe700 1 -- 172.21.15.77:0/3533620951 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6bb401a980 con 0x7f6bc40fbea0 2024-02-10T08:57:53.857 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:53.843+0000 7f6bc958c700 1 -- 172.21.15.77:0/3533620951 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bc410b740 con 0x7f6bc40fbea0 2024-02-10T08:57:53.857 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:53.843+0000 7f6bab7fe700 1 -- 172.21.15.77:0/3533620951 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f6bb400f070 con 0x7f6bc40fbea0 2024-02-10T08:57:53.857 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:53.843+0000 7f6bab7fe700 1 -- 172.21.15.77:0/3533620951 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f6bb4025cc0 con 0x7f6bc40fbea0 2024-02-10T08:57:53.857 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:53.844+0000 7f6bc958c700 1 -- 172.21.15.77:0/3533620951 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bc4045ad0 con 0x7f6bc40fbea0 2024-02-10T08:57:53.857 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:53.846+0000 7f6bab7fe700 1 -- 172.21.15.77:0/3533620951 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f6bb4016070 con 0x7f6bc40fbea0 2024-02-10T08:57:53.918 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:53.914+0000 7f6bc958c700 1 -- 172.21.15.77:0/3533620951 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f6bc40fdc00 con 0x7f6bc40fbea0 2024-02-10T08:57:53.918 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:53.914+0000 7f6bab7fe700 1 -- 172.21.15.77:0/3533620951 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f6bb401aed0 con 0x7f6bc40fbea0 2024-02-10T08:57:53.918 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:53.918 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: { 2024-02-10T08:57:53.918 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "fsid": "6081b54a-c7f2-11ee-95b8-87774f69a715", 2024-02-10T08:57:53.918 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "health": { 2024-02-10T08:57:53.918 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-10T08:57:53.919 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "checks": {}, 2024-02-10T08:57:53.919 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "mutes": [] 2024-02-10T08:57:53.919 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: }, 2024-02-10T08:57:53.919 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-10T08:57:53.919 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "quorum": [ 2024-02-10T08:57:53.919 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 0 2024-02-10T08:57:53.919 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: ], 2024-02-10T08:57:53.919 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-10T08:57:53.919 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "smithi077" 2024-02-10T08:57:53.919 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: ], 2024-02-10T08:57:53.920 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "quorum_age": 6, 2024-02-10T08:57:53.920 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "monmap": { 2024-02-10T08:57:53.920 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T08:57:53.920 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-02-10T08:57:53.920 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-10T08:57:53.920 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: }, 2024-02-10T08:57:53.920 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "osdmap": { 2024-02-10T08:57:53.920 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T08:57:53.925 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-10T08:57:53.926 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-10T08:57:53.926 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-10T08:57:53.926 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-10T08:57:53.926 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-10T08:57:53.926 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-10T08:57:53.926 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: }, 2024-02-10T08:57:53.926 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "pgmap": { 2024-02-10T08:57:53.926 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-10T08:57:53.926 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-10T08:57:53.926 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-10T08:57:53.926 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-10T08:57:53.926 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-10T08:57:53.926 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-10T08:57:53.927 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-10T08:57:53.927 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-10T08:57:53.927 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: }, 2024-02-10T08:57:53.927 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "fsmap": { 2024-02-10T08:57:53.927 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T08:57:53.927 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-10T08:57:53.927 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-10T08:57:53.927 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: }, 2024-02-10T08:57:53.927 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-10T08:57:53.927 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "available": false, 2024-02-10T08:57:53.927 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-10T08:57:53.927 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "modules": [ 2024-02-10T08:57:53.927 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "iostat", 2024-02-10T08:57:53.927 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "restful" 2024-02-10T08:57:53.928 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: ], 2024-02-10T08:57:53.928 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "services": {} 2024-02-10T08:57:53.928 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: }, 2024-02-10T08:57:53.928 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "servicemap": { 2024-02-10T08:57:53.928 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T08:57:53.928 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "modified": "2024-02-10T08:57:42.576518+0000", 2024-02-10T08:57:53.928 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "services": {} 2024-02-10T08:57:53.928 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: }, 2024-02-10T08:57:53.928 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-10T08:57:53.928 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: } 2024-02-10T08:57:53.928 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:53.915+0000 7f6bc958c700 1 -- 172.21.15.77:0/3533620951 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bc40fbea0 msgr2=0x7f6bc410abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:57:53.928 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:53.915+0000 7f6bc958c700 1 --2- 172.21.15.77:0/3533620951 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bc40fbea0 0x7f6bc410abc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f6bb4002f40 tx=0x7f6bb4006ee0).stop 2024-02-10T08:57:53.928 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:53.915+0000 7f6bc958c700 1 -- 172.21.15.77:0/3533620951 shutdown_connections 2024-02-10T08:57:53.928 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:53.915+0000 7f6bc958c700 1 --2- 172.21.15.77:0/3533620951 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6bc40fbea0 0x7f6bc410abc0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:57:53.929 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:53.915+0000 7f6bc958c700 1 -- 172.21.15.77:0/3533620951 >> 172.21.15.77:0/3533620951 conn(0x7f6bc40f76a0 msgr2=0x7f6bc40f9020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:57:53.929 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:53.915+0000 7f6bc958c700 1 -- 172.21.15.77:0/3533620951 shutdown_connections 2024-02-10T08:57:53.929 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:53.915+0000 7f6bc958c700 1 -- 172.21.15.77:0/3533620951 wait complete. 2024-02-10T08:57:54.398 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:53 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:57:53.916052+0000 mon.smithi077 (mon.0) 8 : audit [DBG] from='client.? 172.21.15.77:0/3533620951' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-10T08:57:55.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:54 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T08:57:54.697631+0000 mon.smithi077 (mon.0) 9 : cluster [INF] Activating manager daemon smithi077.dxntvw 2024-02-10T08:57:55.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:54 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T08:57:54.700458+0000 mon.smithi077 (mon.0) 10 : cluster [DBG] mgrmap e2: smithi077.dxntvw(active, starting, since 0.00291354s) 2024-02-10T08:57:55.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:54 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:57:54.701330+0000 mon.smithi077 (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.77:0/3506548665' entity='mgr.smithi077.dxntvw' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T08:57:55.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:54 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:57:54.701508+0000 mon.smithi077 (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.77:0/3506548665' entity='mgr.smithi077.dxntvw' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T08:57:55.400 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:54 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:57:54.701629+0000 mon.smithi077 (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.77:0/3506548665' entity='mgr.smithi077.dxntvw' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T08:57:55.400 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:54 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:57:54.701760+0000 mon.smithi077 (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.77:0/3506548665' entity='mgr.smithi077.dxntvw' cmd=[{"prefix": "mon metadata", "id": "smithi077"}]: dispatch 2024-02-10T08:57:55.400 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:54 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:57:54.702018+0000 mon.smithi077 (mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.77:0/3506548665' entity='mgr.smithi077.dxntvw' cmd=[{"prefix": "mgr metadata", "who": "smithi077.dxntvw", "id": "smithi077.dxntvw"}]: dispatch 2024-02-10T08:57:55.400 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:54 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T08:57:54.715726+0000 mon.smithi077 (mon.0) 16 : cluster [INF] Manager daemon smithi077.dxntvw is now available 2024-02-10T08:57:55.400 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:54 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:57:54.723655+0000 mon.smithi077 (mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.77:0/3506548665' entity='mgr.smithi077.dxntvw' 2024-02-10T08:57:55.400 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:54 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:57:54.727212+0000 mon.smithi077 (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.77:0/3506548665' entity='mgr.smithi077.dxntvw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi077.dxntvw/mirror_snapshot_schedule"}]: dispatch 2024-02-10T08:57:55.400 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:54 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:57:54.728349+0000 mon.smithi077 (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.77:0/3506548665' entity='mgr.smithi077.dxntvw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi077.dxntvw/trash_purge_schedule"}]: dispatch 2024-02-10T08:57:55.400 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:54 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:57:54.732906+0000 mon.smithi077 (mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.77:0/3506548665' entity='mgr.smithi077.dxntvw' 2024-02-10T08:57:55.400 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:54 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:57:54.734892+0000 mon.smithi077 (mon.0) 21 : audit [INF] from='mgr.14100 172.21.15.77:0/3506548665' entity='mgr.smithi077.dxntvw' 2024-02-10T08:57:56.339 INFO:teuthology.orchestra.run.smithi077.stderr:mgr not available, waiting (2/15)... 2024-02-10T08:57:57.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:56 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T08:57:55.703244+0000 mon.smithi077 (mon.0) 22 : cluster [DBG] mgrmap e3: smithi077.dxntvw(active, since 1.0057s) 2024-02-10T08:57:58.342 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpzzyygg7o:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpx5m65_qh:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-02-10T08:57:58.899 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:58 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T08:57:57.703355+0000 mon.smithi077 (mon.0) 23 : cluster [DBG] mgrmap e4: smithi077.dxntvw(active, since 3s) 2024-02-10T08:57:58.907 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:58.905+0000 7f766d76d700 1 Processor -- start 2024-02-10T08:57:58.907 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:58.905+0000 7f766d76d700 1 -- start start 2024-02-10T08:57:58.907 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:58.905+0000 7f766d76d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76680703b0 0x7f7668100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-10T08:57:58.907 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:58.905+0000 7f766d76d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76680fc8c0 con 0x7f76680703b0 2024-02-10T08:57:58.907 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:58.906+0000 7f7666ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76680703b0 0x7f7668100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:57:58.907 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:58.906+0000 7f7666ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76680703b0 0x7f7668100ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42554/0 (socket says 172.21.15.77:42554) 2024-02-10T08:57:58.907 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:58.906+0000 7f7666ffd700 1 -- 172.21.15.77:0/1342897621 learned_addr learned my addr 172.21.15.77:0/1342897621 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-10T08:57:58.910 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:58.906+0000 7f7666ffd700 1 -- 172.21.15.77:0/1342897621 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76681013c0 con 0x7f76680703b0 2024-02-10T08:57:58.910 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:58.906+0000 7f7666ffd700 1 --2- 172.21.15.77:0/1342897621 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76680703b0 0x7f7668100ed0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f765c00b5f0 tx=0x7f765c006ee0).ready entity=mon.0 client_cookie=802d46fb386b4493 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:57:58.910 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:58.906+0000 7f7665ffb700 1 -- 172.21.15.77:0/1342897621 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f765c00d600 con 0x7f76680703b0 2024-02-10T08:57:58.910 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:58.906+0000 7f7665ffb700 1 -- 172.21.15.77:0/1342897621 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f765c00dbf0 con 0x7f76680703b0 2024-02-10T08:57:58.910 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:58.906+0000 7f7665ffb700 1 -- 172.21.15.77:0/1342897621 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f765c011cb0 con 0x7f76680703b0 2024-02-10T08:57:58.910 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:58.907+0000 7f766d76d700 1 -- 172.21.15.77:0/1342897621 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76680703b0 msgr2=0x7f7668100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:57:58.910 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:58.907+0000 7f766d76d700 1 --2- 172.21.15.77:0/1342897621 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76680703b0 0x7f7668100ed0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f765c00b5f0 tx=0x7f765c006ee0).stop 2024-02-10T08:57:58.910 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:58.907+0000 7f766d76d700 1 -- 172.21.15.77:0/1342897621 shutdown_connections 2024-02-10T08:57:58.910 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:58.907+0000 7f766d76d700 1 --2- 172.21.15.77:0/1342897621 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76680703b0 0x7f7668100ed0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:57:58.910 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:58.907+0000 7f766d76d700 1 -- 172.21.15.77:0/1342897621 >> 172.21.15.77:0/1342897621 conn(0x7f76680f6530 msgr2=0x7f76680f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:57:58.911 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:58.907+0000 7f766d76d700 1 -- 172.21.15.77:0/1342897621 shutdown_connections 2024-02-10T08:57:58.911 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:58.907+0000 7f766d76d700 1 -- 172.21.15.77:0/1342897621 wait complete. 2024-02-10T08:57:58.911 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:58.908+0000 7f766d76d700 1 Processor -- start 2024-02-10T08:57:58.911 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:58.908+0000 7f766d76d700 1 -- start start 2024-02-10T08:57:58.911 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:58.908+0000 7f766d76d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76680703b0 0x7f766810cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:57:58.911 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:58.908+0000 7f766d76d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f766810d2c0 con 0x7f76680703b0 2024-02-10T08:57:58.911 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:58.908+0000 7f7666ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76680703b0 0x7f766810cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:57:58.911 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:58.908+0000 7f7666ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76680703b0 0x7f766810cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42568/0 (socket says 172.21.15.77:42568) 2024-02-10T08:57:58.911 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:58.908+0000 7f7666ffd700 1 -- 172.21.15.77:0/695425334 learned_addr learned my addr 172.21.15.77:0/695425334 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-10T08:57:58.912 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:58.908+0000 7f7666ffd700 1 -- 172.21.15.77:0/695425334 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f765c00b200 con 0x7f76680703b0 2024-02-10T08:57:58.912 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:58.908+0000 7f7666ffd700 1 --2- 172.21.15.77:0/695425334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76680703b0 0x7f766810cdd0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f765c00df90 tx=0x7f765c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:57:58.912 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:58.909+0000 7f764f7fe700 1 -- 172.21.15.77:0/695425334 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f765c01b410 con 0x7f76680703b0 2024-02-10T08:57:58.912 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:58.909+0000 7f764f7fe700 1 -- 172.21.15.77:0/695425334 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f765c01ba00 con 0x7f76680703b0 2024-02-10T08:57:58.913 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:58.909+0000 7f766d76d700 1 -- 172.21.15.77:0/695425334 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f766810d4c0 con 0x7f76680703b0 2024-02-10T08:57:58.913 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:58.909+0000 7f764f7fe700 1 -- 172.21.15.77:0/695425334 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f765c01aa30 con 0x7f76680703b0 2024-02-10T08:57:58.913 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:58.909+0000 7f766d76d700 1 -- 172.21.15.77:0/695425334 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f766810e100 con 0x7f76680703b0 2024-02-10T08:57:58.913 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:58.910+0000 7f764f7fe700 1 -- 172.21.15.77:0/695425334 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 4) v1 ==== 36109+0+0 (secure 0 0 0) 0x7f765c00f070 con 0x7f76680703b0 2024-02-10T08:57:58.913 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:58.910+0000 7f764f7fe700 1 --2- 172.21.15.77:0/695425334 >> [v2:172.21.15.77:6800/101381127,v1:172.21.15.77:6801/101381127] conn(0x7f765402fdc0 0x7f7654032280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:57:58.913 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:58.910+0000 7f764f7fe700 1 -- 172.21.15.77:0/695425334 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f765c041080 con 0x7f76680703b0 2024-02-10T08:57:58.913 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:58.910+0000 7f76667fc700 1 --2- 172.21.15.77:0/695425334 >> [v2:172.21.15.77:6800/101381127,v1:172.21.15.77:6801/101381127] conn(0x7f765402fdc0 0x7f7654032280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:57:58.913 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:58.910+0000 7f766d76d700 1 -- 172.21.15.77:0/695425334 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f766810d950 con 0x7f76680703b0 2024-02-10T08:57:58.913 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:58.910+0000 7f76667fc700 1 --2- 172.21.15.77:0/695425334 >> [v2:172.21.15.77:6800/101381127,v1:172.21.15.77:6801/101381127] conn(0x7f765402fdc0 0x7f7654032280 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f76580096a0 tx=0x7f7658006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:57:58.915 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:58.915+0000 7f764f7fe700 1 -- 172.21.15.77:0/695425334 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f765c016070 con 0x7f76680703b0 2024-02-10T08:57:59.125 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:59.117+0000 7f766d76d700 1 -- 172.21.15.77:0/695425334 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f76681020e0 con 0x7f76680703b0 2024-02-10T08:57:59.125 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:59.117+0000 7f764f7fe700 1 -- 172.21.15.77:0/695425334 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f765c022d50 con 0x7f76680703b0 2024-02-10T08:57:59.125 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:59.125 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: { 2024-02-10T08:57:59.125 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "fsid": "6081b54a-c7f2-11ee-95b8-87774f69a715", 2024-02-10T08:57:59.125 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "health": { 2024-02-10T08:57:59.126 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-10T08:57:59.126 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "checks": {}, 2024-02-10T08:57:59.126 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "mutes": [] 2024-02-10T08:57:59.126 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: }, 2024-02-10T08:57:59.126 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-10T08:57:59.126 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "quorum": [ 2024-02-10T08:57:59.126 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 0 2024-02-10T08:57:59.126 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: ], 2024-02-10T08:57:59.126 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-10T08:57:59.126 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "smithi077" 2024-02-10T08:57:59.126 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: ], 2024-02-10T08:57:59.126 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "quorum_age": 11, 2024-02-10T08:57:59.126 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "monmap": { 2024-02-10T08:57:59.126 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T08:57:59.127 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-02-10T08:57:59.127 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-10T08:57:59.127 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: }, 2024-02-10T08:57:59.127 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "osdmap": { 2024-02-10T08:57:59.127 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T08:57:59.127 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-10T08:57:59.127 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-10T08:57:59.127 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-10T08:57:59.127 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-10T08:57:59.127 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-10T08:57:59.127 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-10T08:57:59.127 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: }, 2024-02-10T08:57:59.127 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "pgmap": { 2024-02-10T08:57:59.127 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-10T08:57:59.128 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-10T08:57:59.128 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-10T08:57:59.128 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-10T08:57:59.128 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-10T08:57:59.128 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-10T08:57:59.128 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-10T08:57:59.128 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-10T08:57:59.128 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: }, 2024-02-10T08:57:59.128 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "fsmap": { 2024-02-10T08:57:59.128 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T08:57:59.128 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-10T08:57:59.128 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-10T08:57:59.128 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: }, 2024-02-10T08:57:59.128 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-10T08:57:59.129 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "available": true, 2024-02-10T08:57:59.129 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-10T08:57:59.129 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "modules": [ 2024-02-10T08:57:59.129 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "iostat", 2024-02-10T08:57:59.129 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "restful" 2024-02-10T08:57:59.129 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: ], 2024-02-10T08:57:59.129 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "services": {} 2024-02-10T08:57:59.129 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: }, 2024-02-10T08:57:59.129 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "servicemap": { 2024-02-10T08:57:59.129 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-10T08:57:59.129 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "modified": "2024-02-10T08:57:42.576518+0000", 2024-02-10T08:57:59.129 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "services": {} 2024-02-10T08:57:59.129 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: }, 2024-02-10T08:57:59.129 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-10T08:57:59.129 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: } 2024-02-10T08:57:59.130 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:59.119+0000 7f766d76d700 1 -- 172.21.15.77:0/695425334 >> [v2:172.21.15.77:6800/101381127,v1:172.21.15.77:6801/101381127] conn(0x7f765402fdc0 msgr2=0x7f7654032280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:57:59.130 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:59.119+0000 7f766d76d700 1 --2- 172.21.15.77:0/695425334 >> [v2:172.21.15.77:6800/101381127,v1:172.21.15.77:6801/101381127] conn(0x7f765402fdc0 0x7f7654032280 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f76580096a0 tx=0x7f7658006b40).stop 2024-02-10T08:57:59.130 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:59.119+0000 7f766d76d700 1 -- 172.21.15.77:0/695425334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76680703b0 msgr2=0x7f766810cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:57:59.130 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:59.119+0000 7f766d76d700 1 --2- 172.21.15.77:0/695425334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76680703b0 0x7f766810cdd0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f765c00df90 tx=0x7f765c006ee0).stop 2024-02-10T08:57:59.130 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:59.119+0000 7f766d76d700 1 -- 172.21.15.77:0/695425334 shutdown_connections 2024-02-10T08:57:59.130 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:59.119+0000 7f766d76d700 1 --2- 172.21.15.77:0/695425334 >> [v2:172.21.15.77:6800/101381127,v1:172.21.15.77:6801/101381127] conn(0x7f765402fdc0 0x7f7654032280 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:57:59.130 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:59.119+0000 7f766d76d700 1 --2- 172.21.15.77:0/695425334 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76680703b0 0x7f766810cdd0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:57:59.130 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:59.119+0000 7f766d76d700 1 -- 172.21.15.77:0/695425334 >> 172.21.15.77:0/695425334 conn(0x7f76680f6530 msgr2=0x7f76680f7fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:57:59.130 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:59.119+0000 7f766d76d700 1 -- 172.21.15.77:0/695425334 shutdown_connections 2024-02-10T08:57:59.130 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:57:59.119+0000 7f766d76d700 1 -- 172.21.15.77:0/695425334 wait complete. 2024-02-10T08:57:59.734 INFO:teuthology.orchestra.run.smithi077.stderr:mgr is available 2024-02-10T08:57:59.734 INFO:teuthology.orchestra.run.smithi077.stderr:Enabling cephadm module... 2024-02-10T08:57:59.734 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpzzyygg7o:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpx5m65_qh:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable cephadm 2024-02-10T08:57:59.957 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:57:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:57:59.118707+0000 mon.smithi077 (mon.0) 24 : audit [DBG] from='client.? 172.21.15.77:0/695425334' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-10T08:58:00.233 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:00.229+0000 7f7688b5f700 1 Processor -- start 2024-02-10T08:58:00.233 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:00.229+0000 7f7688b5f700 1 -- start start 2024-02-10T08:58:00.233 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:00.229+0000 7f7688b5f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76840fad90 0x7f76840fb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:00.233 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:00.229+0000 7f7688b5f700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76840fb6a0 con 0x7f76840fad90 2024-02-10T08:58:00.233 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:00.230+0000 7f768259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76840fad90 0x7f76840fb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:00.234 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:00.230+0000 7f768259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76840fad90 0x7f76840fb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42572/0 (socket says 172.21.15.77:42572) 2024-02-10T08:58:00.234 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:00.230+0000 7f768259c700 1 -- 172.21.15.77:0/2478166155 learned_addr learned my addr 172.21.15.77:0/2478166155 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-10T08:58:00.234 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:00.230+0000 7f768259c700 1 -- 172.21.15.77:0/2478166155 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76840fb7e0 con 0x7f76840fad90 2024-02-10T08:58:00.234 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:00.230+0000 7f768259c700 1 --2- 172.21.15.77:0/2478166155 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76840fad90 0x7f76840fb1b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f767800b5f0 tx=0x7f7678006ee0).ready entity=mon.0 client_cookie=fa519cd99edeb7eb server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:00.234 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:00.230+0000 7f768159a700 1 -- 172.21.15.77:0/2478166155 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f767800d600 con 0x7f76840fad90 2024-02-10T08:58:00.234 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:00.230+0000 7f768159a700 1 -- 172.21.15.77:0/2478166155 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f767800dbf0 con 0x7f76840fad90 2024-02-10T08:58:00.234 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:00.230+0000 7f768159a700 1 -- 172.21.15.77:0/2478166155 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7678011cb0 con 0x7f76840fad90 2024-02-10T08:58:00.234 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:00.231+0000 7f7688b5f700 1 -- 172.21.15.77:0/2478166155 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76840fad90 msgr2=0x7f76840fb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:00.234 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:00.231+0000 7f7688b5f700 1 --2- 172.21.15.77:0/2478166155 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76840fad90 0x7f76840fb1b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f767800b5f0 tx=0x7f7678006ee0).stop 2024-02-10T08:58:00.234 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:00.231+0000 7f7688b5f700 1 -- 172.21.15.77:0/2478166155 shutdown_connections 2024-02-10T08:58:00.234 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:00.231+0000 7f7688b5f700 1 --2- 172.21.15.77:0/2478166155 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76840fad90 0x7f76840fb1b0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:00.234 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:00.231+0000 7f7688b5f700 1 -- 172.21.15.77:0/2478166155 >> 172.21.15.77:0/2478166155 conn(0x7f76840f6530 msgr2=0x7f76840f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:58:00.234 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:00.231+0000 7f7688b5f700 1 -- 172.21.15.77:0/2478166155 shutdown_connections 2024-02-10T08:58:00.235 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:00.231+0000 7f7688b5f700 1 -- 172.21.15.77:0/2478166155 wait complete. 2024-02-10T08:58:00.235 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:00.232+0000 7f7688b5f700 1 Processor -- start 2024-02-10T08:58:00.235 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:00.232+0000 7f7688b5f700 1 -- start start 2024-02-10T08:58:00.235 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:00.232+0000 7f7688b5f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76840fad90 0x7f768410d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:00.235 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:00.232+0000 7f7688b5f700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7684070470 con 0x7f76840fad90 2024-02-10T08:58:00.235 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:00.232+0000 7f768259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76840fad90 0x7f768410d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:00.235 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:00.232+0000 7f768259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76840fad90 0x7f768410d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42578/0 (socket says 172.21.15.77:42578) 2024-02-10T08:58:00.235 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:00.232+0000 7f768259c700 1 -- 172.21.15.77:0/3629125206 learned_addr learned my addr 172.21.15.77:0/3629125206 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-10T08:58:00.235 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:00.232+0000 7f768259c700 1 -- 172.21.15.77:0/3629125206 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f767800b200 con 0x7f76840fad90 2024-02-10T08:58:00.236 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:00.232+0000 7f768259c700 1 --2- 172.21.15.77:0/3629125206 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76840fad90 0x7f768410d800 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f7678011ea0 tx=0x7f7678006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:00.236 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:00.232+0000 7f7672ffd700 1 -- 172.21.15.77:0/3629125206 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f767801b570 con 0x7f76840fad90 2024-02-10T08:58:00.236 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:00.232+0000 7f7672ffd700 1 -- 172.21.15.77:0/3629125206 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f767801bb60 con 0x7f76840fad90 2024-02-10T08:58:00.236 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:00.232+0000 7f7688b5f700 1 -- 172.21.15.77:0/3629125206 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7684070670 con 0x7f76840fad90 2024-02-10T08:58:00.236 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:00.232+0000 7f7672ffd700 1 -- 172.21.15.77:0/3629125206 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f767801a8d0 con 0x7f76840fad90 2024-02-10T08:58:00.236 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:00.233+0000 7f7688b5f700 1 -- 172.21.15.77:0/3629125206 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f768410deb0 con 0x7f76840fad90 2024-02-10T08:58:00.236 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:00.233+0000 7f7672ffd700 1 -- 172.21.15.77:0/3629125206 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 4) v1 ==== 36109+0+0 (secure 0 0 0) 0x7f767800f070 con 0x7f76840fad90 2024-02-10T08:58:00.237 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:00.233+0000 7f7672ffd700 1 --2- 172.21.15.77:0/3629125206 >> [v2:172.21.15.77:6800/101381127,v1:172.21.15.77:6801/101381127] conn(0x7f766802fe10 0x7f76680322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:00.237 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:00.233+0000 7f7672ffd700 1 -- 172.21.15.77:0/3629125206 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f76780412d0 con 0x7f76840fad90 2024-02-10T08:58:00.237 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:00.233+0000 7f7681d9b700 1 --2- 172.21.15.77:0/3629125206 >> [v2:172.21.15.77:6800/101381127,v1:172.21.15.77:6801/101381127] conn(0x7f766802fe10 0x7f76680322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:00.237 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:00.234+0000 7f7681d9b700 1 --2- 172.21.15.77:0/3629125206 >> [v2:172.21.15.77:6800/101381127,v1:172.21.15.77:6801/101381127] conn(0x7f766802fe10 0x7f76680322d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f76740096a0 tx=0x7f7674006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:00.237 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:00.234+0000 7f7688b5f700 1 -- 172.21.15.77:0/3629125206 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7684045ad0 con 0x7f76840fad90 2024-02-10T08:58:00.239 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:00.238+0000 7f7672ffd700 1 -- 172.21.15.77:0/3629125206 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7678022b70 con 0x7f76840fad90 2024-02-10T08:58:00.423 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:00.422+0000 7f7688b5f700 1 -- 172.21.15.77:0/3629125206 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f7684027a00 con 0x7f76840fad90 2024-02-10T08:58:00.706 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:00.704+0000 7f7672ffd700 1 -- 172.21.15.77:0/3629125206 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f767801ab40 con 0x7f76840fad90 2024-02-10T08:58:00.706 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:00.705+0000 7f7672ffd700 1 -- 172.21.15.77:0/3629125206 <== mon.0 v2:172.21.15.77:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f76780239b0 con 0x7f76840fad90 2024-02-10T08:58:00.709 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:00.707+0000 7f7688b5f700 1 -- 172.21.15.77:0/3629125206 >> [v2:172.21.15.77:6800/101381127,v1:172.21.15.77:6801/101381127] conn(0x7f766802fe10 msgr2=0x7f76680322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:00.709 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:00.707+0000 7f7688b5f700 1 --2- 172.21.15.77:0/3629125206 >> [v2:172.21.15.77:6800/101381127,v1:172.21.15.77:6801/101381127] conn(0x7f766802fe10 0x7f76680322d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f76740096a0 tx=0x7f7674006b40).stop 2024-02-10T08:58:00.709 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:00.707+0000 7f7688b5f700 1 -- 172.21.15.77:0/3629125206 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76840fad90 msgr2=0x7f768410d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:00.709 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:00.707+0000 7f7688b5f700 1 --2- 172.21.15.77:0/3629125206 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76840fad90 0x7f768410d800 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f7678011ea0 tx=0x7f7678006ee0).stop 2024-02-10T08:58:00.709 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:00.707+0000 7f7688b5f700 1 -- 172.21.15.77:0/3629125206 shutdown_connections 2024-02-10T08:58:00.709 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:00.707+0000 7f7688b5f700 1 --2- 172.21.15.77:0/3629125206 >> [v2:172.21.15.77:6800/101381127,v1:172.21.15.77:6801/101381127] conn(0x7f766802fe10 0x7f76680322d0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:00.709 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:00.707+0000 7f7688b5f700 1 --2- 172.21.15.77:0/3629125206 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f76840fad90 0x7f768410d800 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:00.709 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:00.707+0000 7f7688b5f700 1 -- 172.21.15.77:0/3629125206 >> 172.21.15.77:0/3629125206 conn(0x7f76840f6530 msgr2=0x7f76840f7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:58:00.710 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:00.707+0000 7f7688b5f700 1 -- 172.21.15.77:0/3629125206 shutdown_connections 2024-02-10T08:58:00.710 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:00.707+0000 7f7688b5f700 1 -- 172.21.15.77:0/3629125206 wait complete. 2024-02-10T08:58:01.121 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:00 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:00.423760+0000 mon.smithi077 (mon.0) 25 : audit [INF] from='client.? 172.21.15.77:0/3629125206' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-02-10T08:58:01.473 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpzzyygg7o:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpx5m65_qh:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2024-02-10T08:58:01.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:01 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:00.706827+0000 mon.smithi077 (mon.0) 26 : audit [INF] from='client.? 172.21.15.77:0/3629125206' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-02-10T08:58:01.733 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:01 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T08:58:00.706880+0000 mon.smithi077 (mon.0) 27 : cluster [DBG] mgrmap e5: smithi077.dxntvw(active, since 6s) 2024-02-10T08:58:02.945 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:02.049+0000 7fcf98d22700 1 Processor -- start 2024-02-10T08:58:02.945 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:02.049+0000 7fcf98d22700 1 -- start start 2024-02-10T08:58:02.945 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:02.049+0000 7fcf98d22700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf94070430 0x7fcf940fea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:02.945 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:02.049+0000 7fcf98d22700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf940fef40 con 0x7fcf94070430 2024-02-10T08:58:02.945 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:02.049+0000 7fcf9259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf94070430 0x7fcf940fea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:02.945 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:02.049+0000 7fcf9259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf94070430 0x7fcf940fea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42596/0 (socket says 172.21.15.77:42596) 2024-02-10T08:58:02.945 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:02.049+0000 7fcf9259c700 1 -- 172.21.15.77:0/839138154 learned_addr learned my addr 172.21.15.77:0/839138154 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-10T08:58:02.945 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:02.050+0000 7fcf9259c700 1 -- 172.21.15.77:0/839138154 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf940ff080 con 0x7fcf94070430 2024-02-10T08:58:02.945 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:02.050+0000 7fcf9259c700 1 --2- 172.21.15.77:0/839138154 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf94070430 0x7fcf940fea50 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fcf8400b5f0 tx=0x7fcf84006ee0).ready entity=mon.0 client_cookie=793fcafcb151e8c0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:02.946 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:02.050+0000 7fcf9159a700 1 -- 172.21.15.77:0/839138154 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf8400d600 con 0x7fcf94070430 2024-02-10T08:58:02.946 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:02.050+0000 7fcf9159a700 1 -- 172.21.15.77:0/839138154 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fcf8400dbf0 con 0x7fcf94070430 2024-02-10T08:58:02.946 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:02.050+0000 7fcf9159a700 1 -- 172.21.15.77:0/839138154 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf84011cb0 con 0x7fcf94070430 2024-02-10T08:58:02.946 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:02.050+0000 7fcf98d22700 1 -- 172.21.15.77:0/839138154 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf94070430 msgr2=0x7fcf940fea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:02.946 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:02.050+0000 7fcf98d22700 1 --2- 172.21.15.77:0/839138154 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf94070430 0x7fcf940fea50 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fcf8400b5f0 tx=0x7fcf84006ee0).stop 2024-02-10T08:58:02.946 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:02.051+0000 7fcf98d22700 1 -- 172.21.15.77:0/839138154 shutdown_connections 2024-02-10T08:58:02.946 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:02.051+0000 7fcf98d22700 1 --2- 172.21.15.77:0/839138154 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf94070430 0x7fcf940fea50 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:02.946 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:02.051+0000 7fcf98d22700 1 -- 172.21.15.77:0/839138154 >> 172.21.15.77:0/839138154 conn(0x7fcf940f6530 msgr2=0x7fcf940f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:58:02.946 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:02.051+0000 7fcf98d22700 1 -- 172.21.15.77:0/839138154 shutdown_connections 2024-02-10T08:58:02.946 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:02.051+0000 7fcf98d22700 1 -- 172.21.15.77:0/839138154 wait complete. 2024-02-10T08:58:02.946 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:02.051+0000 7fcf98d22700 1 Processor -- start 2024-02-10T08:58:02.946 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:02.051+0000 7fcf98d22700 1 -- start start 2024-02-10T08:58:02.946 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:02.051+0000 7fcf98d22700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf94070430 0x7fcf940fc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:02.946 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:02.051+0000 7fcf98d22700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf940fa950 con 0x7fcf94070430 2024-02-10T08:58:02.946 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:02.051+0000 7fcf9259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf94070430 0x7fcf940fc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:02.947 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:02.052+0000 7fcf9259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf94070430 0x7fcf940fc4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42598/0 (socket says 172.21.15.77:42598) 2024-02-10T08:58:02.947 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:02.052+0000 7fcf9259c700 1 -- 172.21.15.77:0/2998845440 learned_addr learned my addr 172.21.15.77:0/2998845440 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-10T08:58:02.947 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:02.052+0000 7fcf9259c700 1 -- 172.21.15.77:0/2998845440 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf8400b200 con 0x7fcf94070430 2024-02-10T08:58:02.947 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:02.052+0000 7fcf9259c700 1 --2- 172.21.15.77:0/2998845440 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf94070430 0x7fcf940fc4d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7fcf84006800 tx=0x7fcf84006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:02.947 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:02.052+0000 7fcf82ffd700 1 -- 172.21.15.77:0/2998845440 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf8401b570 con 0x7fcf94070430 2024-02-10T08:58:02.947 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:02.052+0000 7fcf98d22700 1 -- 172.21.15.77:0/2998845440 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf940faaf0 con 0x7fcf94070430 2024-02-10T08:58:02.947 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:02.052+0000 7fcf82ffd700 1 -- 172.21.15.77:0/2998845440 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fcf8401bb60 con 0x7fcf94070430 2024-02-10T08:58:02.947 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:02.052+0000 7fcf82ffd700 1 -- 172.21.15.77:0/2998845440 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf8401a9f0 con 0x7fcf94070430 2024-02-10T08:58:02.947 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:02.052+0000 7fcf98d22700 1 -- 172.21.15.77:0/2998845440 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf940faf80 con 0x7fcf94070430 2024-02-10T08:58:02.947 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:02.053+0000 7fcf82ffd700 1 -- 172.21.15.77:0/2998845440 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fcf8400f070 con 0x7fcf94070430 2024-02-10T08:58:02.947 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:02.053+0000 7fcf82ffd700 1 --2- 172.21.15.77:0/2998845440 >> [v2:172.21.15.77:6800/101381127,v1:172.21.15.77:6801/101381127] conn(0x7fcf7802fe10 0x7fcf780322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:02.947 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:02.053+0000 7fcf98d22700 1 -- 172.21.15.77:0/2998845440 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf94045ad0 con 0x7fcf94070430 2024-02-10T08:58:02.947 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:02.053+0000 7fcf82ffd700 1 -- 172.21.15.77:0/2998845440 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fcf84041470 con 0x7fcf94070430 2024-02-10T08:58:02.947 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:02.053+0000 7fcf91d9b700 1 -- 172.21.15.77:0/2998845440 >> [v2:172.21.15.77:6800/101381127,v1:172.21.15.77:6801/101381127] conn(0x7fcf7802fe10 msgr2=0x7fcf780322d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6800/101381127 2024-02-10T08:58:02.948 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:02.053+0000 7fcf91d9b700 1 --2- 172.21.15.77:0/2998845440 >> [v2:172.21.15.77:6800/101381127,v1:172.21.15.77:6801/101381127] conn(0x7fcf7802fe10 0x7fcf780322d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-10T08:58:02.948 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:02.057+0000 7fcf82ffd700 1 -- 172.21.15.77:0/2998845440 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcf84016080 con 0x7fcf94070430 2024-02-10T08:58:02.948 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:02.254+0000 7fcf91d9b700 1 -- 172.21.15.77:0/2998845440 >> [v2:172.21.15.77:6800/101381127,v1:172.21.15.77:6801/101381127] conn(0x7fcf7802fe10 msgr2=0x7fcf780322d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6800/101381127 2024-02-10T08:58:02.948 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:02.254+0000 7fcf91d9b700 1 --2- 172.21.15.77:0/2998845440 >> [v2:172.21.15.77:6800/101381127,v1:172.21.15.77:6801/101381127] conn(0x7fcf7802fe10 0x7fcf780322d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-10T08:58:02.948 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:02.255+0000 7fcf98d22700 1 -- 172.21.15.77:0/2998845440 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fcf940ffed0 con 0x7fcf94070430 2024-02-10T08:58:02.948 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:02.256+0000 7fcf82ffd700 1 -- 172.21.15.77:0/2998845440 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7fcf84010360 con 0x7fcf94070430 2024-02-10T08:58:02.948 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:02.257+0000 7fcf98d22700 1 -- 172.21.15.77:0/2998845440 >> [v2:172.21.15.77:6800/101381127,v1:172.21.15.77:6801/101381127] conn(0x7fcf7802fe10 msgr2=0x7fcf780322d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-10T08:58:02.948 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:02.257+0000 7fcf98d22700 1 --2- 172.21.15.77:0/2998845440 >> [v2:172.21.15.77:6800/101381127,v1:172.21.15.77:6801/101381127] conn(0x7fcf7802fe10 0x7fcf780322d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-10T08:58:02.948 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:02.257+0000 7fcf98d22700 1 -- 172.21.15.77:0/2998845440 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf94070430 msgr2=0x7fcf940fc4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:02.948 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:02.257+0000 7fcf98d22700 1 --2- 172.21.15.77:0/2998845440 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf94070430 0x7fcf940fc4d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7fcf84006800 tx=0x7fcf84006ee0).stop 2024-02-10T08:58:02.948 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:02.257+0000 7fcf98d22700 1 -- 172.21.15.77:0/2998845440 shutdown_connections 2024-02-10T08:58:02.948 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:02.257+0000 7fcf98d22700 1 --2- 172.21.15.77:0/2998845440 >> [v2:172.21.15.77:6800/101381127,v1:172.21.15.77:6801/101381127] conn(0x7fcf7802fe10 0x7fcf780322d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-10T08:58:02.948 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:02.257+0000 7fcf98d22700 1 --2- 172.21.15.77:0/2998845440 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcf94070430 0x7fcf940fc4d0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:02.949 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:02.258+0000 7fcf98d22700 1 -- 172.21.15.77:0/2998845440 >> 172.21.15.77:0/2998845440 conn(0x7fcf940f6530 msgr2=0x7fcf940f7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:58:02.949 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:02.258+0000 7fcf98d22700 1 -- 172.21.15.77:0/2998845440 shutdown_connections 2024-02-10T08:58:02.949 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:02.258+0000 7fcf98d22700 1 -- 172.21.15.77:0/2998845440 wait complete. 2024-02-10T08:58:02.949 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: { 2024-02-10T08:58:02.949 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "epoch": 5, 2024-02-10T08:58:02.949 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "available": true, 2024-02-10T08:58:02.949 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "active_name": "smithi077.dxntvw", 2024-02-10T08:58:02.949 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_standby": 0 2024-02-10T08:58:02.949 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: } 2024-02-10T08:58:03.148 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:02 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:02.256271+0000 mon.smithi077 (mon.0) 28 : audit [DBG] from='client.? 172.21.15.77:0/2998845440' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-10T08:58:07.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:06 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T08:58:05.651226+0000 mon.smithi077 (mon.0) 29 : cluster [INF] Active manager daemon smithi077.dxntvw restarted 2024-02-10T08:58:07.150 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:06 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T08:58:05.661204+0000 mon.smithi077 (mon.0) 30 : cluster [INF] Activating manager daemon smithi077.dxntvw 2024-02-10T08:58:07.150 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:06 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T08:58:05.665223+0000 mon.smithi077 (mon.0) 31 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2024-02-10T08:58:07.150 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:06 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T08:58:05.715874+0000 mon.smithi077 (mon.0) 32 : cluster [DBG] mgrmap e6: smithi077.dxntvw(active, starting, since 0.0547578s) 2024-02-10T08:58:07.150 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:06 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:05.716616+0000 mon.smithi077 (mon.0) 33 : audit [DBG] from='mgr.14114 172.21.15.77:0/2499169851' entity='mgr.smithi077.dxntvw' cmd=[{"prefix": "mon metadata", "id": "smithi077"}]: dispatch 2024-02-10T08:58:07.150 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:06 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:05.717253+0000 mon.smithi077 (mon.0) 34 : audit [DBG] from='mgr.14114 172.21.15.77:0/2499169851' entity='mgr.smithi077.dxntvw' cmd=[{"prefix": "mgr metadata", "who": "smithi077.dxntvw", "id": "smithi077.dxntvw"}]: dispatch 2024-02-10T08:58:07.150 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:06 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:05.717914+0000 mon.smithi077 (mon.0) 35 : audit [DBG] from='mgr.14114 172.21.15.77:0/2499169851' entity='mgr.smithi077.dxntvw' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T08:58:07.150 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:06 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:05.718088+0000 mon.smithi077 (mon.0) 36 : audit [DBG] from='mgr.14114 172.21.15.77:0/2499169851' entity='mgr.smithi077.dxntvw' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T08:58:07.150 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:06 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:05.718201+0000 mon.smithi077 (mon.0) 37 : audit [DBG] from='mgr.14114 172.21.15.77:0/2499169851' entity='mgr.smithi077.dxntvw' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T08:58:07.150 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:06 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T08:58:05.731749+0000 mon.smithi077 (mon.0) 38 : cluster [INF] Manager daemon smithi077.dxntvw is now available 2024-02-10T08:58:07.150 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:06 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:05.744474+0000 mon.smithi077 (mon.0) 39 : audit [INF] from='mgr.14114 172.21.15.77:0/2499169851' entity='mgr.smithi077.dxntvw' 2024-02-10T08:58:07.150 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:06 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:05.747177+0000 mon.smithi077 (mon.0) 40 : audit [INF] from='mgr.14114 172.21.15.77:0/2499169851' entity='mgr.smithi077.dxntvw' 2024-02-10T08:58:07.151 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:06 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:05.749608+0000 mon.smithi077 (mon.0) 41 : audit [INF] from='mgr.14114 172.21.15.77:0/2499169851' entity='mgr.smithi077.dxntvw' 2024-02-10T08:58:07.151 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:06 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:05.837806+0000 mon.smithi077 (mon.0) 42 : audit [INF] from='mgr.14114 172.21.15.77:0/2499169851' entity='mgr.smithi077.dxntvw' 2024-02-10T08:58:07.151 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:06 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:05.838464+0000 mon.smithi077 (mon.0) 43 : audit [DBG] from='mgr.14114 172.21.15.77:0/2499169851' entity='mgr.smithi077.dxntvw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:58:07.151 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:06 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:05.839619+0000 mon.smithi077 (mon.0) 44 : audit [DBG] from='mgr.14114 172.21.15.77:0/2499169851' entity='mgr.smithi077.dxntvw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:58:07.151 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:06 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:05.841366+0000 mon.smithi077 (mon.0) 45 : audit [DBG] from='mgr.14114 172.21.15.77:0/2499169851' entity='mgr.smithi077.dxntvw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:58:07.151 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:06 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:05.851688+0000 mon.smithi077 (mon.0) 46 : audit [INF] from='mgr.14114 172.21.15.77:0/2499169851' entity='mgr.smithi077.dxntvw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi077.dxntvw/mirror_snapshot_schedule"}]: dispatch 2024-02-10T08:58:07.151 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:06 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:05.852829+0000 mon.smithi077 (mon.0) 47 : audit [INF] from='mgr.14114 172.21.15.77:0/2499169851' entity='mgr.smithi077.dxntvw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi077.dxntvw/trash_purge_schedule"}]: dispatch 2024-02-10T08:58:07.811 INFO:teuthology.orchestra.run.smithi077.stderr:Waiting for the mgr to restart... 2024-02-10T08:58:07.812 INFO:teuthology.orchestra.run.smithi077.stderr:Waiting for mgr epoch 5... 2024-02-10T08:58:07.812 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpzzyygg7o:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpx5m65_qh:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2024-02-10T08:58:08.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:07 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T08:58:06.720771+0000 mon.smithi077 (mon.0) 48 : cluster [DBG] mgrmap e7: smithi077.dxntvw(active, since 1.05965s) 2024-02-10T08:58:09.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:08 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T08:58:07.722293+0000 mon.smithi077 (mon.0) 49 : cluster [DBG] mgrmap e8: smithi077.dxntvw(active, since 2s) 2024-02-10T08:58:11.751 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:10.809+0000 7fba885c6700 1 Processor -- start 2024-02-10T08:58:11.751 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:10.809+0000 7fba885c6700 1 -- start start 2024-02-10T08:58:11.751 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:10.809+0000 7fba885c6700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba800724e0 0x7fba80070a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:11.751 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:10.809+0000 7fba885c6700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba80072900 con 0x7fba800724e0 2024-02-10T08:58:11.751 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:10.809+0000 7fba86362700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba800724e0 0x7fba80070a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:11.752 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:10.809+0000 7fba86362700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba800724e0 0x7fba80070a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54150/0 (socket says 172.21.15.77:54150) 2024-02-10T08:58:11.752 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:10.809+0000 7fba86362700 1 -- 172.21.15.77:0/1129935004 learned_addr learned my addr 172.21.15.77:0/1129935004 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-10T08:58:11.752 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:10.809+0000 7fba86362700 1 -- 172.21.15.77:0/1129935004 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba80070f70 con 0x7fba800724e0 2024-02-10T08:58:11.752 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:10.809+0000 7fba86362700 1 --2- 172.21.15.77:0/1129935004 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba800724e0 0x7fba80070a80 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fba7000b5f0 tx=0x7fba70006ee0).ready entity=mon.0 client_cookie=b3286f03fa81e81e server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:11.752 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:10.810+0000 7fba85360700 1 -- 172.21.15.77:0/1129935004 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba7000d600 con 0x7fba800724e0 2024-02-10T08:58:11.752 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:10.810+0000 7fba85360700 1 -- 172.21.15.77:0/1129935004 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fba7000dbf0 con 0x7fba800724e0 2024-02-10T08:58:11.752 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:10.810+0000 7fba85360700 1 -- 172.21.15.77:0/1129935004 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba70011cb0 con 0x7fba800724e0 2024-02-10T08:58:11.752 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:10.810+0000 7fba885c6700 1 -- 172.21.15.77:0/1129935004 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba800724e0 msgr2=0x7fba80070a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:11.752 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:10.810+0000 7fba885c6700 1 --2- 172.21.15.77:0/1129935004 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba800724e0 0x7fba80070a80 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fba7000b5f0 tx=0x7fba70006ee0).stop 2024-02-10T08:58:11.752 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:10.810+0000 7fba885c6700 1 -- 172.21.15.77:0/1129935004 shutdown_connections 2024-02-10T08:58:11.752 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:10.810+0000 7fba885c6700 1 --2- 172.21.15.77:0/1129935004 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba800724e0 0x7fba80070a80 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:11.752 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:10.810+0000 7fba885c6700 1 -- 172.21.15.77:0/1129935004 >> 172.21.15.77:0/1129935004 conn(0x7fba800f76a0 msgr2=0x7fba800f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:58:11.752 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:10.810+0000 7fba885c6700 1 -- 172.21.15.77:0/1129935004 shutdown_connections 2024-02-10T08:58:11.753 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:10.810+0000 7fba885c6700 1 -- 172.21.15.77:0/1129935004 wait complete. 2024-02-10T08:58:11.753 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:10.811+0000 7fba885c6700 1 Processor -- start 2024-02-10T08:58:11.753 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:10.811+0000 7fba885c6700 1 -- start start 2024-02-10T08:58:11.753 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:10.811+0000 7fba885c6700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba800724e0 0x7fba8010ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:11.753 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:10.811+0000 7fba885c6700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba8010d2f0 con 0x7fba800724e0 2024-02-10T08:58:11.753 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:10.811+0000 7fba86362700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba800724e0 0x7fba8010ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:11.753 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:10.811+0000 7fba86362700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba800724e0 0x7fba8010ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54154/0 (socket says 172.21.15.77:54154) 2024-02-10T08:58:11.753 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:10.811+0000 7fba86362700 1 -- 172.21.15.77:0/4201376220 learned_addr learned my addr 172.21.15.77:0/4201376220 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-10T08:58:11.753 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:10.812+0000 7fba86362700 1 -- 172.21.15.77:0/4201376220 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba7000b200 con 0x7fba800724e0 2024-02-10T08:58:11.753 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:10.812+0000 7fba86362700 1 --2- 172.21.15.77:0/4201376220 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba800724e0 0x7fba8010ce00 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7fba70006800 tx=0x7fba70006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:11.753 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:10.812+0000 7fba6effd700 1 -- 172.21.15.77:0/4201376220 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba7001b570 con 0x7fba800724e0 2024-02-10T08:58:11.753 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:10.812+0000 7fba6effd700 1 -- 172.21.15.77:0/4201376220 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fba7001bb60 con 0x7fba800724e0 2024-02-10T08:58:11.753 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:10.812+0000 7fba885c6700 1 -- 172.21.15.77:0/4201376220 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba8010d4f0 con 0x7fba800724e0 2024-02-10T08:58:11.753 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:10.812+0000 7fba6effd700 1 -- 172.21.15.77:0/4201376220 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba7001a9f0 con 0x7fba800724e0 2024-02-10T08:58:11.754 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:10.812+0000 7fba885c6700 1 -- 172.21.15.77:0/4201376220 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba8010e100 con 0x7fba800724e0 2024-02-10T08:58:11.754 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:10.813+0000 7fba6effd700 1 -- 172.21.15.77:0/4201376220 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fba7000f070 con 0x7fba800724e0 2024-02-10T08:58:11.754 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:10.813+0000 7fba6effd700 1 --2- 172.21.15.77:0/4201376220 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7fba6802fe70 0x7fba68032330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:11.754 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:10.813+0000 7fba6effd700 1 -- 172.21.15.77:0/4201376220 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fba70041630 con 0x7fba800724e0 2024-02-10T08:58:11.754 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:10.813+0000 7fba885c6700 1 -- 172.21.15.77:0/4201376220 --> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fba80059100 con 0x7fba6802fe70 2024-02-10T08:58:11.754 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:10.813+0000 7fba85b61700 1 --2- 172.21.15.77:0/4201376220 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7fba6802fe70 0x7fba68032330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:11.754 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:10.813+0000 7fba85b61700 1 --2- 172.21.15.77:0/4201376220 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7fba6802fe70 0x7fba68032330 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fba7c0096a0 tx=0x7fba7c006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:11.754 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:10.814+0000 7fba6effd700 1 -- 172.21.15.77:0/4201376220 <== mgr.14114 v2:172.21.15.77:6800/2974869056 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7fba80059100 con 0x7fba6802fe70 2024-02-10T08:58:11.754 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:10.819+0000 7fba885c6700 1 -- 172.21.15.77:0/4201376220 --> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fba8010d980 con 0x7fba6802fe70 2024-02-10T08:58:11.754 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:10.819+0000 7fba6effd700 1 -- 172.21.15.77:0/4201376220 <== mgr.14114 v2:172.21.15.77:6800/2974869056 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fba8010d980 con 0x7fba6802fe70 2024-02-10T08:58:11.754 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:10.820+0000 7fba885c6700 1 -- 172.21.15.77:0/4201376220 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7fba6802fe70 msgr2=0x7fba68032330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:11.755 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:10.820+0000 7fba885c6700 1 --2- 172.21.15.77:0/4201376220 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7fba6802fe70 0x7fba68032330 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fba7c0096a0 tx=0x7fba7c006b40).stop 2024-02-10T08:58:11.755 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:10.820+0000 7fba885c6700 1 -- 172.21.15.77:0/4201376220 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba800724e0 msgr2=0x7fba8010ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:11.755 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:10.820+0000 7fba885c6700 1 --2- 172.21.15.77:0/4201376220 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba800724e0 0x7fba8010ce00 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7fba70006800 tx=0x7fba70006ee0).stop 2024-02-10T08:58:11.755 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:10.820+0000 7fba885c6700 1 -- 172.21.15.77:0/4201376220 shutdown_connections 2024-02-10T08:58:11.755 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:10.820+0000 7fba885c6700 1 --2- 172.21.15.77:0/4201376220 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7fba6802fe70 0x7fba68032330 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:11.755 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:10.820+0000 7fba885c6700 1 --2- 172.21.15.77:0/4201376220 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fba800724e0 0x7fba8010ce00 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:11.755 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:10.820+0000 7fba885c6700 1 -- 172.21.15.77:0/4201376220 >> 172.21.15.77:0/4201376220 conn(0x7fba800f76a0 msgr2=0x7fba800f9040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:58:11.755 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:10.820+0000 7fba885c6700 1 -- 172.21.15.77:0/4201376220 shutdown_connections 2024-02-10T08:58:11.755 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:10.820+0000 7fba885c6700 1 -- 172.21.15.77:0/4201376220 wait complete. 2024-02-10T08:58:11.755 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: { 2024-02-10T08:58:11.756 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "mgrmap_epoch": 8, 2024-02-10T08:58:11.756 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "initialized": true 2024-02-10T08:58:11.756 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: } 2024-02-10T08:58:12.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:11 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:10.814106+0000 mgr.smithi077.dxntvw (mgr.14114) 1 : audit [DBG] from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-10T08:58:12.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:11 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:10.819988+0000 mgr.smithi077.dxntvw (mgr.14114) 2 : audit [DBG] from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-10T08:58:12.343 INFO:teuthology.orchestra.run.smithi077.stderr:mgr epoch 5 is available 2024-02-10T08:58:12.343 INFO:teuthology.orchestra.run.smithi077.stderr:Setting orchestrator backend to cephadm... 2024-02-10T08:58:12.343 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpzzyygg7o:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpx5m65_qh:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch set backend cephadm 2024-02-10T08:58:12.938 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:12.935+0000 7f9d7d518700 1 Processor -- start 2024-02-10T08:58:12.939 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:12.935+0000 7f9d7d518700 1 -- start start 2024-02-10T08:58:12.939 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:12.935+0000 7f9d7d518700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d780724e0 0x7f9d78070a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:12.939 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:12.936+0000 7f9d7d518700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d78072900 con 0x7f9d780724e0 2024-02-10T08:58:12.939 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:12.936+0000 7f9d76ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d780724e0 0x7f9d78070a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:12.939 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:12.936+0000 7f9d76ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d780724e0 0x7f9d78070a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54164/0 (socket says 172.21.15.77:54164) 2024-02-10T08:58:12.939 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:12.936+0000 7f9d76ffd700 1 -- 172.21.15.77:0/420372912 learned_addr learned my addr 172.21.15.77:0/420372912 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-10T08:58:12.939 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:12.936+0000 7f9d76ffd700 1 -- 172.21.15.77:0/420372912 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d78070f70 con 0x7f9d780724e0 2024-02-10T08:58:12.939 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:12.936+0000 7f9d76ffd700 1 --2- 172.21.15.77:0/420372912 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d780724e0 0x7f9d78070a80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f9d6800b5f0 tx=0x7f9d68006ee0).ready entity=mon.0 client_cookie=51e86426186dbf99 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:12.939 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:12.936+0000 7f9d75ffb700 1 -- 172.21.15.77:0/420372912 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d6800d600 con 0x7f9d780724e0 2024-02-10T08:58:12.939 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:12.936+0000 7f9d75ffb700 1 -- 172.21.15.77:0/420372912 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f9d6800dbf0 con 0x7f9d780724e0 2024-02-10T08:58:12.939 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:12.936+0000 7f9d75ffb700 1 -- 172.21.15.77:0/420372912 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d68011cb0 con 0x7f9d780724e0 2024-02-10T08:58:12.939 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:12.937+0000 7f9d7d518700 1 -- 172.21.15.77:0/420372912 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d780724e0 msgr2=0x7f9d78070a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:12.939 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:12.937+0000 7f9d7d518700 1 --2- 172.21.15.77:0/420372912 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d780724e0 0x7f9d78070a80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f9d6800b5f0 tx=0x7f9d68006ee0).stop 2024-02-10T08:58:12.940 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:12.937+0000 7f9d7d518700 1 -- 172.21.15.77:0/420372912 shutdown_connections 2024-02-10T08:58:12.940 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:12.937+0000 7f9d7d518700 1 --2- 172.21.15.77:0/420372912 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d780724e0 0x7f9d78070a80 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:12.940 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:12.937+0000 7f9d7d518700 1 -- 172.21.15.77:0/420372912 >> 172.21.15.77:0/420372912 conn(0x7f9d780f76a0 msgr2=0x7f9d780f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:58:12.940 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:12.937+0000 7f9d7d518700 1 -- 172.21.15.77:0/420372912 shutdown_connections 2024-02-10T08:58:12.940 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:12.937+0000 7f9d7d518700 1 -- 172.21.15.77:0/420372912 wait complete. 2024-02-10T08:58:12.940 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:12.938+0000 7f9d7d518700 1 Processor -- start 2024-02-10T08:58:12.940 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:12.938+0000 7f9d7d518700 1 -- start start 2024-02-10T08:58:12.940 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:12.938+0000 7f9d7d518700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d780724e0 0x7f9d7810ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:12.940 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:12.938+0000 7f9d7d518700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d7810d2f0 con 0x7f9d780724e0 2024-02-10T08:58:12.940 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:12.938+0000 7f9d76ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d780724e0 0x7f9d7810ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:12.940 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:12.938+0000 7f9d76ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d780724e0 0x7f9d7810ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54174/0 (socket says 172.21.15.77:54174) 2024-02-10T08:58:12.940 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:12.938+0000 7f9d76ffd700 1 -- 172.21.15.77:0/3170505529 learned_addr learned my addr 172.21.15.77:0/3170505529 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-10T08:58:12.941 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:12.938+0000 7f9d76ffd700 1 -- 172.21.15.77:0/3170505529 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d6800b200 con 0x7f9d780724e0 2024-02-10T08:58:12.942 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:12.938+0000 7f9d76ffd700 1 --2- 172.21.15.77:0/3170505529 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d780724e0 0x7f9d7810ce00 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f9d68006800 tx=0x7f9d68006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:12.942 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:12.938+0000 7f9d5f7fe700 1 -- 172.21.15.77:0/3170505529 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d6801b570 con 0x7f9d780724e0 2024-02-10T08:58:12.942 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:12.938+0000 7f9d7d518700 1 -- 172.21.15.77:0/3170505529 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d7810d4f0 con 0x7f9d780724e0 2024-02-10T08:58:12.942 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:12.938+0000 7f9d5f7fe700 1 -- 172.21.15.77:0/3170505529 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f9d6801bb60 con 0x7f9d780724e0 2024-02-10T08:58:12.942 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:12.938+0000 7f9d5f7fe700 1 -- 172.21.15.77:0/3170505529 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d6801a9f0 con 0x7f9d780724e0 2024-02-10T08:58:12.942 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:12.938+0000 7f9d7d518700 1 -- 172.21.15.77:0/3170505529 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d7810e100 con 0x7f9d780724e0 2024-02-10T08:58:12.942 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:12.939+0000 7f9d5f7fe700 1 -- 172.21.15.77:0/3170505529 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f9d6800f070 con 0x7f9d780724e0 2024-02-10T08:58:12.942 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:12.939+0000 7f9d5f7fe700 1 --2- 172.21.15.77:0/3170505529 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7f9d6402fe10 0x7f9d640322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:12.942 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:12.939+0000 7f9d5f7fe700 1 -- 172.21.15.77:0/3170505529 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f9d68041510 con 0x7f9d780724e0 2024-02-10T08:58:12.942 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:12.940+0000 7f9d767fc700 1 --2- 172.21.15.77:0/3170505529 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7f9d6402fe10 0x7f9d640322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:12.942 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:12.939+0000 7f9d7d518700 1 -- 172.21.15.77:0/3170505529 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d78059100 con 0x7f9d780724e0 2024-02-10T08:58:12.942 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:12.940+0000 7f9d767fc700 1 --2- 172.21.15.77:0/3170505529 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7f9d6402fe10 0x7f9d640322d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f9d6c0096a0 tx=0x7f9d6c006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:12.944 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:12.944+0000 7f9d5f7fe700 1 -- 172.21.15.77:0/3170505529 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9d68016070 con 0x7f9d780724e0 2024-02-10T08:58:13.120 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:13.119+0000 7f9d7d518700 1 -- 172.21.15.77:0/3170505529 --> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f9d781021d0 con 0x7f9d6402fe10 2024-02-10T08:58:13.125 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:13.124+0000 7f9d5f7fe700 1 -- 172.21.15.77:0/3170505529 <== mgr.14114 v2:172.21.15.77:6800/2974869056 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f9d781021d0 con 0x7f9d6402fe10 2024-02-10T08:58:13.127 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:13.126+0000 7f9d7d518700 1 -- 172.21.15.77:0/3170505529 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7f9d6402fe10 msgr2=0x7f9d640322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:13.128 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:13.126+0000 7f9d7d518700 1 --2- 172.21.15.77:0/3170505529 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7f9d6402fe10 0x7f9d640322d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f9d6c0096a0 tx=0x7f9d6c006b40).stop 2024-02-10T08:58:13.128 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:13.126+0000 7f9d7d518700 1 -- 172.21.15.77:0/3170505529 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d780724e0 msgr2=0x7f9d7810ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:13.128 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:13.126+0000 7f9d7d518700 1 --2- 172.21.15.77:0/3170505529 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d780724e0 0x7f9d7810ce00 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f9d68006800 tx=0x7f9d68006ee0).stop 2024-02-10T08:58:13.128 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:13.126+0000 7f9d7d518700 1 -- 172.21.15.77:0/3170505529 shutdown_connections 2024-02-10T08:58:13.128 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:13.126+0000 7f9d7d518700 1 --2- 172.21.15.77:0/3170505529 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7f9d6402fe10 0x7f9d640322d0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:13.128 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:13.126+0000 7f9d7d518700 1 --2- 172.21.15.77:0/3170505529 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d780724e0 0x7f9d7810ce00 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:13.128 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:13.126+0000 7f9d7d518700 1 -- 172.21.15.77:0/3170505529 >> 172.21.15.77:0/3170505529 conn(0x7f9d780f76a0 msgr2=0x7f9d780f9040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:58:13.128 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:13.126+0000 7f9d7d518700 1 -- 172.21.15.77:0/3170505529 shutdown_connections 2024-02-10T08:58:13.128 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:13.126+0000 7f9d7d518700 1 -- 172.21.15.77:0/3170505529 wait complete. 2024-02-10T08:58:13.816 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpzzyygg7o:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpx5m65_qh:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm set-user root 2024-02-10T08:58:14.385 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:14.381+0000 7fed8adc7700 1 Processor -- start 2024-02-10T08:58:14.385 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:14.381+0000 7fed8adc7700 1 -- start start 2024-02-10T08:58:14.385 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:14.381+0000 7fed8adc7700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed840fad90 0x7fed840fb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:14.385 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:14.381+0000 7fed8adc7700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed840fb6a0 con 0x7fed840fad90 2024-02-10T08:58:14.385 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:14.382+0000 7fed88b63700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed840fad90 0x7fed840fb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:14.386 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:14.382+0000 7fed88b63700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed840fad90 0x7fed840fb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54190/0 (socket says 172.21.15.77:54190) 2024-02-10T08:58:14.386 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:14.382+0000 7fed88b63700 1 -- 172.21.15.77:0/3570254433 learned_addr learned my addr 172.21.15.77:0/3570254433 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-10T08:58:14.386 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:14.382+0000 7fed88b63700 1 -- 172.21.15.77:0/3570254433 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed840fb7e0 con 0x7fed840fad90 2024-02-10T08:58:14.386 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:14.382+0000 7fed88b63700 1 --2- 172.21.15.77:0/3570254433 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed840fad90 0x7fed840fb1b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fed7800b5f0 tx=0x7fed78006ee0).ready entity=mon.0 client_cookie=1f3d0192bf710f30 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:14.386 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:14.382+0000 7fed837fe700 1 -- 172.21.15.77:0/3570254433 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed7800d600 con 0x7fed840fad90 2024-02-10T08:58:14.386 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:14.382+0000 7fed837fe700 1 -- 172.21.15.77:0/3570254433 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fed7800dbf0 con 0x7fed840fad90 2024-02-10T08:58:14.386 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:14.382+0000 7fed837fe700 1 -- 172.21.15.77:0/3570254433 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed78011cb0 con 0x7fed840fad90 2024-02-10T08:58:14.386 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:14.383+0000 7fed8adc7700 1 -- 172.21.15.77:0/3570254433 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed840fad90 msgr2=0x7fed840fb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:14.386 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:14.383+0000 7fed8adc7700 1 --2- 172.21.15.77:0/3570254433 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed840fad90 0x7fed840fb1b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fed7800b5f0 tx=0x7fed78006ee0).stop 2024-02-10T08:58:14.386 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:14.383+0000 7fed8adc7700 1 -- 172.21.15.77:0/3570254433 shutdown_connections 2024-02-10T08:58:14.386 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:14.383+0000 7fed8adc7700 1 --2- 172.21.15.77:0/3570254433 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed840fad90 0x7fed840fb1b0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:14.386 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:14.383+0000 7fed8adc7700 1 -- 172.21.15.77:0/3570254433 >> 172.21.15.77:0/3570254433 conn(0x7fed840f6530 msgr2=0x7fed840f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:58:14.386 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:14.383+0000 7fed8adc7700 1 -- 172.21.15.77:0/3570254433 shutdown_connections 2024-02-10T08:58:14.386 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:14.383+0000 7fed8adc7700 1 -- 172.21.15.77:0/3570254433 wait complete. 2024-02-10T08:58:14.386 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:14.383+0000 7fed8adc7700 1 Processor -- start 2024-02-10T08:58:14.387 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:14.383+0000 7fed8adc7700 1 -- start start 2024-02-10T08:58:14.387 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:14.384+0000 7fed8adc7700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed840fad90 0x7fed8410cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:14.387 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:14.384+0000 7fed8adc7700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed8410d280 con 0x7fed840fad90 2024-02-10T08:58:14.387 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:14.384+0000 7fed88b63700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed840fad90 0x7fed8410cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:14.387 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:14.384+0000 7fed88b63700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed840fad90 0x7fed8410cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54204/0 (socket says 172.21.15.77:54204) 2024-02-10T08:58:14.387 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:14.384+0000 7fed88b63700 1 -- 172.21.15.77:0/3980472894 learned_addr learned my addr 172.21.15.77:0/3980472894 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-10T08:58:14.388 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:14.384+0000 7fed88b63700 1 -- 172.21.15.77:0/3980472894 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed7800b200 con 0x7fed840fad90 2024-02-10T08:58:14.388 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:14.384+0000 7fed88b63700 1 --2- 172.21.15.77:0/3980472894 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed840fad90 0x7fed8410cd90 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fed78011ea0 tx=0x7fed78006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:14.388 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:14.384+0000 7fed817fa700 1 -- 172.21.15.77:0/3980472894 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed7801b570 con 0x7fed840fad90 2024-02-10T08:58:14.388 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:14.384+0000 7fed8adc7700 1 -- 172.21.15.77:0/3980472894 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed8410d480 con 0x7fed840fad90 2024-02-10T08:58:14.388 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:14.384+0000 7fed817fa700 1 -- 172.21.15.77:0/3980472894 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fed7801bb60 con 0x7fed840fad90 2024-02-10T08:58:14.388 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:14.384+0000 7fed817fa700 1 -- 172.21.15.77:0/3980472894 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed7801a8d0 con 0x7fed840fad90 2024-02-10T08:58:14.388 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:14.384+0000 7fed8adc7700 1 -- 172.21.15.77:0/3980472894 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed8410e090 con 0x7fed840fad90 2024-02-10T08:58:14.388 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:14.385+0000 7fed817fa700 1 -- 172.21.15.77:0/3980472894 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fed7800f070 con 0x7fed840fad90 2024-02-10T08:58:14.389 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:14.385+0000 7fed817fa700 1 --2- 172.21.15.77:0/3980472894 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7fed6802fe10 0x7fed680322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:14.389 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:14.385+0000 7fed8adc7700 1 -- 172.21.15.77:0/3980472894 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed84059100 con 0x7fed840fad90 2024-02-10T08:58:14.389 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:14.385+0000 7fed817fa700 1 -- 172.21.15.77:0/3980472894 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fed78041460 con 0x7fed840fad90 2024-02-10T08:58:14.389 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:14.386+0000 7fed83fff700 1 --2- 172.21.15.77:0/3980472894 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7fed6802fe10 0x7fed680322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:14.389 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:14.386+0000 7fed83fff700 1 --2- 172.21.15.77:0/3980472894 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7fed6802fe10 0x7fed680322d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fed700096a0 tx=0x7fed70006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:14.390 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:14.390+0000 7fed817fa700 1 -- 172.21.15.77:0/3980472894 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fed78022b70 con 0x7fed840fad90 2024-02-10T08:58:14.398 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:14 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:13.121129+0000 mgr.smithi077.dxntvw (mgr.14114) 3 : audit [DBG] from='client.14124 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:58:14.398 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:14 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:13.125196+0000 mon.smithi077 (mon.0) 50 : audit [INF] from='mgr.14114 172.21.15.77:0/2499169851' entity='mgr.smithi077.dxntvw' 2024-02-10T08:58:14.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:14 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:13.129154+0000 mon.smithi077 (mon.0) 51 : audit [DBG] from='mgr.14114 172.21.15.77:0/2499169851' entity='mgr.smithi077.dxntvw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:58:14.540 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:14.539+0000 7fed8adc7700 1 -- 172.21.15.77:0/3980472894 --> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7fed8410e390 con 0x7fed6802fe10 2024-02-10T08:58:14.541 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:14.539+0000 7fed817fa700 1 -- 172.21.15.77:0/3980472894 <== mgr.14114 v2:172.21.15.77:6800/2974869056 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7fed8410e390 con 0x7fed6802fe10 2024-02-10T08:58:14.541 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: value unchanged 2024-02-10T08:58:14.543 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:14.541+0000 7fed8adc7700 1 -- 172.21.15.77:0/3980472894 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7fed6802fe10 msgr2=0x7fed680322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:14.543 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:14.541+0000 7fed8adc7700 1 --2- 172.21.15.77:0/3980472894 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7fed6802fe10 0x7fed680322d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fed700096a0 tx=0x7fed70006b40).stop 2024-02-10T08:58:14.543 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:14.541+0000 7fed8adc7700 1 -- 172.21.15.77:0/3980472894 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed840fad90 msgr2=0x7fed8410cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:14.543 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:14.541+0000 7fed8adc7700 1 --2- 172.21.15.77:0/3980472894 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed840fad90 0x7fed8410cd90 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fed78011ea0 tx=0x7fed78006ee0).stop 2024-02-10T08:58:14.543 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:14.541+0000 7fed8adc7700 1 -- 172.21.15.77:0/3980472894 shutdown_connections 2024-02-10T08:58:14.543 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:14.541+0000 7fed8adc7700 1 --2- 172.21.15.77:0/3980472894 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7fed6802fe10 0x7fed680322d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:14.543 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:14.541+0000 7fed8adc7700 1 --2- 172.21.15.77:0/3980472894 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed840fad90 0x7fed8410cd90 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:14.543 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:14.541+0000 7fed8adc7700 1 -- 172.21.15.77:0/3980472894 >> 172.21.15.77:0/3980472894 conn(0x7fed840f6530 msgr2=0x7fed840700d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:58:14.543 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:14.542+0000 7fed8adc7700 1 -- 172.21.15.77:0/3980472894 shutdown_connections 2024-02-10T08:58:14.544 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:14.542+0000 7fed8adc7700 1 -- 172.21.15.77:0/3980472894 wait complete. 2024-02-10T08:58:15.171 INFO:teuthology.orchestra.run.smithi077.stderr:Generating ssh key... 2024-02-10T08:58:15.171 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpzzyygg7o:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpx5m65_qh:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm generate-key 2024-02-10T08:58:15.398 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:15 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:14.541088+0000 mgr.smithi077.dxntvw (mgr.14114) 4 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:58:15.725 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:15.720+0000 7fe595a22700 1 Processor -- start 2024-02-10T08:58:15.726 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:15.721+0000 7fe595a22700 1 -- start start 2024-02-10T08:58:15.726 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:15.721+0000 7fe595a22700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5900703e0 0x7fe5900febc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:15.726 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:15.721+0000 7fe595a22700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe590070800 con 0x7fe5900703e0 2024-02-10T08:58:15.726 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:15.721+0000 7fe58effd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5900703e0 0x7fe5900febc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:15.726 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:15.721+0000 7fe58effd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5900703e0 0x7fe5900febc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54216/0 (socket says 172.21.15.77:54216) 2024-02-10T08:58:15.726 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:15.721+0000 7fe58effd700 1 -- 172.21.15.77:0/3820314858 learned_addr learned my addr 172.21.15.77:0/3820314858 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-10T08:58:15.726 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:15.721+0000 7fe58effd700 1 -- 172.21.15.77:0/3820314858 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5900ff0b0 con 0x7fe5900703e0 2024-02-10T08:58:15.726 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:15.721+0000 7fe58effd700 1 --2- 172.21.15.77:0/3820314858 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5900703e0 0x7fe5900febc0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fe58400b5f0 tx=0x7fe584006ee0).ready entity=mon.0 client_cookie=c0cc5a16cd2ee441 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:15.726 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:15.721+0000 7fe58dffb700 1 -- 172.21.15.77:0/3820314858 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe58400d600 con 0x7fe5900703e0 2024-02-10T08:58:15.726 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:15.721+0000 7fe58dffb700 1 -- 172.21.15.77:0/3820314858 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fe58400dbf0 con 0x7fe5900703e0 2024-02-10T08:58:15.726 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:15.721+0000 7fe58dffb700 1 -- 172.21.15.77:0/3820314858 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe584011cb0 con 0x7fe5900703e0 2024-02-10T08:58:15.726 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:15.722+0000 7fe595a22700 1 -- 172.21.15.77:0/3820314858 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5900703e0 msgr2=0x7fe5900febc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:15.726 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:15.722+0000 7fe595a22700 1 --2- 172.21.15.77:0/3820314858 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5900703e0 0x7fe5900febc0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fe58400b5f0 tx=0x7fe584006ee0).stop 2024-02-10T08:58:15.726 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:15.722+0000 7fe595a22700 1 -- 172.21.15.77:0/3820314858 shutdown_connections 2024-02-10T08:58:15.727 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:15.722+0000 7fe595a22700 1 --2- 172.21.15.77:0/3820314858 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5900703e0 0x7fe5900febc0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:15.727 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:15.722+0000 7fe595a22700 1 -- 172.21.15.77:0/3820314858 >> 172.21.15.77:0/3820314858 conn(0x7fe5900f65c0 msgr2=0x7fe5900f89e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:58:15.727 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:15.722+0000 7fe595a22700 1 -- 172.21.15.77:0/3820314858 shutdown_connections 2024-02-10T08:58:15.727 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:15.722+0000 7fe595a22700 1 -- 172.21.15.77:0/3820314858 wait complete. 2024-02-10T08:58:15.727 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:15.723+0000 7fe595a22700 1 Processor -- start 2024-02-10T08:58:15.727 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:15.723+0000 7fe595a22700 1 -- start start 2024-02-10T08:58:15.727 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:15.723+0000 7fe595a22700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5900703e0 0x7fe59010abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:15.727 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:15.723+0000 7fe595a22700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe59010b0c0 con 0x7fe5900703e0 2024-02-10T08:58:15.727 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:15.723+0000 7fe58effd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5900703e0 0x7fe59010abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:15.727 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:15.723+0000 7fe58effd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5900703e0 0x7fe59010abd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54226/0 (socket says 172.21.15.77:54226) 2024-02-10T08:58:15.727 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:15.723+0000 7fe58effd700 1 -- 172.21.15.77:0/2358578609 learned_addr learned my addr 172.21.15.77:0/2358578609 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-10T08:58:15.727 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:15.723+0000 7fe58effd700 1 -- 172.21.15.77:0/2358578609 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe58400b200 con 0x7fe5900703e0 2024-02-10T08:58:15.727 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:15.723+0000 7fe58effd700 1 --2- 172.21.15.77:0/2358578609 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5900703e0 0x7fe59010abd0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fe58400dfc0 tx=0x7fe584006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:15.727 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:15.723+0000 7fe5777fe700 1 -- 172.21.15.77:0/2358578609 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe58401b410 con 0x7fe5900703e0 2024-02-10T08:58:15.727 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:15.723+0000 7fe5777fe700 1 -- 172.21.15.77:0/2358578609 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fe58401ba00 con 0x7fe5900703e0 2024-02-10T08:58:15.728 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:15.724+0000 7fe595a22700 1 -- 172.21.15.77:0/2358578609 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe59010b2c0 con 0x7fe5900703e0 2024-02-10T08:58:15.728 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:15.724+0000 7fe5777fe700 1 -- 172.21.15.77:0/2358578609 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe58401a990 con 0x7fe5900703e0 2024-02-10T08:58:15.728 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:15.724+0000 7fe595a22700 1 -- 172.21.15.77:0/2358578609 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe59010b750 con 0x7fe5900703e0 2024-02-10T08:58:15.728 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:15.724+0000 7fe5777fe700 1 -- 172.21.15.77:0/2358578609 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fe58400f070 con 0x7fe5900703e0 2024-02-10T08:58:15.728 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:15.724+0000 7fe595a22700 1 -- 172.21.15.77:0/2358578609 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe590045ad0 con 0x7fe5900703e0 2024-02-10T08:58:15.728 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:15.725+0000 7fe5777fe700 1 --2- 172.21.15.77:0/2358578609 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7fe57c02fe10 0x7fe57c0322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:15.728 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:15.725+0000 7fe5777fe700 1 -- 172.21.15.77:0/2358578609 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fe584041310 con 0x7fe5900703e0 2024-02-10T08:58:15.729 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:15.725+0000 7fe58e7fc700 1 --2- 172.21.15.77:0/2358578609 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7fe57c02fe10 0x7fe57c0322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:15.729 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:15.725+0000 7fe58e7fc700 1 --2- 172.21.15.77:0/2358578609 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7fe57c02fe10 0x7fe57c0322d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fe5800096a0 tx=0x7fe580006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:15.730 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:15.729+0000 7fe5777fe700 1 -- 172.21.15.77:0/2358578609 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe584022310 con 0x7fe5900703e0 2024-02-10T08:58:15.878 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:15.877+0000 7fe595a22700 1 -- 172.21.15.77:0/2358578609 --> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fe59010c000 con 0x7fe57c02fe10 2024-02-10T08:58:15.992 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:15.991+0000 7fe5777fe700 1 -- 172.21.15.77:0/2358578609 <== mgr.14114 v2:172.21.15.77:6800/2974869056 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe59010c000 con 0x7fe57c02fe10 2024-02-10T08:58:15.994 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:15.992+0000 7fe595a22700 1 -- 172.21.15.77:0/2358578609 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7fe57c02fe10 msgr2=0x7fe57c0322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:15.994 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:15.992+0000 7fe595a22700 1 --2- 172.21.15.77:0/2358578609 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7fe57c02fe10 0x7fe57c0322d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fe5800096a0 tx=0x7fe580006b40).stop 2024-02-10T08:58:15.994 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:15.992+0000 7fe595a22700 1 -- 172.21.15.77:0/2358578609 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5900703e0 msgr2=0x7fe59010abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:15.994 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:15.992+0000 7fe595a22700 1 --2- 172.21.15.77:0/2358578609 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5900703e0 0x7fe59010abd0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fe58400dfc0 tx=0x7fe584006ee0).stop 2024-02-10T08:58:15.994 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:15.992+0000 7fe595a22700 1 -- 172.21.15.77:0/2358578609 shutdown_connections 2024-02-10T08:58:15.995 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:15.992+0000 7fe595a22700 1 --2- 172.21.15.77:0/2358578609 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7fe57c02fe10 0x7fe57c0322d0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:15.995 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:15.992+0000 7fe595a22700 1 --2- 172.21.15.77:0/2358578609 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5900703e0 0x7fe59010abd0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:15.995 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:15.992+0000 7fe595a22700 1 -- 172.21.15.77:0/2358578609 >> 172.21.15.77:0/2358578609 conn(0x7fe5900f65c0 msgr2=0x7fe5900f7fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:58:15.995 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:15.992+0000 7fe595a22700 1 -- 172.21.15.77:0/2358578609 shutdown_connections 2024-02-10T08:58:15.995 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:15.992+0000 7fe595a22700 1 -- 172.21.15.77:0/2358578609 wait complete. 2024-02-10T08:58:16.628 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpzzyygg7o:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpx5m65_qh:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm get-pub-key 2024-02-10T08:58:17.246 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:17.243+0000 7efd65490700 1 Processor -- start 2024-02-10T08:58:17.246 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:17.243+0000 7efd65490700 1 -- start start 2024-02-10T08:58:17.246 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:17.243+0000 7efd65490700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd600fbea0 0x7efd600fc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:17.246 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:17.243+0000 7efd65490700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd600fc7b0 con 0x7efd600fbea0 2024-02-10T08:58:17.246 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:17.244+0000 7efd5effd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd600fbea0 0x7efd600fc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:17.246 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:17.244+0000 7efd5effd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd600fbea0 0x7efd600fc2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54228/0 (socket says 172.21.15.77:54228) 2024-02-10T08:58:17.247 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:17.244+0000 7efd5effd700 1 -- 172.21.15.77:0/261031027 learned_addr learned my addr 172.21.15.77:0/261031027 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-10T08:58:17.247 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:17.244+0000 7efd5effd700 1 -- 172.21.15.77:0/261031027 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd600fc8f0 con 0x7efd600fbea0 2024-02-10T08:58:17.247 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:17.244+0000 7efd5effd700 1 --2- 172.21.15.77:0/261031027 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd600fbea0 0x7efd600fc2c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7efd4800b5f0 tx=0x7efd48006ee0).ready entity=mon.0 client_cookie=e7b3fc1f940ebf69 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:17.247 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:17.244+0000 7efd5dffb700 1 -- 172.21.15.77:0/261031027 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efd4800d600 con 0x7efd600fbea0 2024-02-10T08:58:17.247 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:17.244+0000 7efd5dffb700 1 -- 172.21.15.77:0/261031027 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7efd4800dbf0 con 0x7efd600fbea0 2024-02-10T08:58:17.247 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:17.244+0000 7efd5dffb700 1 -- 172.21.15.77:0/261031027 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efd48011cb0 con 0x7efd600fbea0 2024-02-10T08:58:17.247 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:17.245+0000 7efd65490700 1 -- 172.21.15.77:0/261031027 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd600fbea0 msgr2=0x7efd600fc2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:17.247 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:17.245+0000 7efd65490700 1 --2- 172.21.15.77:0/261031027 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd600fbea0 0x7efd600fc2c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7efd4800b5f0 tx=0x7efd48006ee0).stop 2024-02-10T08:58:17.247 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:17.245+0000 7efd65490700 1 -- 172.21.15.77:0/261031027 shutdown_connections 2024-02-10T08:58:17.250 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:17.245+0000 7efd65490700 1 --2- 172.21.15.77:0/261031027 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd600fbea0 0x7efd600fc2c0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:17.250 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:17.245+0000 7efd65490700 1 -- 172.21.15.77:0/261031027 >> 172.21.15.77:0/261031027 conn(0x7efd600f76a0 msgr2=0x7efd600f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:58:17.250 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:17.245+0000 7efd65490700 1 -- 172.21.15.77:0/261031027 shutdown_connections 2024-02-10T08:58:17.250 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:17.245+0000 7efd65490700 1 -- 172.21.15.77:0/261031027 wait complete. 2024-02-10T08:58:17.250 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:17.246+0000 7efd65490700 1 Processor -- start 2024-02-10T08:58:17.250 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:17.246+0000 7efd65490700 1 -- start start 2024-02-10T08:58:17.250 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:17.246+0000 7efd65490700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd600fbea0 0x7efd6010ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:17.250 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:17.246+0000 7efd65490700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd6010b080 con 0x7efd600fbea0 2024-02-10T08:58:17.250 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:17.246+0000 7efd5effd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd600fbea0 0x7efd6010ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:17.251 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:17.246+0000 7efd5effd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd600fbea0 0x7efd6010ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54244/0 (socket says 172.21.15.77:54244) 2024-02-10T08:58:17.251 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:17.246+0000 7efd5effd700 1 -- 172.21.15.77:0/1461922929 learned_addr learned my addr 172.21.15.77:0/1461922929 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-10T08:58:17.251 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:17.247+0000 7efd5effd700 1 -- 172.21.15.77:0/1461922929 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd4800b200 con 0x7efd600fbea0 2024-02-10T08:58:17.251 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:17.247+0000 7efd5effd700 1 --2- 172.21.15.77:0/1461922929 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd600fbea0 0x7efd6010ab90 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7efd4800bf40 tx=0x7efd48006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:17.251 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:17.247+0000 7efd577fe700 1 -- 172.21.15.77:0/1461922929 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efd4801b570 con 0x7efd600fbea0 2024-02-10T08:58:17.251 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:17.247+0000 7efd65490700 1 -- 172.21.15.77:0/1461922929 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd6010b280 con 0x7efd600fbea0 2024-02-10T08:58:17.251 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:17.247+0000 7efd577fe700 1 -- 172.21.15.77:0/1461922929 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7efd4801bb60 con 0x7efd600fbea0 2024-02-10T08:58:17.251 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:17.247+0000 7efd577fe700 1 -- 172.21.15.77:0/1461922929 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efd4801aa20 con 0x7efd600fbea0 2024-02-10T08:58:17.251 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:17.247+0000 7efd65490700 1 -- 172.21.15.77:0/1461922929 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd6010b710 con 0x7efd600fbea0 2024-02-10T08:58:17.251 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:17.248+0000 7efd577fe700 1 -- 172.21.15.77:0/1461922929 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7efd4800f070 con 0x7efd600fbea0 2024-02-10T08:58:17.251 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:17.248+0000 7efd577fe700 1 --2- 172.21.15.77:0/1461922929 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7efd4002fe10 0x7efd400322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:17.251 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:17.248+0000 7efd577fe700 1 -- 172.21.15.77:0/1461922929 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7efd480414e0 con 0x7efd600fbea0 2024-02-10T08:58:17.251 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:17.248+0000 7efd5e7fc700 1 --2- 172.21.15.77:0/1461922929 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7efd4002fe10 0x7efd400322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:17.251 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:17.248+0000 7efd65490700 1 -- 172.21.15.77:0/1461922929 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd60045ad0 con 0x7efd600fbea0 2024-02-10T08:58:17.251 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:17.248+0000 7efd5e7fc700 1 --2- 172.21.15.77:0/1461922929 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7efd4002fe10 0x7efd400322d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7efd500096a0 tx=0x7efd50006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:17.253 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:17.252+0000 7efd577fe700 1 -- 172.21.15.77:0/1461922929 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efd48016070 con 0x7efd600fbea0 2024-02-10T08:58:17.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:16 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cephadm 2024-02-10T08:58:15.879388+0000 mgr.smithi077.dxntvw (mgr.14114) 6 : cephadm [INF] Generating ssh key... 2024-02-10T08:58:17.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:16 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:15.988967+0000 mon.smithi077 (mon.0) 52 : audit [INF] from='mgr.14114 172.21.15.77:0/2499169851' entity='mgr.smithi077.dxntvw' 2024-02-10T08:58:17.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:16 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:15.991490+0000 mon.smithi077 (mon.0) 53 : audit [INF] from='mgr.14114 172.21.15.77:0/2499169851' entity='mgr.smithi077.dxntvw' 2024-02-10T08:58:17.405 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:17.404+0000 7efd65490700 1 -- 172.21.15.77:0/1461922929 --> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7efd60104c80 con 0x7efd4002fe10 2024-02-10T08:58:17.406 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:17.404+0000 7efd577fe700 1 -- 172.21.15.77:0/1461922929 <== mgr.14114 v2:172.21.15.77:6800/2974869056 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7efd60104c80 con 0x7efd4002fe10 2024-02-10T08:58:17.406 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDV3XILkUclmDRAU4yCQDRVwgwrV6pZ3hfbzzMuXfddDui8gFX8bu+wpWRC74vg9ZkO0KKWbpD3qWql9QgtyhCZTOJnmQyF/69EwbX5I7xLEs1UW2Rh2P639kMfhCJWdIMYieAqCSKZl+GbfdIh5ZORxYDnDQxSWI6i0VvnhVnlURFdHCSAAjlkRxF2qf6EhfzJzrr9cjZv6nDabYoYButG5G7rhMxQDddFqoQANwB6w7xHk3Ec+1eyC8L/uAEwhNXObj94W4js38qQj/pFpasf9ctVl4PyFJ0f9CqMgd2MssEqGlbAvga+bFVDfE3Jzn8MH1kiOasSQiaZ6iPU32Xn7MbksFD/n+a6CDQawAMhaKjVaDInGvNTE22imoZFq+RUfKJKDlBEXM/0g8ddRNwxNjA32WTIovpa7DnW7b5mZUoZ4r+kJHTSpVu7f5GcHPkkeqljIUlUwRLd5wCtEgwOE7NBDfSi1uD0BVFCryays2EJ4g8NlhAbS3dyQFHSTXk= ceph-6081b54a-c7f2-11ee-95b8-87774f69a715 2024-02-10T08:58:17.408 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:17.406+0000 7efd65490700 1 -- 172.21.15.77:0/1461922929 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7efd4002fe10 msgr2=0x7efd400322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:17.408 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:17.406+0000 7efd65490700 1 --2- 172.21.15.77:0/1461922929 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7efd4002fe10 0x7efd400322d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7efd500096a0 tx=0x7efd50006b40).stop 2024-02-10T08:58:17.408 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:17.406+0000 7efd65490700 1 -- 172.21.15.77:0/1461922929 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd600fbea0 msgr2=0x7efd6010ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:17.408 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:17.406+0000 7efd65490700 1 --2- 172.21.15.77:0/1461922929 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd600fbea0 0x7efd6010ab90 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7efd4800bf40 tx=0x7efd48006ee0).stop 2024-02-10T08:58:17.408 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:17.406+0000 7efd65490700 1 -- 172.21.15.77:0/1461922929 shutdown_connections 2024-02-10T08:58:17.408 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:17.406+0000 7efd65490700 1 --2- 172.21.15.77:0/1461922929 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7efd4002fe10 0x7efd400322d0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:17.408 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:17.406+0000 7efd65490700 1 --2- 172.21.15.77:0/1461922929 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd600fbea0 0x7efd6010ab90 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:17.408 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:17.406+0000 7efd65490700 1 -- 172.21.15.77:0/1461922929 >> 172.21.15.77:0/1461922929 conn(0x7efd600f76a0 msgr2=0x7efd600f8fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:58:17.408 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:17.407+0000 7efd65490700 1 -- 172.21.15.77:0/1461922929 shutdown_connections 2024-02-10T08:58:17.408 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:17.407+0000 7efd65490700 1 -- 172.21.15.77:0/1461922929 wait complete. 2024-02-10T08:58:18.068 INFO:teuthology.orchestra.run.smithi077.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-02-10T08:58:18.068 INFO:teuthology.orchestra.run.smithi077.stderr:Adding key to root@localhost authorized_keys... 2024-02-10T08:58:18.069 INFO:teuthology.orchestra.run.smithi077.stderr:Adding host smithi077... 2024-02-10T08:58:18.069 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpzzyygg7o:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpx5m65_qh:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch host add smithi077 172.21.15.77 2024-02-10T08:58:18.596 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:18.590+0000 7f33d4df6700 1 Processor -- start 2024-02-10T08:58:18.596 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:18.590+0000 7f33d4df6700 1 -- start start 2024-02-10T08:58:18.596 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:18.590+0000 7f33d4df6700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33d00703f0 0x7f33d00fea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:18.596 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:18.590+0000 7f33d4df6700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33d00fef80 con 0x7f33d00703f0 2024-02-10T08:58:18.596 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:18.590+0000 7f33ce59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33d00703f0 0x7f33d00fea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:18.596 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:18.590+0000 7f33ce59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33d00703f0 0x7f33d00fea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46250/0 (socket says 172.21.15.77:46250) 2024-02-10T08:58:18.597 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:18.590+0000 7f33ce59c700 1 -- 172.21.15.77:0/1511784610 learned_addr learned my addr 172.21.15.77:0/1511784610 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-10T08:58:18.597 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:18.590+0000 7f33ce59c700 1 -- 172.21.15.77:0/1511784610 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33d00ff0c0 con 0x7f33d00703f0 2024-02-10T08:58:18.597 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:18.591+0000 7f33ce59c700 1 --2- 172.21.15.77:0/1511784610 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33d00703f0 0x7f33d00fea90 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f33c000b5f0 tx=0x7f33c0006ee0).ready entity=mon.0 client_cookie=2549755b5e4a26d9 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:18.597 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:18.591+0000 7f33cd59a700 1 -- 172.21.15.77:0/1511784610 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33c000d600 con 0x7f33d00703f0 2024-02-10T08:58:18.597 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:18.591+0000 7f33cd59a700 1 -- 172.21.15.77:0/1511784610 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f33c000dbf0 con 0x7f33d00703f0 2024-02-10T08:58:18.597 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:18.591+0000 7f33cd59a700 1 -- 172.21.15.77:0/1511784610 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33c0011ce0 con 0x7f33d00703f0 2024-02-10T08:58:18.597 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:18.591+0000 7f33d4df6700 1 -- 172.21.15.77:0/1511784610 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33d00703f0 msgr2=0x7f33d00fea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:18.597 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:18.591+0000 7f33d4df6700 1 --2- 172.21.15.77:0/1511784610 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33d00703f0 0x7f33d00fea90 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f33c000b5f0 tx=0x7f33c0006ee0).stop 2024-02-10T08:58:18.597 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:18.591+0000 7f33d4df6700 1 -- 172.21.15.77:0/1511784610 shutdown_connections 2024-02-10T08:58:18.597 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:18.591+0000 7f33d4df6700 1 --2- 172.21.15.77:0/1511784610 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33d00703f0 0x7f33d00fea90 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:18.597 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:18.591+0000 7f33d4df6700 1 -- 172.21.15.77:0/1511784610 >> 172.21.15.77:0/1511784610 conn(0x7f33d00f6530 msgr2=0x7f33d00f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:58:18.597 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:18.591+0000 7f33d4df6700 1 -- 172.21.15.77:0/1511784610 shutdown_connections 2024-02-10T08:58:18.597 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:18.591+0000 7f33d4df6700 1 -- 172.21.15.77:0/1511784610 wait complete. 2024-02-10T08:58:18.597 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:18.592+0000 7f33d4df6700 1 Processor -- start 2024-02-10T08:58:18.598 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:18.592+0000 7f33d4df6700 1 -- start start 2024-02-10T08:58:18.598 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:18.592+0000 7f33d4df6700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33d00703f0 0x7f33d010ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:18.598 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:18.592+0000 7f33d4df6700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33d010b010 con 0x7f33d00703f0 2024-02-10T08:58:18.598 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:18.592+0000 7f33ce59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33d00703f0 0x7f33d010ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:18.598 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:18.592+0000 7f33ce59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33d00703f0 0x7f33d010ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46264/0 (socket says 172.21.15.77:46264) 2024-02-10T08:58:18.598 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:18.592+0000 7f33ce59c700 1 -- 172.21.15.77:0/4283076290 learned_addr learned my addr 172.21.15.77:0/4283076290 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-10T08:58:18.598 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:18.592+0000 7f33ce59c700 1 -- 172.21.15.77:0/4283076290 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33c000b200 con 0x7f33d00703f0 2024-02-10T08:58:18.598 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:18.593+0000 7f33ce59c700 1 --2- 172.21.15.77:0/4283076290 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33d00703f0 0x7f33d010ab20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f33c0011ed0 tx=0x7f33c0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:18.598 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:18.593+0000 7f33beffd700 1 -- 172.21.15.77:0/4283076290 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33c001b570 con 0x7f33d00703f0 2024-02-10T08:58:18.598 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:18.593+0000 7f33d4df6700 1 -- 172.21.15.77:0/4283076290 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33d010b210 con 0x7f33d00703f0 2024-02-10T08:58:18.598 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:18.593+0000 7f33beffd700 1 -- 172.21.15.77:0/4283076290 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f33c001bb60 con 0x7f33d00703f0 2024-02-10T08:58:18.598 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:18.593+0000 7f33beffd700 1 -- 172.21.15.77:0/4283076290 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33c001aad0 con 0x7f33d00703f0 2024-02-10T08:58:18.599 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:18.593+0000 7f33d4df6700 1 -- 172.21.15.77:0/4283076290 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33d010b6a0 con 0x7f33d00703f0 2024-02-10T08:58:18.599 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:18.593+0000 7f33beffd700 1 -- 172.21.15.77:0/4283076290 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f33c000f070 con 0x7f33d00703f0 2024-02-10T08:58:18.599 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:18.594+0000 7f33beffd700 1 --2- 172.21.15.77:0/4283076290 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7f33b402fe10 0x7f33b40322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:18.599 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:18.594+0000 7f33beffd700 1 -- 172.21.15.77:0/4283076290 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f33c0041550 con 0x7f33d00703f0 2024-02-10T08:58:18.599 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:18.594+0000 7f33cdd9b700 1 --2- 172.21.15.77:0/4283076290 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7f33b402fe10 0x7f33b40322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:18.599 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:18.594+0000 7f33d4df6700 1 -- 172.21.15.77:0/4283076290 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33d0045ad0 con 0x7f33d00703f0 2024-02-10T08:58:18.599 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:18.594+0000 7f33cdd9b700 1 --2- 172.21.15.77:0/4283076290 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7f33b402fe10 0x7f33b40322d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f33c40096a0 tx=0x7f33c4006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:18.599 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:18.598+0000 7f33beffd700 1 -- 172.21.15.77:0/4283076290 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f33c0016030 con 0x7f33d00703f0 2024-02-10T08:58:18.771 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:18.770+0000 7f33d4df6700 1 -- 172.21.15.77:0/4283076290 --> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi077", "addr": "172.21.15.77", "target": ["mon-mgr", ""]}) v1 -- 0x7f33d010bfc0 con 0x7f33b402fe10 2024-02-10T08:58:18.924 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:18 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:17.405988+0000 mgr.smithi077.dxntvw (mgr.14114) 7 : audit [DBG] from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:58:19.376 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: Added host 'smithi077' 2024-02-10T08:58:19.376 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:19.374+0000 7f33beffd700 1 -- 172.21.15.77:0/4283076290 <== mgr.14114 v2:172.21.15.77:6800/2974869056 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7f33d010bfc0 con 0x7f33b402fe10 2024-02-10T08:58:19.378 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:19.376+0000 7f33d4df6700 1 -- 172.21.15.77:0/4283076290 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7f33b402fe10 msgr2=0x7f33b40322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:19.378 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:19.376+0000 7f33d4df6700 1 --2- 172.21.15.77:0/4283076290 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7f33b402fe10 0x7f33b40322d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f33c40096a0 tx=0x7f33c4006b40).stop 2024-02-10T08:58:19.378 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:19.376+0000 7f33d4df6700 1 -- 172.21.15.77:0/4283076290 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33d00703f0 msgr2=0x7f33d010ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:19.378 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:19.376+0000 7f33d4df6700 1 --2- 172.21.15.77:0/4283076290 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33d00703f0 0x7f33d010ab20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f33c0011ed0 tx=0x7f33c0006ee0).stop 2024-02-10T08:58:19.378 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:19.376+0000 7f33d4df6700 1 -- 172.21.15.77:0/4283076290 shutdown_connections 2024-02-10T08:58:19.378 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:19.376+0000 7f33d4df6700 1 --2- 172.21.15.77:0/4283076290 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7f33b402fe10 0x7f33b40322d0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:19.378 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:19.376+0000 7f33d4df6700 1 --2- 172.21.15.77:0/4283076290 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33d00703f0 0x7f33d010ab20 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:19.378 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:19.376+0000 7f33d4df6700 1 -- 172.21.15.77:0/4283076290 >> 172.21.15.77:0/4283076290 conn(0x7f33d00f6530 msgr2=0x7f33d00f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:58:19.378 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:19.376+0000 7f33d4df6700 1 -- 172.21.15.77:0/4283076290 shutdown_connections 2024-02-10T08:58:19.378 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:19.376+0000 7f33d4df6700 1 -- 172.21.15.77:0/4283076290 wait complete. 2024-02-10T08:58:20.017 INFO:teuthology.orchestra.run.smithi077.stderr:Deploying mon service with default placement... 2024-02-10T08:58:20.017 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpzzyygg7o:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpx5m65_qh:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mon 2024-02-10T08:58:20.561 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:20.558+0000 7f9ef5bbc700 1 Processor -- start 2024-02-10T08:58:20.561 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:20.558+0000 7f9ef5bbc700 1 -- start start 2024-02-10T08:58:20.561 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:20.558+0000 7f9ef5bbc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ef00fad90 0x7f9ef00fb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:20.561 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:20.558+0000 7f9ef5bbc700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ef00fb6a0 con 0x7f9ef00fad90 2024-02-10T08:58:20.561 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:20.559+0000 7f9eef7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ef00fad90 0x7f9ef00fb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:20.561 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:20.559+0000 7f9eef7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ef00fad90 0x7f9ef00fb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46272/0 (socket says 172.21.15.77:46272) 2024-02-10T08:58:20.561 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:20.559+0000 7f9eef7fe700 1 -- 172.21.15.77:0/581461257 learned_addr learned my addr 172.21.15.77:0/581461257 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-10T08:58:20.561 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:20.559+0000 7f9eef7fe700 1 -- 172.21.15.77:0/581461257 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ef00fb7e0 con 0x7f9ef00fad90 2024-02-10T08:58:20.562 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:20.559+0000 7f9eef7fe700 1 --2- 172.21.15.77:0/581461257 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ef00fad90 0x7f9ef00fb1b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f9ee400b5f0 tx=0x7f9ee4006ee0).ready entity=mon.0 client_cookie=12e7b604091efe33 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:20.564 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:20.559+0000 7f9eee7fc700 1 -- 172.21.15.77:0/581461257 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ee400d600 con 0x7f9ef00fad90 2024-02-10T08:58:20.565 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:20.559+0000 7f9eee7fc700 1 -- 172.21.15.77:0/581461257 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f9ee400dbf0 con 0x7f9ef00fad90 2024-02-10T08:58:20.565 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:20.559+0000 7f9eee7fc700 1 -- 172.21.15.77:0/581461257 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ee4011cb0 con 0x7f9ef00fad90 2024-02-10T08:58:20.565 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:20.560+0000 7f9ef5bbc700 1 -- 172.21.15.77:0/581461257 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ef00fad90 msgr2=0x7f9ef00fb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:20.565 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:20.560+0000 7f9ef5bbc700 1 --2- 172.21.15.77:0/581461257 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ef00fad90 0x7f9ef00fb1b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f9ee400b5f0 tx=0x7f9ee4006ee0).stop 2024-02-10T08:58:20.565 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:20.560+0000 7f9ef5bbc700 1 -- 172.21.15.77:0/581461257 shutdown_connections 2024-02-10T08:58:20.565 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:20.560+0000 7f9ef5bbc700 1 --2- 172.21.15.77:0/581461257 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ef00fad90 0x7f9ef00fb1b0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:20.565 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:20.560+0000 7f9ef5bbc700 1 -- 172.21.15.77:0/581461257 >> 172.21.15.77:0/581461257 conn(0x7f9ef00f6530 msgr2=0x7f9ef00f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:58:20.565 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:20.560+0000 7f9ef5bbc700 1 -- 172.21.15.77:0/581461257 shutdown_connections 2024-02-10T08:58:20.565 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:20.560+0000 7f9ef5bbc700 1 -- 172.21.15.77:0/581461257 wait complete. 2024-02-10T08:58:20.565 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:20.560+0000 7f9ef5bbc700 1 Processor -- start 2024-02-10T08:58:20.565 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:20.561+0000 7f9ef5bbc700 1 -- start start 2024-02-10T08:58:20.565 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:20.561+0000 7f9ef5bbc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ef00fad90 0x7f9ef0100e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:20.565 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:20.561+0000 7f9ef5bbc700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ef00ff2e0 con 0x7f9ef00fad90 2024-02-10T08:58:20.566 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:20.561+0000 7f9eef7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ef00fad90 0x7f9ef0100e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:20.566 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:20.561+0000 7f9eef7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ef00fad90 0x7f9ef0100e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46280/0 (socket says 172.21.15.77:46280) 2024-02-10T08:58:20.566 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:20.561+0000 7f9eef7fe700 1 -- 172.21.15.77:0/1468468005 learned_addr learned my addr 172.21.15.77:0/1468468005 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-10T08:58:20.566 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:20.561+0000 7f9eef7fe700 1 -- 172.21.15.77:0/1468468005 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ee400b200 con 0x7f9ef00fad90 2024-02-10T08:58:20.566 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:20.561+0000 7f9eef7fe700 1 --2- 172.21.15.77:0/1468468005 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ef00fad90 0x7f9ef0100e60 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f9ee4011ea0 tx=0x7f9ee4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:20.566 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:20.561+0000 7f9ed7fff700 1 -- 172.21.15.77:0/1468468005 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ee401b570 con 0x7f9ef00fad90 2024-02-10T08:58:20.566 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:20.561+0000 7f9ed7fff700 1 -- 172.21.15.77:0/1468468005 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f9ee401bb60 con 0x7f9ef00fad90 2024-02-10T08:58:20.566 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:20.561+0000 7f9ef5bbc700 1 -- 172.21.15.77:0/1468468005 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ef00ff480 con 0x7f9ef00fad90 2024-02-10T08:58:20.566 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:20.561+0000 7f9ed7fff700 1 -- 172.21.15.77:0/1468468005 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ee401a8d0 con 0x7f9ef00fad90 2024-02-10T08:58:20.566 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:20.561+0000 7f9ef5bbc700 1 -- 172.21.15.77:0/1468468005 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ef00ff910 con 0x7f9ef00fad90 2024-02-10T08:58:20.566 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:20.562+0000 7f9ed7fff700 1 -- 172.21.15.77:0/1468468005 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f9ee400f070 con 0x7f9ef00fad90 2024-02-10T08:58:20.566 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:20.562+0000 7f9ed7fff700 1 --2- 172.21.15.77:0/1468468005 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7f9edc02fe70 0x7f9edc032330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:20.566 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:20.562+0000 7f9ed7fff700 1 -- 172.21.15.77:0/1468468005 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f9ee40422b0 con 0x7f9ef00fad90 2024-02-10T08:58:20.566 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:20.562+0000 7f9eeeffd700 1 --2- 172.21.15.77:0/1468468005 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7f9edc02fe70 0x7f9edc032330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:20.567 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:20.562+0000 7f9ef5bbc700 1 -- 172.21.15.77:0/1468468005 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ef0059100 con 0x7f9ef00fad90 2024-02-10T08:58:20.567 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:20.563+0000 7f9eeeffd700 1 --2- 172.21.15.77:0/1468468005 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7f9edc02fe70 0x7f9edc032330 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f9ee00096a0 tx=0x7f9ee0006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:20.567 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:20.566+0000 7f9ed7fff700 1 -- 172.21.15.77:0/1468468005 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9ee4016070 con 0x7f9ef00fad90 2024-02-10T08:58:20.649 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:20 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:18.771903+0000 mgr.smithi077.dxntvw (mgr.14114) 8 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi077", "addr": "172.21.15.77", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:58:20.649 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:20 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:19.374593+0000 mon.smithi077 (mon.0) 54 : audit [INF] from='mgr.14114 172.21.15.77:0/2499169851' entity='mgr.smithi077.dxntvw' 2024-02-10T08:58:20.649 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:20 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cephadm 2024-02-10T08:58:19.374871+0000 mgr.smithi077.dxntvw (mgr.14114) 9 : cephadm [INF] Added host smithi077 2024-02-10T08:58:20.649 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:20 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:19.375077+0000 mon.smithi077 (mon.0) 55 : audit [DBG] from='mgr.14114 172.21.15.77:0/2499169851' entity='mgr.smithi077.dxntvw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:58:20.748 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:20.747+0000 7f9ef5bbc700 1 -- 172.21.15.77:0/1468468005 --> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f9ef01049f0 con 0x7f9edc02fe70 2024-02-10T08:58:20.752 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:20.751+0000 7f9ed7fff700 1 -- 172.21.15.77:0/1468468005 <== mgr.14114 v2:172.21.15.77:6800/2974869056 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f9ef01049f0 con 0x7f9edc02fe70 2024-02-10T08:58:20.752 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: Scheduled mon update... 2024-02-10T08:58:20.755 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:20.753+0000 7f9ef5bbc700 1 -- 172.21.15.77:0/1468468005 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7f9edc02fe70 msgr2=0x7f9edc032330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:20.755 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:20.753+0000 7f9ef5bbc700 1 --2- 172.21.15.77:0/1468468005 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7f9edc02fe70 0x7f9edc032330 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f9ee00096a0 tx=0x7f9ee0006b40).stop 2024-02-10T08:58:20.755 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:20.753+0000 7f9ef5bbc700 1 -- 172.21.15.77:0/1468468005 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ef00fad90 msgr2=0x7f9ef0100e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:20.755 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:20.753+0000 7f9ef5bbc700 1 --2- 172.21.15.77:0/1468468005 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ef00fad90 0x7f9ef0100e60 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f9ee4011ea0 tx=0x7f9ee4006ee0).stop 2024-02-10T08:58:20.755 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:20.753+0000 7f9ef5bbc700 1 -- 172.21.15.77:0/1468468005 shutdown_connections 2024-02-10T08:58:20.755 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:20.753+0000 7f9ef5bbc700 1 --2- 172.21.15.77:0/1468468005 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7f9edc02fe70 0x7f9edc032330 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:20.755 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:20.753+0000 7f9ef5bbc700 1 --2- 172.21.15.77:0/1468468005 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ef00fad90 0x7f9ef0100e60 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:20.755 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:20.753+0000 7f9ef5bbc700 1 -- 172.21.15.77:0/1468468005 >> 172.21.15.77:0/1468468005 conn(0x7f9ef00f6530 msgr2=0x7f9ef00f7130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:58:20.755 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:20.753+0000 7f9ef5bbc700 1 -- 172.21.15.77:0/1468468005 shutdown_connections 2024-02-10T08:58:20.755 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:20.753+0000 7f9ef5bbc700 1 -- 172.21.15.77:0/1468468005 wait complete. 2024-02-10T08:58:21.431 INFO:teuthology.orchestra.run.smithi077.stderr:Deploying mgr service with default placement... 2024-02-10T08:58:21.431 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpzzyygg7o:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpx5m65_qh:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mgr 2024-02-10T08:58:21.994 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:21.968+0000 7f320f8d5700 1 Processor -- start 2024-02-10T08:58:21.994 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:21.968+0000 7f320f8d5700 1 -- start start 2024-02-10T08:58:21.994 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:21.968+0000 7f320f8d5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32080fad90 0x7f32080fb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:21.995 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:21.968+0000 7f320f8d5700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32080fb6a0 con 0x7f32080fad90 2024-02-10T08:58:21.995 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:21.968+0000 7f320d671700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32080fad90 0x7f32080fb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:21.995 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:21.968+0000 7f320d671700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32080fad90 0x7f32080fb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46296/0 (socket says 172.21.15.77:46296) 2024-02-10T08:58:21.995 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:21.968+0000 7f320d671700 1 -- 172.21.15.77:0/1788112132 learned_addr learned my addr 172.21.15.77:0/1788112132 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-10T08:58:21.995 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:21.969+0000 7f320d671700 1 -- 172.21.15.77:0/1788112132 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32080fb7e0 con 0x7f32080fad90 2024-02-10T08:58:21.995 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:21.969+0000 7f320d671700 1 --2- 172.21.15.77:0/1788112132 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32080fad90 0x7f32080fb1b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f31f800b5f0 tx=0x7f31f8006ee0).ready entity=mon.0 client_cookie=596ed9d79385a120 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:21.995 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:21.969+0000 7f31f7fff700 1 -- 172.21.15.77:0/1788112132 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f31f800d600 con 0x7f32080fad90 2024-02-10T08:58:21.995 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:21.969+0000 7f31f7fff700 1 -- 172.21.15.77:0/1788112132 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f31f800dbf0 con 0x7f32080fad90 2024-02-10T08:58:21.995 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:21.969+0000 7f31f7fff700 1 -- 172.21.15.77:0/1788112132 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f31f8011cb0 con 0x7f32080fad90 2024-02-10T08:58:21.995 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:21.969+0000 7f320f8d5700 1 -- 172.21.15.77:0/1788112132 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32080fad90 msgr2=0x7f32080fb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:21.995 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:21.969+0000 7f320f8d5700 1 --2- 172.21.15.77:0/1788112132 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32080fad90 0x7f32080fb1b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f31f800b5f0 tx=0x7f31f8006ee0).stop 2024-02-10T08:58:21.995 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:21.969+0000 7f320f8d5700 1 -- 172.21.15.77:0/1788112132 shutdown_connections 2024-02-10T08:58:21.995 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:21.969+0000 7f320f8d5700 1 --2- 172.21.15.77:0/1788112132 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32080fad90 0x7f32080fb1b0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:21.995 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:21.969+0000 7f320f8d5700 1 -- 172.21.15.77:0/1788112132 >> 172.21.15.77:0/1788112132 conn(0x7f32080f6530 msgr2=0x7f32080f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:58:21.996 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:21.970+0000 7f320f8d5700 1 -- 172.21.15.77:0/1788112132 shutdown_connections 2024-02-10T08:58:21.996 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:21.970+0000 7f320f8d5700 1 -- 172.21.15.77:0/1788112132 wait complete. 2024-02-10T08:58:21.996 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:21.970+0000 7f320f8d5700 1 Processor -- start 2024-02-10T08:58:21.996 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:21.970+0000 7f320f8d5700 1 -- start start 2024-02-10T08:58:21.996 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:21.970+0000 7f320f8d5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f320810ac90 0x7f320810b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:21.996 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:21.970+0000 7f320f8d5700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f320810b5a0 con 0x7f320810ac90 2024-02-10T08:58:21.996 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:21.970+0000 7f320d671700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f320810ac90 0x7f320810b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:21.996 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:21.970+0000 7f320d671700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f320810ac90 0x7f320810b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46302/0 (socket says 172.21.15.77:46302) 2024-02-10T08:58:21.996 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:21.970+0000 7f320d671700 1 -- 172.21.15.77:0/2334477614 learned_addr learned my addr 172.21.15.77:0/2334477614 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-10T08:58:21.996 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:21.971+0000 7f320d671700 1 -- 172.21.15.77:0/2334477614 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31f800b200 con 0x7f320810ac90 2024-02-10T08:58:21.996 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:21.971+0000 7f320d671700 1 --2- 172.21.15.77:0/2334477614 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f320810ac90 0x7f320810b0b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f31f800b5c0 tx=0x7f31f8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:21.996 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:21.971+0000 7f31f5ffb700 1 -- 172.21.15.77:0/2334477614 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f31f801b570 con 0x7f320810ac90 2024-02-10T08:58:21.996 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:21.971+0000 7f31f5ffb700 1 -- 172.21.15.77:0/2334477614 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f31f801bb60 con 0x7f320810ac90 2024-02-10T08:58:21.996 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:21.971+0000 7f320f8d5700 1 -- 172.21.15.77:0/2334477614 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f320810b7a0 con 0x7f320810ac90 2024-02-10T08:58:21.997 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:21.971+0000 7f31f5ffb700 1 -- 172.21.15.77:0/2334477614 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f31f801a970 con 0x7f320810ac90 2024-02-10T08:58:21.997 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:21.971+0000 7f320f8d5700 1 -- 172.21.15.77:0/2334477614 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f320810c1b0 con 0x7f320810ac90 2024-02-10T08:58:21.997 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:21.972+0000 7f31f5ffb700 1 -- 172.21.15.77:0/2334477614 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f31f800f070 con 0x7f320810ac90 2024-02-10T08:58:21.997 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:21.972+0000 7f31f5ffb700 1 --2- 172.21.15.77:0/2334477614 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7f31f002fe70 0x7f31f0032330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:21.997 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:21.972+0000 7f31f5ffb700 1 -- 172.21.15.77:0/2334477614 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f31f8041280 con 0x7f320810ac90 2024-02-10T08:58:21.997 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:21.972+0000 7f320ce70700 1 --2- 172.21.15.77:0/2334477614 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7f31f002fe70 0x7f31f0032330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:21.997 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:21.972+0000 7f320f8d5700 1 -- 172.21.15.77:0/2334477614 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3208059100 con 0x7f320810ac90 2024-02-10T08:58:21.997 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:21.972+0000 7f320ce70700 1 --2- 172.21.15.77:0/2334477614 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7f31f002fe70 0x7f31f0032330 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f32040096a0 tx=0x7f3204006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:21.997 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:21.976+0000 7f31f5ffb700 1 -- 172.21.15.77:0/2334477614 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f31f8025460 con 0x7f320810ac90 2024-02-10T08:58:22.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:21 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:20.748961+0000 mgr.smithi077.dxntvw (mgr.14114) 10 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:58:22.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:21 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cephadm 2024-02-10T08:58:20.749646+0000 mgr.smithi077.dxntvw (mgr.14114) 11 : cephadm [INF] Saving service mon spec with placement count:5 2024-02-10T08:58:22.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:21 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:20.752535+0000 mon.smithi077 (mon.0) 56 : audit [INF] from='mgr.14114 172.21.15.77:0/2499169851' entity='mgr.smithi077.dxntvw' 2024-02-10T08:58:22.153 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:22.152+0000 7f320f8d5700 1 -- 172.21.15.77:0/2334477614 --> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f3208104ca0 con 0x7f31f002fe70 2024-02-10T08:58:22.157 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:22.156+0000 7f31f5ffb700 1 -- 172.21.15.77:0/2334477614 <== mgr.14114 v2:172.21.15.77:6800/2974869056 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f3208104ca0 con 0x7f31f002fe70 2024-02-10T08:58:22.157 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-02-10T08:58:22.159 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:22.157+0000 7f320f8d5700 1 -- 172.21.15.77:0/2334477614 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7f31f002fe70 msgr2=0x7f31f0032330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:22.159 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:22.157+0000 7f320f8d5700 1 --2- 172.21.15.77:0/2334477614 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7f31f002fe70 0x7f31f0032330 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f32040096a0 tx=0x7f3204006b40).stop 2024-02-10T08:58:22.159 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:22.157+0000 7f320f8d5700 1 -- 172.21.15.77:0/2334477614 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f320810ac90 msgr2=0x7f320810b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:22.160 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:22.157+0000 7f320f8d5700 1 --2- 172.21.15.77:0/2334477614 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f320810ac90 0x7f320810b0b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f31f800b5c0 tx=0x7f31f8006ee0).stop 2024-02-10T08:58:22.160 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:22.157+0000 7f320f8d5700 1 -- 172.21.15.77:0/2334477614 shutdown_connections 2024-02-10T08:58:22.160 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:22.157+0000 7f320f8d5700 1 --2- 172.21.15.77:0/2334477614 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7f31f002fe70 0x7f31f0032330 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:22.160 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:22.157+0000 7f320f8d5700 1 --2- 172.21.15.77:0/2334477614 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f320810ac90 0x7f320810b0b0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:22.160 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:22.157+0000 7f320f8d5700 1 -- 172.21.15.77:0/2334477614 >> 172.21.15.77:0/2334477614 conn(0x7f32080f6530 msgr2=0x7f3208070030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:58:22.160 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:22.157+0000 7f320f8d5700 1 -- 172.21.15.77:0/2334477614 shutdown_connections 2024-02-10T08:58:22.160 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:22.157+0000 7f320f8d5700 1 -- 172.21.15.77:0/2334477614 wait complete. 2024-02-10T08:58:22.813 INFO:teuthology.orchestra.run.smithi077.stderr:Deploying crash service with default placement... 2024-02-10T08:58:22.813 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpzzyygg7o:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpx5m65_qh:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply crash 2024-02-10T08:58:23.358 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:23.352+0000 7f0467fff700 1 Processor -- start 2024-02-10T08:58:23.358 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:23.352+0000 7f0467fff700 1 -- start start 2024-02-10T08:58:23.358 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:23.352+0000 7f0467fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04680fab30 0x7f04680faf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:23.358 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:23.352+0000 7f0467fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04680fb440 con 0x7f04680fab30 2024-02-10T08:58:23.358 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:23.352+0000 7f0466ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04680fab30 0x7f04680faf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:23.358 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:23.352+0000 7f0466ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04680fab30 0x7f04680faf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46312/0 (socket says 172.21.15.77:46312) 2024-02-10T08:58:23.358 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:23.352+0000 7f0466ffd700 1 -- 172.21.15.77:0/803895668 learned_addr learned my addr 172.21.15.77:0/803895668 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-10T08:58:23.358 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:23.353+0000 7f0466ffd700 1 -- 172.21.15.77:0/803895668 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04680fb580 con 0x7f04680fab30 2024-02-10T08:58:23.358 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:23.353+0000 7f0466ffd700 1 --2- 172.21.15.77:0/803895668 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04680fab30 0x7f04680faf50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f045c00b870 tx=0x7f045c006f70).ready entity=mon.0 client_cookie=2f516de458c7adb server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:23.359 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:23.353+0000 7f0465ffb700 1 -- 172.21.15.77:0/803895668 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f045c00da20 con 0x7f04680fab30 2024-02-10T08:58:23.359 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:23.353+0000 7f0465ffb700 1 -- 172.21.15.77:0/803895668 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f045c010470 con 0x7f04680fab30 2024-02-10T08:58:23.359 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:23.353+0000 7f0465ffb700 1 -- 172.21.15.77:0/803895668 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f045c00f5b0 con 0x7f04680fab30 2024-02-10T08:58:23.359 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:23.353+0000 7f0467fff700 1 -- 172.21.15.77:0/803895668 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04680fab30 msgr2=0x7f04680faf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:23.359 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:23.353+0000 7f0467fff700 1 --2- 172.21.15.77:0/803895668 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04680fab30 0x7f04680faf50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f045c00b870 tx=0x7f045c006f70).stop 2024-02-10T08:58:23.359 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:23.354+0000 7f0467fff700 1 -- 172.21.15.77:0/803895668 shutdown_connections 2024-02-10T08:58:23.359 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:23.354+0000 7f0467fff700 1 --2- 172.21.15.77:0/803895668 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04680fab30 0x7f04680faf50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:23.359 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:23.354+0000 7f0467fff700 1 -- 172.21.15.77:0/803895668 >> 172.21.15.77:0/803895668 conn(0x7f04680f6390 msgr2=0x7f04680f87b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:58:23.359 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:23.354+0000 7f0467fff700 1 -- 172.21.15.77:0/803895668 shutdown_connections 2024-02-10T08:58:23.359 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:23.354+0000 7f0467fff700 1 -- 172.21.15.77:0/803895668 wait complete. 2024-02-10T08:58:23.359 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:23.354+0000 7f0467fff700 1 Processor -- start 2024-02-10T08:58:23.359 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:23.354+0000 7f0467fff700 1 -- start start 2024-02-10T08:58:23.359 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:23.354+0000 7f0467fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04680fab30 0x7f0468108750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:23.359 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:23.354+0000 7f0467fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0468108c40 con 0x7f04680fab30 2024-02-10T08:58:23.360 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:23.355+0000 7f0466ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04680fab30 0x7f0468108750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:23.360 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:23.355+0000 7f0466ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04680fab30 0x7f0468108750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46324/0 (socket says 172.21.15.77:46324) 2024-02-10T08:58:23.360 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:23.355+0000 7f0466ffd700 1 -- 172.21.15.77:0/1399852146 learned_addr learned my addr 172.21.15.77:0/1399852146 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-10T08:58:23.360 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:23.355+0000 7f0466ffd700 1 -- 172.21.15.77:0/1399852146 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f045c00b480 con 0x7f04680fab30 2024-02-10T08:58:23.360 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:23.355+0000 7f0466ffd700 1 --2- 172.21.15.77:0/1399852146 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04680fab30 0x7f0468108750 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f045c000c00 tx=0x7f045c006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:23.360 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:23.355+0000 7f044f7fe700 1 -- 172.21.15.77:0/1399852146 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f045c0109e0 con 0x7f04680fab30 2024-02-10T08:58:23.360 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:23.355+0000 7f0467fff700 1 -- 172.21.15.77:0/1399852146 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0468108e40 con 0x7f04680fab30 2024-02-10T08:58:23.360 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:23.355+0000 7f044f7fe700 1 -- 172.21.15.77:0/1399852146 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f045c00fcb0 con 0x7f04680fab30 2024-02-10T08:58:23.360 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:23.355+0000 7f044f7fe700 1 -- 172.21.15.77:0/1399852146 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f045c019cc0 con 0x7f04680fab30 2024-02-10T08:58:23.360 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:23.355+0000 7f0467fff700 1 -- 172.21.15.77:0/1399852146 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04681092d0 con 0x7f04680fab30 2024-02-10T08:58:23.360 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:23.356+0000 7f044f7fe700 1 -- 172.21.15.77:0/1399852146 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f045c016030 con 0x7f04680fab30 2024-02-10T08:58:23.360 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:23.356+0000 7f0467fff700 1 -- 172.21.15.77:0/1399852146 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0468045ad0 con 0x7f04680fab30 2024-02-10T08:58:23.360 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:23.356+0000 7f044f7fe700 1 --2- 172.21.15.77:0/1399852146 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7f045402fe10 0x7f04540322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:23.361 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:23.356+0000 7f044f7fe700 1 -- 172.21.15.77:0/1399852146 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f045c043ef0 con 0x7f04680fab30 2024-02-10T08:58:23.361 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:23.356+0000 7f04667fc700 1 --2- 172.21.15.77:0/1399852146 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7f045402fe10 0x7f04540322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:23.361 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:23.357+0000 7f04667fc700 1 --2- 172.21.15.77:0/1399852146 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7f045402fe10 0x7f04540322d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f04580096a0 tx=0x7f0458006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:23.361 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:23.360+0000 7f044f7fe700 1 -- 172.21.15.77:0/1399852146 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f045c022780 con 0x7f04680fab30 2024-02-10T08:58:23.536 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:23.535+0000 7f0467fff700 1 -- 172.21.15.77:0/1399852146 --> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f0468109bf0 con 0x7f045402fe10 2024-02-10T08:58:23.540 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:23.539+0000 7f044f7fe700 1 -- 172.21.15.77:0/1399852146 <== mgr.14114 v2:172.21.15.77:6800/2974869056 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f0468109bf0 con 0x7f045402fe10 2024-02-10T08:58:23.540 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: Scheduled crash update... 2024-02-10T08:58:23.542 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:23.541+0000 7f0467fff700 1 -- 172.21.15.77:0/1399852146 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7f045402fe10 msgr2=0x7f04540322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:23.543 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:23.541+0000 7f0467fff700 1 --2- 172.21.15.77:0/1399852146 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7f045402fe10 0x7f04540322d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f04580096a0 tx=0x7f0458006b40).stop 2024-02-10T08:58:23.543 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:23.541+0000 7f0467fff700 1 -- 172.21.15.77:0/1399852146 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04680fab30 msgr2=0x7f0468108750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:23.543 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:23.541+0000 7f0467fff700 1 --2- 172.21.15.77:0/1399852146 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04680fab30 0x7f0468108750 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f045c000c00 tx=0x7f045c006f70).stop 2024-02-10T08:58:23.543 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:23.541+0000 7f0467fff700 1 -- 172.21.15.77:0/1399852146 shutdown_connections 2024-02-10T08:58:23.543 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:23.541+0000 7f0467fff700 1 --2- 172.21.15.77:0/1399852146 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7f045402fe10 0x7f04540322d0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:23.543 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:23.541+0000 7f0467fff700 1 --2- 172.21.15.77:0/1399852146 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f04680fab30 0x7f0468108750 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:23.543 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:23.541+0000 7f0467fff700 1 -- 172.21.15.77:0/1399852146 >> 172.21.15.77:0/1399852146 conn(0x7f04680f6390 msgr2=0x7f04680ff8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:58:23.543 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:23.541+0000 7f0467fff700 1 -- 172.21.15.77:0/1399852146 shutdown_connections 2024-02-10T08:58:23.543 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:23.541+0000 7f0467fff700 1 -- 172.21.15.77:0/1399852146 wait complete. 2024-02-10T08:58:23.564 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:23 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:22.154289+0000 mgr.smithi077.dxntvw (mgr.14114) 12 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:58:23.564 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:23 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cephadm 2024-02-10T08:58:22.154849+0000 mgr.smithi077.dxntvw (mgr.14114) 13 : cephadm [INF] Saving service mgr spec with placement count:2 2024-02-10T08:58:23.564 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:23 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:22.156974+0000 mon.smithi077 (mon.0) 57 : audit [INF] from='mgr.14114 172.21.15.77:0/2499169851' entity='mgr.smithi077.dxntvw' 2024-02-10T08:58:24.186 INFO:teuthology.orchestra.run.smithi077.stderr:Enabling mgr prometheus module... 2024-02-10T08:58:24.186 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpzzyygg7o:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpx5m65_qh:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable prometheus 2024-02-10T08:58:24.746 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:24.744+0000 7f5b043b1700 1 Processor -- start 2024-02-10T08:58:24.747 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:24.744+0000 7f5b043b1700 1 -- start start 2024-02-10T08:58:24.747 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:24.744+0000 7f5b043b1700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5afc0fbea0 0x7f5afc0fc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:24.747 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:24.744+0000 7f5b043b1700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5afc0fc7b0 con 0x7f5afc0fbea0 2024-02-10T08:58:24.747 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:24.745+0000 7f5b0214d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5afc0fbea0 0x7f5afc0fc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:24.747 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:24.745+0000 7f5b0214d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5afc0fbea0 0x7f5afc0fc2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46340/0 (socket says 172.21.15.77:46340) 2024-02-10T08:58:24.747 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:24.745+0000 7f5b0214d700 1 -- 172.21.15.77:0/4204965639 learned_addr learned my addr 172.21.15.77:0/4204965639 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-10T08:58:24.747 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:24.745+0000 7f5b0214d700 1 -- 172.21.15.77:0/4204965639 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5afc0fc8f0 con 0x7f5afc0fbea0 2024-02-10T08:58:24.747 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:24.745+0000 7f5b0214d700 1 --2- 172.21.15.77:0/4204965639 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5afc0fbea0 0x7f5afc0fc2c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f5aec00b5f0 tx=0x7f5aec006ee0).ready entity=mon.0 client_cookie=839f16817483187d server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:24.747 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:24.745+0000 7f5b0114b700 1 -- 172.21.15.77:0/4204965639 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5aec00d600 con 0x7f5afc0fbea0 2024-02-10T08:58:24.747 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:24.745+0000 7f5b0114b700 1 -- 172.21.15.77:0/4204965639 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f5aec00dbf0 con 0x7f5afc0fbea0 2024-02-10T08:58:24.747 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:24.745+0000 7f5b0114b700 1 -- 172.21.15.77:0/4204965639 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5aec011cb0 con 0x7f5afc0fbea0 2024-02-10T08:58:24.750 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:24.746+0000 7f5b043b1700 1 -- 172.21.15.77:0/4204965639 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5afc0fbea0 msgr2=0x7f5afc0fc2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:24.750 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:24.746+0000 7f5b043b1700 1 --2- 172.21.15.77:0/4204965639 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5afc0fbea0 0x7f5afc0fc2c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f5aec00b5f0 tx=0x7f5aec006ee0).stop 2024-02-10T08:58:24.750 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:24.746+0000 7f5b043b1700 1 -- 172.21.15.77:0/4204965639 shutdown_connections 2024-02-10T08:58:24.750 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:24.746+0000 7f5b043b1700 1 --2- 172.21.15.77:0/4204965639 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5afc0fbea0 0x7f5afc0fc2c0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:24.751 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:24.746+0000 7f5b043b1700 1 -- 172.21.15.77:0/4204965639 >> 172.21.15.77:0/4204965639 conn(0x7f5afc0f76a0 msgr2=0x7f5afc0f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:58:24.751 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:24.746+0000 7f5b043b1700 1 -- 172.21.15.77:0/4204965639 shutdown_connections 2024-02-10T08:58:24.751 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:24.746+0000 7f5b043b1700 1 -- 172.21.15.77:0/4204965639 wait complete. 2024-02-10T08:58:24.751 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:24.747+0000 7f5b043b1700 1 Processor -- start 2024-02-10T08:58:24.751 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:24.747+0000 7f5b043b1700 1 -- start start 2024-02-10T08:58:24.751 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:24.747+0000 7f5b043b1700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5afc0fbea0 0x7f5afc10abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:24.751 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:24.747+0000 7f5b043b1700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5afc10b0b0 con 0x7f5afc0fbea0 2024-02-10T08:58:24.751 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:24.747+0000 7f5b0214d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5afc0fbea0 0x7f5afc10abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:24.751 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:24.747+0000 7f5b0214d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5afc0fbea0 0x7f5afc10abc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46352/0 (socket says 172.21.15.77:46352) 2024-02-10T08:58:24.751 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:24.747+0000 7f5b0214d700 1 -- 172.21.15.77:0/303995531 learned_addr learned my addr 172.21.15.77:0/303995531 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-10T08:58:24.751 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:24.747+0000 7f5b0214d700 1 -- 172.21.15.77:0/303995531 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aec00b200 con 0x7f5afc0fbea0 2024-02-10T08:58:24.751 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:24.747+0000 7f5b0214d700 1 --2- 172.21.15.77:0/303995531 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5afc0fbea0 0x7f5afc10abc0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f5aec002f40 tx=0x7f5aec006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:24.751 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:24.747+0000 7f5aeaffd700 1 -- 172.21.15.77:0/303995531 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5aec01b410 con 0x7f5afc0fbea0 2024-02-10T08:58:24.752 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:24.747+0000 7f5b043b1700 1 -- 172.21.15.77:0/303995531 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5afc10b2b0 con 0x7f5afc0fbea0 2024-02-10T08:58:24.752 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:24.747+0000 7f5aeaffd700 1 -- 172.21.15.77:0/303995531 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f5aec01ba00 con 0x7f5afc0fbea0 2024-02-10T08:58:24.752 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:24.747+0000 7f5aeaffd700 1 -- 172.21.15.77:0/303995531 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5aec01a980 con 0x7f5afc0fbea0 2024-02-10T08:58:24.752 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:24.747+0000 7f5b043b1700 1 -- 172.21.15.77:0/303995531 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5afc10b740 con 0x7f5afc0fbea0 2024-02-10T08:58:24.752 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:24.748+0000 7f5aeaffd700 1 -- 172.21.15.77:0/303995531 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f5aec00f070 con 0x7f5afc0fbea0 2024-02-10T08:58:24.752 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:24.748+0000 7f5b043b1700 1 -- 172.21.15.77:0/303995531 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5afc045ad0 con 0x7f5afc0fbea0 2024-02-10T08:58:24.752 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:24.749+0000 7f5aeaffd700 1 --2- 172.21.15.77:0/303995531 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7f5ae402fe10 0x7f5ae40322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:24.752 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:24.749+0000 7f5aeaffd700 1 -- 172.21.15.77:0/303995531 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f5aec041420 con 0x7f5afc0fbea0 2024-02-10T08:58:24.752 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:24.749+0000 7f5b0194c700 1 --2- 172.21.15.77:0/303995531 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7f5ae402fe10 0x7f5ae40322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:24.752 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:24.749+0000 7f5b0194c700 1 --2- 172.21.15.77:0/303995531 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7f5ae402fe10 0x7f5ae40322d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f5af80096a0 tx=0x7f5af8006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:24.754 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:24.753+0000 7f5aeaffd700 1 -- 172.21.15.77:0/303995531 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5aec022310 con 0x7f5afc0fbea0 2024-02-10T08:58:24.898 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:24 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:23.537240+0000 mgr.smithi077.dxntvw (mgr.14114) 14 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:58:24.899 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:24 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cephadm 2024-02-10T08:58:23.537929+0000 mgr.smithi077.dxntvw (mgr.14114) 15 : cephadm [INF] Saving service crash spec with placement * 2024-02-10T08:58:24.899 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:24 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:23.540121+0000 mon.smithi077 (mon.0) 58 : audit [INF] from='mgr.14114 172.21.15.77:0/2499169851' entity='mgr.smithi077.dxntvw' 2024-02-10T08:58:24.941 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:24.939+0000 7f5b043b1700 1 -- 172.21.15.77:0/303995531 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "prometheus"} v 0) v1 -- 0x7f5afc0fdc00 con 0x7f5afc0fbea0 2024-02-10T08:58:25.544 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:25.542+0000 7f5aeaffd700 1 -- 172.21.15.77:0/303995531 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7f5aec01aae0 con 0x7f5afc0fbea0 2024-02-10T08:58:25.545 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:25.544+0000 7f5aeaffd700 1 -- 172.21.15.77:0/303995531 <== mon.0 v2:172.21.15.77:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "prometheus"}]=0 v9) v1 ==== 89+0+0 (secure 0 0 0) 0x7f5aec0249d0 con 0x7f5afc0fbea0 2024-02-10T08:58:25.548 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:25.546+0000 7f5b043b1700 1 -- 172.21.15.77:0/303995531 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7f5ae402fe10 msgr2=0x7f5ae40322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:25.548 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:25.546+0000 7f5b043b1700 1 --2- 172.21.15.77:0/303995531 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7f5ae402fe10 0x7f5ae40322d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f5af80096a0 tx=0x7f5af8006b40).stop 2024-02-10T08:58:25.548 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:25.546+0000 7f5b043b1700 1 -- 172.21.15.77:0/303995531 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5afc0fbea0 msgr2=0x7f5afc10abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:25.548 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:25.546+0000 7f5b043b1700 1 --2- 172.21.15.77:0/303995531 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5afc0fbea0 0x7f5afc10abc0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f5aec002f40 tx=0x7f5aec006ee0).stop 2024-02-10T08:58:25.548 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:25.546+0000 7f5b043b1700 1 -- 172.21.15.77:0/303995531 shutdown_connections 2024-02-10T08:58:25.548 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:25.546+0000 7f5b043b1700 1 --2- 172.21.15.77:0/303995531 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7f5ae402fe10 0x7f5ae40322d0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:25.548 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:25.546+0000 7f5b043b1700 1 --2- 172.21.15.77:0/303995531 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5afc0fbea0 0x7f5afc10abc0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:25.549 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:25.546+0000 7f5b043b1700 1 -- 172.21.15.77:0/303995531 >> 172.21.15.77:0/303995531 conn(0x7f5afc0f76a0 msgr2=0x7f5afc0f9020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:58:25.549 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:25.546+0000 7f5b043b1700 1 -- 172.21.15.77:0/303995531 shutdown_connections 2024-02-10T08:58:25.549 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:25.546+0000 7f5b043b1700 1 -- 172.21.15.77:0/303995531 wait complete. 2024-02-10T08:58:25.899 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:25 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:24.941481+0000 mon.smithi077 (mon.0) 59 : audit [INF] from='client.? 172.21.15.77:0/303995531' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-10T08:58:26.226 INFO:teuthology.orchestra.run.smithi077.stderr:Deploying prometheus service with default placement... 2024-02-10T08:58:26.226 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpzzyygg7o:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpx5m65_qh:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply prometheus 2024-02-10T08:58:26.755 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:26.752+0000 7fd051447700 1 Processor -- start 2024-02-10T08:58:26.755 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:26.752+0000 7fd051447700 1 -- start start 2024-02-10T08:58:26.755 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:26.752+0000 7fd051447700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd04c0fbea0 0x7fd04c0fc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:26.755 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:26.752+0000 7fd051447700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd04c0fc7b0 con 0x7fd04c0fbea0 2024-02-10T08:58:26.755 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:26.752+0000 7fd04affd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd04c0fbea0 0x7fd04c0fc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:26.755 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:26.752+0000 7fd04affd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd04c0fbea0 0x7fd04c0fc2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46376/0 (socket says 172.21.15.77:46376) 2024-02-10T08:58:26.755 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:26.752+0000 7fd04affd700 1 -- 172.21.15.77:0/1522496217 learned_addr learned my addr 172.21.15.77:0/1522496217 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-10T08:58:26.755 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:26.753+0000 7fd04affd700 1 -- 172.21.15.77:0/1522496217 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd04c0fc8f0 con 0x7fd04c0fbea0 2024-02-10T08:58:26.755 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:26.753+0000 7fd04affd700 1 --2- 172.21.15.77:0/1522496217 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd04c0fbea0 0x7fd04c0fc2c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fd03c00b5f0 tx=0x7fd03c006ee0).ready entity=mon.0 client_cookie=534067760590a61a server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:26.756 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:26.753+0000 7fd049ffb700 1 -- 172.21.15.77:0/1522496217 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd03c00d600 con 0x7fd04c0fbea0 2024-02-10T08:58:26.756 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:26.753+0000 7fd049ffb700 1 -- 172.21.15.77:0/1522496217 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fd03c00dbf0 con 0x7fd04c0fbea0 2024-02-10T08:58:26.756 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:26.753+0000 7fd049ffb700 1 -- 172.21.15.77:0/1522496217 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd03c011cb0 con 0x7fd04c0fbea0 2024-02-10T08:58:26.756 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:26.754+0000 7fd051447700 1 -- 172.21.15.77:0/1522496217 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd04c0fbea0 msgr2=0x7fd04c0fc2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:26.756 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:26.754+0000 7fd051447700 1 --2- 172.21.15.77:0/1522496217 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd04c0fbea0 0x7fd04c0fc2c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fd03c00b5f0 tx=0x7fd03c006ee0).stop 2024-02-10T08:58:26.758 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:26.754+0000 7fd051447700 1 -- 172.21.15.77:0/1522496217 shutdown_connections 2024-02-10T08:58:26.758 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:26.754+0000 7fd051447700 1 --2- 172.21.15.77:0/1522496217 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd04c0fbea0 0x7fd04c0fc2c0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:26.758 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:26.754+0000 7fd051447700 1 -- 172.21.15.77:0/1522496217 >> 172.21.15.77:0/1522496217 conn(0x7fd04c0f76a0 msgr2=0x7fd04c0f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:58:26.758 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:26.754+0000 7fd051447700 1 -- 172.21.15.77:0/1522496217 shutdown_connections 2024-02-10T08:58:26.758 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:26.754+0000 7fd051447700 1 -- 172.21.15.77:0/1522496217 wait complete. 2024-02-10T08:58:26.758 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:26.755+0000 7fd051447700 1 Processor -- start 2024-02-10T08:58:26.758 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:26.755+0000 7fd051447700 1 -- start start 2024-02-10T08:58:26.758 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:26.755+0000 7fd051447700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd04c0fbea0 0x7fd04c10abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:26.758 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:26.755+0000 7fd051447700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd04c10b0b0 con 0x7fd04c0fbea0 2024-02-10T08:58:26.759 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:26.755+0000 7fd04affd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd04c0fbea0 0x7fd04c10abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:26.759 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:26.755+0000 7fd04affd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd04c0fbea0 0x7fd04c10abc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46388/0 (socket says 172.21.15.77:46388) 2024-02-10T08:58:26.759 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:26.755+0000 7fd04affd700 1 -- 172.21.15.77:0/2690634656 learned_addr learned my addr 172.21.15.77:0/2690634656 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-10T08:58:26.759 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:26.755+0000 7fd04affd700 1 -- 172.21.15.77:0/2690634656 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd03c00b200 con 0x7fd04c0fbea0 2024-02-10T08:58:26.759 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:26.755+0000 7fd04affd700 1 --2- 172.21.15.77:0/2690634656 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd04c0fbea0 0x7fd04c10abc0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fd03c002f40 tx=0x7fd03c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:26.759 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:26.755+0000 7fd0337fe700 1 -- 172.21.15.77:0/2690634656 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd03c01b410 con 0x7fd04c0fbea0 2024-02-10T08:58:26.759 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:26.756+0000 7fd051447700 1 -- 172.21.15.77:0/2690634656 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd04c10b2b0 con 0x7fd04c0fbea0 2024-02-10T08:58:26.759 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:26.756+0000 7fd0337fe700 1 -- 172.21.15.77:0/2690634656 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fd03c01ba00 con 0x7fd04c0fbea0 2024-02-10T08:58:26.759 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:26.756+0000 7fd0337fe700 1 -- 172.21.15.77:0/2690634656 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd03c01a980 con 0x7fd04c0fbea0 2024-02-10T08:58:26.759 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:26.756+0000 7fd051447700 1 -- 172.21.15.77:0/2690634656 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd04c10b740 con 0x7fd04c0fbea0 2024-02-10T08:58:26.759 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:26.757+0000 7fd051447700 1 -- 172.21.15.77:0/2690634656 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd04c045ad0 con 0x7fd04c0fbea0 2024-02-10T08:58:26.762 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:26.761+0000 7fd0337fe700 1 -- 172.21.15.77:0/2690634656 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7fd03c00f070 con 0x7fd04c0fbea0 2024-02-10T08:58:26.762 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:26.761+0000 7fd0337fe700 1 --2- 172.21.15.77:0/2690634656 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7fd03802fec0 0x7fd038032380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:26.763 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:26.761+0000 7fd0337fe700 1 -- 172.21.15.77:0/2690634656 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fd03c041cb0 con 0x7fd04c0fbea0 2024-02-10T08:58:26.763 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:26.761+0000 7fd04a7fc700 1 -- 172.21.15.77:0/2690634656 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7fd03802fec0 msgr2=0x7fd038032380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6800/2974869056 2024-02-10T08:58:26.763 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:26.761+0000 7fd04a7fc700 1 --2- 172.21.15.77:0/2690634656 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7fd03802fec0 0x7fd038032380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-10T08:58:26.766 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:26.765+0000 7fd0337fe700 1 -- 172.21.15.77:0/2690634656 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd03c019300 con 0x7fd04c0fbea0 2024-02-10T08:58:26.899 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:26 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:25.545384+0000 mon.smithi077 (mon.0) 60 : audit [INF] from='client.? 172.21.15.77:0/303995531' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-10T08:58:26.899 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:26 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T08:58:25.545443+0000 mon.smithi077 (mon.0) 61 : cluster [DBG] mgrmap e9: smithi077.dxntvw(active, since 19s) 2024-02-10T08:58:26.963 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:26.962+0000 7fd04a7fc700 1 -- 172.21.15.77:0/2690634656 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7fd03802fec0 msgr2=0x7fd038032380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6800/2974869056 2024-02-10T08:58:26.963 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:26.962+0000 7fd04a7fc700 1 --2- 172.21.15.77:0/2690634656 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7fd03802fec0 0x7fd038032380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-10T08:58:26.993 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:26.992+0000 7fd051447700 1 -- 172.21.15.77:0/2690634656 --> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7fd04c10c060 con 0x7fd03802fec0 2024-02-10T08:58:27.364 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:27.363+0000 7fd04a7fc700 1 -- 172.21.15.77:0/2690634656 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7fd03802fec0 msgr2=0x7fd038032380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6800/2974869056 2024-02-10T08:58:27.364 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:27.363+0000 7fd04a7fc700 1 --2- 172.21.15.77:0/2690634656 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7fd03802fec0 0x7fd038032380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-02-10T08:58:28.165 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:28.164+0000 7fd04a7fc700 1 -- 172.21.15.77:0/2690634656 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7fd03802fec0 msgr2=0x7fd038032380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6800/2974869056 2024-02-10T08:58:28.165 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:28.164+0000 7fd04a7fc700 1 --2- 172.21.15.77:0/2690634656 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7fd03802fec0 0x7fd038032380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-02-10T08:58:29.767 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:29.765+0000 7fd04a7fc700 1 -- 172.21.15.77:0/2690634656 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7fd03802fec0 msgr2=0x7fd038032380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6800/2974869056 2024-02-10T08:58:29.767 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:29.765+0000 7fd04a7fc700 1 --2- 172.21.15.77:0/2690634656 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7fd03802fec0 0x7fd038032380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2024-02-10T08:58:30.537 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:30.536+0000 7fd0337fe700 1 -- 172.21.15.77:0/2690634656 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mgrmap(e 10) v1 ==== 35984+0+0 (secure 0 0 0) 0x7fd03c01aae0 con 0x7fd04c0fbea0 2024-02-10T08:58:30.537 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:30.536+0000 7fd0337fe700 1 -- 172.21.15.77:0/2690634656 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7fd03802fec0 msgr2=0x7fd038032380 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-10T08:58:30.537 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:30.536+0000 7fd0337fe700 1 --2- 172.21.15.77:0/2690634656 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7fd03802fec0 0x7fd038032380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-10T08:58:30.899 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:30 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T08:58:30.483056+0000 mon.smithi077 (mon.0) 62 : cluster [INF] Active manager daemon smithi077.dxntvw restarted 2024-02-10T08:58:30.899 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:30 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T08:58:30.483438+0000 mon.smithi077 (mon.0) 63 : cluster [INF] Activating manager daemon smithi077.dxntvw 2024-02-10T08:58:30.899 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:30 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T08:58:30.487742+0000 mon.smithi077 (mon.0) 64 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2024-02-10T08:58:31.547 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:31.544+0000 7fd0337fe700 1 -- 172.21.15.77:0/2690634656 <== mon.0 v2:172.21.15.77:3300/0 8 ==== mgrmap(e 11) v1 ==== 36054+0+0 (secure 0 0 0) 0x7fd03c023360 con 0x7fd04c0fbea0 2024-02-10T08:58:31.547 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:31.544+0000 7fd0337fe700 1 --2- 172.21.15.77:0/2690634656 >> [v2:172.21.15.77:6800/3154363712,v1:172.21.15.77:6801/3154363712] conn(0x7fd038033ce0 0x7fd0380360d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:31.547 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:31.544+0000 7fd0337fe700 1 -- 172.21.15.77:0/2690634656 --> [v2:172.21.15.77:6800/3154363712,v1:172.21.15.77:6801/3154363712] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7fd04c10c060 con 0x7fd038033ce0 2024-02-10T08:58:31.547 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:31.545+0000 7fd04a7fc700 1 --2- 172.21.15.77:0/2690634656 >> [v2:172.21.15.77:6800/3154363712,v1:172.21.15.77:6801/3154363712] conn(0x7fd038033ce0 0x7fd0380360d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:31.547 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:31.545+0000 7fd04a7fc700 1 --2- 172.21.15.77:0/2690634656 >> [v2:172.21.15.77:6800/3154363712,v1:172.21.15.77:6801/3154363712] conn(0x7fd038033ce0 0x7fd0380360d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fd040006fd0 tx=0x7fd040006d30).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:31.551 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:31.550+0000 7fd0337fe700 1 -- 172.21.15.77:0/2690634656 <== mgr.14142 v2:172.21.15.77:6800/3154363712 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7fd04c10c060 con 0x7fd038033ce0 2024-02-10T08:58:31.551 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: Scheduled prometheus update... 2024-02-10T08:58:31.554 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:31.552+0000 7fd051447700 1 -- 172.21.15.77:0/2690634656 >> [v2:172.21.15.77:6800/3154363712,v1:172.21.15.77:6801/3154363712] conn(0x7fd038033ce0 msgr2=0x7fd0380360d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:31.554 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:31.552+0000 7fd051447700 1 --2- 172.21.15.77:0/2690634656 >> [v2:172.21.15.77:6800/3154363712,v1:172.21.15.77:6801/3154363712] conn(0x7fd038033ce0 0x7fd0380360d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fd040006fd0 tx=0x7fd040006d30).stop 2024-02-10T08:58:31.554 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:31.552+0000 7fd051447700 1 -- 172.21.15.77:0/2690634656 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd04c0fbea0 msgr2=0x7fd04c10abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:31.554 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:31.552+0000 7fd051447700 1 --2- 172.21.15.77:0/2690634656 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd04c0fbea0 0x7fd04c10abc0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fd03c002f40 tx=0x7fd03c006ee0).stop 2024-02-10T08:58:31.554 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:31.552+0000 7fd051447700 1 -- 172.21.15.77:0/2690634656 shutdown_connections 2024-02-10T08:58:31.554 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:31.552+0000 7fd051447700 1 --2- 172.21.15.77:0/2690634656 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd04c0fbea0 0x7fd04c10abc0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:31.554 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:31.552+0000 7fd051447700 1 --2- 172.21.15.77:0/2690634656 >> [v2:172.21.15.77:6800/3154363712,v1:172.21.15.77:6801/3154363712] conn(0x7fd038033ce0 0x7fd0380360d0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:31.554 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:31.552+0000 7fd051447700 1 --2- 172.21.15.77:0/2690634656 >> [v2:172.21.15.77:6800/2974869056,v1:172.21.15.77:6801/2974869056] conn(0x7fd03802fec0 0x7fd038032380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-10T08:58:31.554 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:31.552+0000 7fd051447700 1 -- 172.21.15.77:0/2690634656 >> 172.21.15.77:0/2690634656 conn(0x7fd04c0f76a0 msgr2=0x7fd04c0f9020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:58:31.554 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:31.553+0000 7fd051447700 1 -- 172.21.15.77:0/2690634656 shutdown_connections 2024-02-10T08:58:31.554 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:31.553+0000 7fd051447700 1 -- 172.21.15.77:0/2690634656 wait complete. 2024-02-10T08:58:31.891 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:31 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T08:58:30.538397+0000 mon.smithi077 (mon.0) 65 : cluster [DBG] mgrmap e10: smithi077.dxntvw(active, starting, since 0.0550394s) 2024-02-10T08:58:31.891 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:31 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:30.540867+0000 mon.smithi077 (mon.0) 66 : audit [DBG] from='mgr.14142 172.21.15.77:0/3691491939' entity='mgr.smithi077.dxntvw' cmd=[{"prefix": "mon metadata", "id": "smithi077"}]: dispatch 2024-02-10T08:58:31.891 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:31 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:30.541434+0000 mon.smithi077 (mon.0) 67 : audit [DBG] from='mgr.14142 172.21.15.77:0/3691491939' entity='mgr.smithi077.dxntvw' cmd=[{"prefix": "mgr metadata", "who": "smithi077.dxntvw", "id": "smithi077.dxntvw"}]: dispatch 2024-02-10T08:58:31.891 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:31 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:30.542052+0000 mon.smithi077 (mon.0) 68 : audit [DBG] from='mgr.14142 172.21.15.77:0/3691491939' entity='mgr.smithi077.dxntvw' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T08:58:31.891 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:31 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:30.542234+0000 mon.smithi077 (mon.0) 69 : audit [DBG] from='mgr.14142 172.21.15.77:0/3691491939' entity='mgr.smithi077.dxntvw' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T08:58:31.891 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:31 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:30.542413+0000 mon.smithi077 (mon.0) 70 : audit [DBG] from='mgr.14142 172.21.15.77:0/3691491939' entity='mgr.smithi077.dxntvw' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T08:58:31.891 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:31 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T08:58:30.555797+0000 mon.smithi077 (mon.0) 71 : cluster [INF] Manager daemon smithi077.dxntvw is now available 2024-02-10T08:58:31.891 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:31 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:30.651823+0000 mon.smithi077 (mon.0) 72 : audit [DBG] from='mgr.14142 172.21.15.77:0/3691491939' entity='mgr.smithi077.dxntvw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:58:31.891 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:31 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:30.653190+0000 mon.smithi077 (mon.0) 73 : audit [DBG] from='mgr.14142 172.21.15.77:0/3691491939' entity='mgr.smithi077.dxntvw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:58:31.891 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:31 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:30.667711+0000 mon.smithi077 (mon.0) 74 : audit [INF] from='mgr.14142 172.21.15.77:0/3691491939' entity='mgr.smithi077.dxntvw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi077.dxntvw/mirror_snapshot_schedule"}]: dispatch 2024-02-10T08:58:31.892 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:31 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:30.674736+0000 mon.smithi077 (mon.0) 75 : audit [INF] from='mgr.14142 172.21.15.77:0/3691491939' entity='mgr.smithi077.dxntvw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi077.dxntvw/trash_purge_schedule"}]: dispatch 2024-02-10T08:58:32.168 INFO:teuthology.orchestra.run.smithi077.stderr:Deploying grafana service with default placement... 2024-02-10T08:58:32.168 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpzzyygg7o:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpx5m65_qh:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply grafana 2024-02-10T08:58:32.846 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:32.841+0000 7fe9c0bd9700 1 Processor -- start 2024-02-10T08:58:32.847 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:32.841+0000 7fe9c0bd9700 1 -- start start 2024-02-10T08:58:32.847 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:32.842+0000 7fe9c0bd9700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9bc0703d0 0x7fe9bc0feab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:32.847 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:32.842+0000 7fe9c0bd9700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9bc0fefa0 con 0x7fe9bc0703d0 2024-02-10T08:58:32.847 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:32.842+0000 7fe9ba59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9bc0703d0 0x7fe9bc0feab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:32.847 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:32.842+0000 7fe9ba59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9bc0703d0 0x7fe9bc0feab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44308/0 (socket says 172.21.15.77:44308) 2024-02-10T08:58:32.847 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:32.842+0000 7fe9ba59c700 1 -- 172.21.15.77:0/368211765 learned_addr learned my addr 172.21.15.77:0/368211765 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-10T08:58:32.847 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:32.842+0000 7fe9ba59c700 1 -- 172.21.15.77:0/368211765 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9bc0ff0e0 con 0x7fe9bc0703d0 2024-02-10T08:58:32.847 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:32.842+0000 7fe9ba59c700 1 --2- 172.21.15.77:0/368211765 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9bc0703d0 0x7fe9bc0feab0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fe9b000b5f0 tx=0x7fe9b0006ee0).ready entity=mon.0 client_cookie=e16fbcd157ef40ca server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:32.847 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:32.842+0000 7fe9b959a700 1 -- 172.21.15.77:0/368211765 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe9b000d600 con 0x7fe9bc0703d0 2024-02-10T08:58:32.847 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:32.842+0000 7fe9b959a700 1 -- 172.21.15.77:0/368211765 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fe9b000dbf0 con 0x7fe9bc0703d0 2024-02-10T08:58:32.847 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:32.842+0000 7fe9b959a700 1 -- 172.21.15.77:0/368211765 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe9b0011cb0 con 0x7fe9bc0703d0 2024-02-10T08:58:32.847 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:32.843+0000 7fe9c0bd9700 1 -- 172.21.15.77:0/368211765 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9bc0703d0 msgr2=0x7fe9bc0feab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:32.847 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:32.843+0000 7fe9c0bd9700 1 --2- 172.21.15.77:0/368211765 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9bc0703d0 0x7fe9bc0feab0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fe9b000b5f0 tx=0x7fe9b0006ee0).stop 2024-02-10T08:58:32.847 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:32.843+0000 7fe9c0bd9700 1 -- 172.21.15.77:0/368211765 shutdown_connections 2024-02-10T08:58:32.848 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:32.843+0000 7fe9c0bd9700 1 --2- 172.21.15.77:0/368211765 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9bc0703d0 0x7fe9bc0feab0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:32.848 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:32.843+0000 7fe9c0bd9700 1 -- 172.21.15.77:0/368211765 >> 172.21.15.77:0/368211765 conn(0x7fe9bc0f6530 msgr2=0x7fe9bc0f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:58:32.848 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:32.843+0000 7fe9c0bd9700 1 -- 172.21.15.77:0/368211765 shutdown_connections 2024-02-10T08:58:32.848 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:32.843+0000 7fe9c0bd9700 1 -- 172.21.15.77:0/368211765 wait complete. 2024-02-10T08:58:32.848 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:32.844+0000 7fe9c0bd9700 1 Processor -- start 2024-02-10T08:58:32.848 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:32.844+0000 7fe9c0bd9700 1 -- start start 2024-02-10T08:58:32.848 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:32.844+0000 7fe9c0bd9700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9bc0703d0 0x7fe9bc10ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:32.848 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:32.844+0000 7fe9c0bd9700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9bc10b060 con 0x7fe9bc0703d0 2024-02-10T08:58:32.848 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:32.844+0000 7fe9ba59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9bc0703d0 0x7fe9bc10ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:32.848 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:32.844+0000 7fe9ba59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9bc0703d0 0x7fe9bc10ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44312/0 (socket says 172.21.15.77:44312) 2024-02-10T08:58:32.848 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:32.844+0000 7fe9ba59c700 1 -- 172.21.15.77:0/3807002399 learned_addr learned my addr 172.21.15.77:0/3807002399 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-10T08:58:32.848 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:32.844+0000 7fe9ba59c700 1 -- 172.21.15.77:0/3807002399 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9b000b200 con 0x7fe9bc0703d0 2024-02-10T08:58:32.848 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:32.844+0000 7fe9ba59c700 1 --2- 172.21.15.77:0/3807002399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9bc0703d0 0x7fe9bc10ab70 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fe9b0006a20 tx=0x7fe9b0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:32.849 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:32.844+0000 7fe9aaffd700 1 -- 172.21.15.77:0/3807002399 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe9b001b570 con 0x7fe9bc0703d0 2024-02-10T08:58:32.849 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:32.845+0000 7fe9aaffd700 1 -- 172.21.15.77:0/3807002399 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fe9b001bb60 con 0x7fe9bc0703d0 2024-02-10T08:58:32.849 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:32.845+0000 7fe9c0bd9700 1 -- 172.21.15.77:0/3807002399 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9bc10b260 con 0x7fe9bc0703d0 2024-02-10T08:58:32.849 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:32.845+0000 7fe9aaffd700 1 -- 172.21.15.77:0/3807002399 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe9b001aa20 con 0x7fe9bc0703d0 2024-02-10T08:58:32.849 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:32.845+0000 7fe9c0bd9700 1 -- 172.21.15.77:0/3807002399 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9bc10b6f0 con 0x7fe9bc0703d0 2024-02-10T08:58:32.849 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:32.845+0000 7fe9aaffd700 1 -- 172.21.15.77:0/3807002399 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fe9b000f070 con 0x7fe9bc0703d0 2024-02-10T08:58:32.849 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:32.845+0000 7fe9aaffd700 1 --2- 172.21.15.77:0/3807002399 >> [v2:172.21.15.77:6800/3154363712,v1:172.21.15.77:6801/3154363712] conn(0x7fe9a002ff20 0x7fe9a00323e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:32.849 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:32.846+0000 7fe9aaffd700 1 -- 172.21.15.77:0/3807002399 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fe9b00414c0 con 0x7fe9bc0703d0 2024-02-10T08:58:32.849 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:32.846+0000 7fe9c0bd9700 1 -- 172.21.15.77:0/3807002399 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9bc045ad0 con 0x7fe9bc0703d0 2024-02-10T08:58:32.850 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:32.846+0000 7fe9b9d9b700 1 --2- 172.21.15.77:0/3807002399 >> [v2:172.21.15.77:6800/3154363712,v1:172.21.15.77:6801/3154363712] conn(0x7fe9a002ff20 0x7fe9a00323e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:32.850 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:32.846+0000 7fe9b9d9b700 1 --2- 172.21.15.77:0/3807002399 >> [v2:172.21.15.77:6800/3154363712,v1:172.21.15.77:6801/3154363712] conn(0x7fe9a002ff20 0x7fe9a00323e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fe9ac0096a0 tx=0x7fe9ac006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:32.851 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:32.850+0000 7fe9aaffd700 1 -- 172.21.15.77:0/3807002399 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe9b0022d20 con 0x7fe9bc0703d0 2024-02-10T08:58:32.898 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:32 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T08:58:31.548052+0000 mon.smithi077 (mon.0) 76 : cluster [DBG] mgrmap e11: smithi077.dxntvw(active, since 1.06469s) 2024-02-10T08:58:32.899 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:32 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cephadm 2024-02-10T08:58:31.548876+0000 mgr.smithi077.dxntvw (mgr.14142) 2 : cephadm [INF] Saving service prometheus spec with placement count:1 2024-02-10T08:58:32.899 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:32 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:31.550945+0000 mon.smithi077 (mon.0) 77 : audit [INF] from='mgr.14142 172.21.15.77:0/3691491939' entity='mgr.smithi077.dxntvw' 2024-02-10T08:58:33.026 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:33.025+0000 7fe9c0bd9700 1 -- 172.21.15.77:0/3807002399 --> [v2:172.21.15.77:6800/3154363712,v1:172.21.15.77:6801/3154363712] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7fe9bc10bf50 con 0x7fe9a002ff20 2024-02-10T08:58:33.033 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:33.032+0000 7fe9aaffd700 1 -- 172.21.15.77:0/3807002399 <== mgr.14142 v2:172.21.15.77:6800/3154363712 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7fe9bc10bf50 con 0x7fe9a002ff20 2024-02-10T08:58:33.033 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: Scheduled grafana update... 2024-02-10T08:58:33.035 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:33.034+0000 7fe9c0bd9700 1 -- 172.21.15.77:0/3807002399 >> [v2:172.21.15.77:6800/3154363712,v1:172.21.15.77:6801/3154363712] conn(0x7fe9a002ff20 msgr2=0x7fe9a00323e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:33.035 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:33.034+0000 7fe9c0bd9700 1 --2- 172.21.15.77:0/3807002399 >> [v2:172.21.15.77:6800/3154363712,v1:172.21.15.77:6801/3154363712] conn(0x7fe9a002ff20 0x7fe9a00323e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fe9ac0096a0 tx=0x7fe9ac006b40).stop 2024-02-10T08:58:33.036 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:33.034+0000 7fe9c0bd9700 1 -- 172.21.15.77:0/3807002399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9bc0703d0 msgr2=0x7fe9bc10ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:33.036 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:33.034+0000 7fe9c0bd9700 1 --2- 172.21.15.77:0/3807002399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9bc0703d0 0x7fe9bc10ab70 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fe9b0006a20 tx=0x7fe9b0006ee0).stop 2024-02-10T08:58:33.036 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:33.034+0000 7fe9c0bd9700 1 -- 172.21.15.77:0/3807002399 shutdown_connections 2024-02-10T08:58:33.036 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:33.034+0000 7fe9c0bd9700 1 --2- 172.21.15.77:0/3807002399 >> [v2:172.21.15.77:6800/3154363712,v1:172.21.15.77:6801/3154363712] conn(0x7fe9a002ff20 0x7fe9a00323e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:33.036 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:33.034+0000 7fe9c0bd9700 1 --2- 172.21.15.77:0/3807002399 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9bc0703d0 0x7fe9bc10ab70 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:33.036 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:33.034+0000 7fe9c0bd9700 1 -- 172.21.15.77:0/3807002399 >> 172.21.15.77:0/3807002399 conn(0x7fe9bc0f6530 msgr2=0x7fe9bc0f7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:58:33.036 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:33.034+0000 7fe9c0bd9700 1 -- 172.21.15.77:0/3807002399 shutdown_connections 2024-02-10T08:58:33.036 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:33.034+0000 7fe9c0bd9700 1 -- 172.21.15.77:0/3807002399 wait complete. 2024-02-10T08:58:33.757 INFO:teuthology.orchestra.run.smithi077.stderr:Deploying node-exporter service with default placement... 2024-02-10T08:58:33.757 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpzzyygg7o:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpx5m65_qh:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply node-exporter 2024-02-10T08:58:33.760 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:33 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T08:58:32.555072+0000 mon.smithi077 (mon.0) 78 : cluster [DBG] mgrmap e12: smithi077.dxntvw(active, since 2s) 2024-02-10T08:58:33.760 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:33 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:33.033335+0000 mon.smithi077 (mon.0) 79 : audit [INF] from='mgr.14142 172.21.15.77:0/3691491939' entity='mgr.smithi077.dxntvw' 2024-02-10T08:58:34.299 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:34.297+0000 7f2689214700 1 Processor -- start 2024-02-10T08:58:34.300 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:34.297+0000 7f2689214700 1 -- start start 2024-02-10T08:58:34.300 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:34.297+0000 7f2689214700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26840fad90 0x7f26840fb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:34.300 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:34.297+0000 7f2689214700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26840fb6a0 con 0x7f26840fad90 2024-02-10T08:58:34.300 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:34.297+0000 7f2682d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26840fad90 0x7f26840fb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:34.300 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:34.298+0000 7f2682d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26840fad90 0x7f26840fb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44324/0 (socket says 172.21.15.77:44324) 2024-02-10T08:58:34.300 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:34.298+0000 7f2682d9d700 1 -- 172.21.15.77:0/460060301 learned_addr learned my addr 172.21.15.77:0/460060301 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-10T08:58:34.300 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:34.298+0000 7f2682d9d700 1 -- 172.21.15.77:0/460060301 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26840fb7e0 con 0x7f26840fad90 2024-02-10T08:58:34.300 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:34.298+0000 7f2682d9d700 1 --2- 172.21.15.77:0/460060301 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26840fad90 0x7f26840fb1b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f267800b870 tx=0x7f2678006f70).ready entity=mon.0 client_cookie=7739f1609d45e9ef server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:34.302 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:34.298+0000 7f2681d9b700 1 -- 172.21.15.77:0/460060301 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f267800da20 con 0x7f26840fad90 2024-02-10T08:58:34.302 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:34.298+0000 7f2681d9b700 1 -- 172.21.15.77:0/460060301 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f2678010470 con 0x7f26840fad90 2024-02-10T08:58:34.303 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:34.298+0000 7f2681d9b700 1 -- 172.21.15.77:0/460060301 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f267800f5b0 con 0x7f26840fad90 2024-02-10T08:58:34.303 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:34.299+0000 7f2689214700 1 -- 172.21.15.77:0/460060301 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26840fad90 msgr2=0x7f26840fb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:34.303 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:34.299+0000 7f2689214700 1 --2- 172.21.15.77:0/460060301 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26840fad90 0x7f26840fb1b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f267800b870 tx=0x7f2678006f70).stop 2024-02-10T08:58:34.303 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:34.299+0000 7f2689214700 1 -- 172.21.15.77:0/460060301 shutdown_connections 2024-02-10T08:58:34.303 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:34.299+0000 7f2689214700 1 --2- 172.21.15.77:0/460060301 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26840fad90 0x7f26840fb1b0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:34.303 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:34.299+0000 7f2689214700 1 -- 172.21.15.77:0/460060301 >> 172.21.15.77:0/460060301 conn(0x7f26840f6530 msgr2=0x7f26840f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:58:34.303 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:34.299+0000 7f2689214700 1 -- 172.21.15.77:0/460060301 shutdown_connections 2024-02-10T08:58:34.303 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:34.299+0000 7f2689214700 1 -- 172.21.15.77:0/460060301 wait complete. 2024-02-10T08:58:34.303 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:34.299+0000 7f2689214700 1 Processor -- start 2024-02-10T08:58:34.303 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:34.299+0000 7f2689214700 1 -- start start 2024-02-10T08:58:34.303 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:34.299+0000 7f2689214700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26840fad90 0x7f268410cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:34.303 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:34.299+0000 7f2689214700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f268410d270 con 0x7f26840fad90 2024-02-10T08:58:34.303 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:34.300+0000 7f2682d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26840fad90 0x7f268410cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:34.303 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:34.300+0000 7f2682d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26840fad90 0x7f268410cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44328/0 (socket says 172.21.15.77:44328) 2024-02-10T08:58:34.304 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:34.300+0000 7f2682d9d700 1 -- 172.21.15.77:0/3076521657 learned_addr learned my addr 172.21.15.77:0/3076521657 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-10T08:58:34.304 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:34.300+0000 7f2682d9d700 1 -- 172.21.15.77:0/3076521657 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f267800b480 con 0x7f26840fad90 2024-02-10T08:58:34.304 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:34.300+0000 7f2682d9d700 1 --2- 172.21.15.77:0/3076521657 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26840fad90 0x7f268410cd80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f2678006800 tx=0x7f2678006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:34.304 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:34.300+0000 7f266b7fe700 1 -- 172.21.15.77:0/3076521657 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f267800f870 con 0x7f26840fad90 2024-02-10T08:58:34.304 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:34.300+0000 7f2689214700 1 -- 172.21.15.77:0/3076521657 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f268410d470 con 0x7f26840fad90 2024-02-10T08:58:34.304 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:34.300+0000 7f266b7fe700 1 -- 172.21.15.77:0/3076521657 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f267800fe60 con 0x7f26840fad90 2024-02-10T08:58:34.304 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:34.300+0000 7f266b7fe700 1 -- 172.21.15.77:0/3076521657 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2678019e40 con 0x7f26840fad90 2024-02-10T08:58:34.304 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:34.300+0000 7f2689214700 1 -- 172.21.15.77:0/3076521657 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f268410e0b0 con 0x7f26840fad90 2024-02-10T08:58:34.304 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:34.301+0000 7f266b7fe700 1 -- 172.21.15.77:0/3076521657 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f267802a070 con 0x7f26840fad90 2024-02-10T08:58:34.304 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:34.301+0000 7f2689214700 1 -- 172.21.15.77:0/3076521657 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f268410d900 con 0x7f26840fad90 2024-02-10T08:58:34.304 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:34.301+0000 7f266b7fe700 1 --2- 172.21.15.77:0/3076521657 >> [v2:172.21.15.77:6800/3154363712,v1:172.21.15.77:6801/3154363712] conn(0x7f267002ff20 0x7f26700323e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:34.304 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:34.301+0000 7f266b7fe700 1 -- 172.21.15.77:0/3076521657 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f2678041b60 con 0x7f26840fad90 2024-02-10T08:58:34.304 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:34.301+0000 7f268259c700 1 --2- 172.21.15.77:0/3076521657 >> [v2:172.21.15.77:6800/3154363712,v1:172.21.15.77:6801/3154363712] conn(0x7f267002ff20 0x7f26700323e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:34.305 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:34.302+0000 7f268259c700 1 --2- 172.21.15.77:0/3076521657 >> [v2:172.21.15.77:6800/3154363712,v1:172.21.15.77:6801/3154363712] conn(0x7f267002ff20 0x7f26700323e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f26740096a0 tx=0x7f2674006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:34.306 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:34.305+0000 7f266b7fe700 1 -- 172.21.15.77:0/3076521657 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f26780196c0 con 0x7f26840fad90 2024-02-10T08:58:34.481 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:34.480+0000 7f2689214700 1 -- 172.21.15.77:0/3076521657 --> [v2:172.21.15.77:6800/3154363712,v1:172.21.15.77:6801/3154363712] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f26840fcca0 con 0x7f267002ff20 2024-02-10T08:58:34.485 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:34.484+0000 7f266b7fe700 1 -- 172.21.15.77:0/3076521657 <== mgr.14142 v2:172.21.15.77:6800/3154363712 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f26840fcca0 con 0x7f267002ff20 2024-02-10T08:58:34.485 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2024-02-10T08:58:34.487 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:34.485+0000 7f2689214700 1 -- 172.21.15.77:0/3076521657 >> [v2:172.21.15.77:6800/3154363712,v1:172.21.15.77:6801/3154363712] conn(0x7f267002ff20 msgr2=0x7f26700323e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:34.487 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:34.485+0000 7f2689214700 1 --2- 172.21.15.77:0/3076521657 >> [v2:172.21.15.77:6800/3154363712,v1:172.21.15.77:6801/3154363712] conn(0x7f267002ff20 0x7f26700323e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f26740096a0 tx=0x7f2674006b40).stop 2024-02-10T08:58:34.487 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:34.485+0000 7f2689214700 1 -- 172.21.15.77:0/3076521657 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26840fad90 msgr2=0x7f268410cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:34.488 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:34.485+0000 7f2689214700 1 --2- 172.21.15.77:0/3076521657 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26840fad90 0x7f268410cd80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f2678006800 tx=0x7f2678006f70).stop 2024-02-10T08:58:34.488 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:34.485+0000 7f2689214700 1 -- 172.21.15.77:0/3076521657 shutdown_connections 2024-02-10T08:58:34.488 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:34.485+0000 7f2689214700 1 --2- 172.21.15.77:0/3076521657 >> [v2:172.21.15.77:6800/3154363712,v1:172.21.15.77:6801/3154363712] conn(0x7f267002ff20 0x7f26700323e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:34.488 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:34.485+0000 7f2689214700 1 --2- 172.21.15.77:0/3076521657 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26840fad90 0x7f268410cd80 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:34.488 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:34.485+0000 7f2689214700 1 -- 172.21.15.77:0/3076521657 >> 172.21.15.77:0/3076521657 conn(0x7f26840f6530 msgr2=0x7f26840ff9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:58:34.488 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:34.485+0000 7f2689214700 1 -- 172.21.15.77:0/3076521657 shutdown_connections 2024-02-10T08:58:34.488 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:34.485+0000 7f2689214700 1 -- 172.21.15.77:0/3076521657 wait complete. 2024-02-10T08:58:34.835 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:34 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:33.027750+0000 mgr.smithi077.dxntvw (mgr.14142) 3 : audit [DBG] from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:58:34.835 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:34 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cephadm 2024-02-10T08:58:33.028263+0000 mgr.smithi077.dxntvw (mgr.14142) 4 : cephadm [INF] Saving service grafana spec with placement count:1 2024-02-10T08:58:34.835 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:34 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:34.484967+0000 mon.smithi077 (mon.0) 80 : audit [INF] from='mgr.14142 172.21.15.77:0/3691491939' entity='mgr.smithi077.dxntvw' 2024-02-10T08:58:35.130 INFO:teuthology.orchestra.run.smithi077.stderr:Deploying alertmanager service with default placement... 2024-02-10T08:58:35.131 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpzzyygg7o:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpx5m65_qh:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply alertmanager 2024-02-10T08:58:35.671 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:35.669+0000 7f87bb0a1700 1 Processor -- start 2024-02-10T08:58:35.672 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:35.669+0000 7f87bb0a1700 1 -- start start 2024-02-10T08:58:35.672 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:35.669+0000 7f87bb0a1700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87b4070430 0x7f87b40fea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:35.672 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:35.669+0000 7f87bb0a1700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b40fef40 con 0x7f87b4070430 2024-02-10T08:58:35.672 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:35.669+0000 7f87b8e3d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87b4070430 0x7f87b40fea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:35.672 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:35.670+0000 7f87b8e3d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87b4070430 0x7f87b40fea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44344/0 (socket says 172.21.15.77:44344) 2024-02-10T08:58:35.672 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:35.670+0000 7f87b8e3d700 1 -- 172.21.15.77:0/1100921759 learned_addr learned my addr 172.21.15.77:0/1100921759 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-10T08:58:35.672 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:35.670+0000 7f87b8e3d700 1 -- 172.21.15.77:0/1100921759 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87b40ff080 con 0x7f87b4070430 2024-02-10T08:58:35.672 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:35.670+0000 7f87b8e3d700 1 --2- 172.21.15.77:0/1100921759 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87b4070430 0x7f87b40fea50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f87ac009960 tx=0x7f87ac0092d0).ready entity=mon.0 client_cookie=bc251d4460b7517b server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:35.672 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:35.670+0000 7f87b37fe700 1 -- 172.21.15.77:0/1100921759 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f87ac00b9c0 con 0x7f87b4070430 2024-02-10T08:58:35.674 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:35.670+0000 7f87b37fe700 1 -- 172.21.15.77:0/1100921759 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f87ac011400 con 0x7f87b4070430 2024-02-10T08:58:35.674 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:35.670+0000 7f87b37fe700 1 -- 172.21.15.77:0/1100921759 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f87ac010580 con 0x7f87b4070430 2024-02-10T08:58:35.675 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:35.671+0000 7f87bb0a1700 1 -- 172.21.15.77:0/1100921759 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87b4070430 msgr2=0x7f87b40fea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:35.675 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:35.671+0000 7f87bb0a1700 1 --2- 172.21.15.77:0/1100921759 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87b4070430 0x7f87b40fea50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f87ac009960 tx=0x7f87ac0092d0).stop 2024-02-10T08:58:35.675 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:35.671+0000 7f87bb0a1700 1 -- 172.21.15.77:0/1100921759 shutdown_connections 2024-02-10T08:58:35.675 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:35.671+0000 7f87bb0a1700 1 --2- 172.21.15.77:0/1100921759 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87b4070430 0x7f87b40fea50 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:35.675 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:35.671+0000 7f87bb0a1700 1 -- 172.21.15.77:0/1100921759 >> 172.21.15.77:0/1100921759 conn(0x7f87b40f6530 msgr2=0x7f87b40f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:58:35.675 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:35.671+0000 7f87bb0a1700 1 -- 172.21.15.77:0/1100921759 shutdown_connections 2024-02-10T08:58:35.675 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:35.671+0000 7f87bb0a1700 1 -- 172.21.15.77:0/1100921759 wait complete. 2024-02-10T08:58:35.675 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:35.671+0000 7f87bb0a1700 1 Processor -- start 2024-02-10T08:58:35.675 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:35.671+0000 7f87bb0a1700 1 -- start start 2024-02-10T08:58:35.675 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:35.672+0000 7f87bb0a1700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87b4070430 0x7f87b40fc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:35.675 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:35.672+0000 7f87bb0a1700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b40fa950 con 0x7f87b4070430 2024-02-10T08:58:35.675 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:35.672+0000 7f87b8e3d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87b4070430 0x7f87b40fc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:35.675 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:35.672+0000 7f87b8e3d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87b4070430 0x7f87b40fc4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44356/0 (socket says 172.21.15.77:44356) 2024-02-10T08:58:35.675 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:35.672+0000 7f87b8e3d700 1 -- 172.21.15.77:0/2486501285 learned_addr learned my addr 172.21.15.77:0/2486501285 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-10T08:58:35.675 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:35.672+0000 7f87b8e3d700 1 -- 172.21.15.77:0/2486501285 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87ac009570 con 0x7f87b4070430 2024-02-10T08:58:35.676 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:35.672+0000 7f87b8e3d700 1 --2- 172.21.15.77:0/2486501285 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87b4070430 0x7f87b40fc4d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f87ac00b5c0 tx=0x7f87ac011f90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:35.676 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:35.673+0000 7f87b17fa700 1 -- 172.21.15.77:0/2486501285 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f87ac0107f0 con 0x7f87b4070430 2024-02-10T08:58:35.676 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:35.673+0000 7f87b17fa700 1 -- 172.21.15.77:0/2486501285 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f87ac010de0 con 0x7f87b4070430 2024-02-10T08:58:35.676 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:35.673+0000 7f87bb0a1700 1 -- 172.21.15.77:0/2486501285 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87b40faaf0 con 0x7f87b4070430 2024-02-10T08:58:35.676 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:35.673+0000 7f87b17fa700 1 -- 172.21.15.77:0/2486501285 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f87ac01adf0 con 0x7f87b4070430 2024-02-10T08:58:35.676 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:35.673+0000 7f87bb0a1700 1 -- 172.21.15.77:0/2486501285 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87b40faf80 con 0x7f87b4070430 2024-02-10T08:58:35.676 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:35.673+0000 7f87b17fa700 1 -- 172.21.15.77:0/2486501285 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f87ac00f070 con 0x7f87b4070430 2024-02-10T08:58:35.676 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:35.674+0000 7f87b17fa700 1 --2- 172.21.15.77:0/2486501285 >> [v2:172.21.15.77:6800/3154363712,v1:172.21.15.77:6801/3154363712] conn(0x7f879402ff20 0x7f87940323e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:35.677 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:35.674+0000 7f87b17fa700 1 -- 172.21.15.77:0/2486501285 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f87ac041cd0 con 0x7f87b4070430 2024-02-10T08:58:35.677 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:35.674+0000 7f87bb0a1700 1 -- 172.21.15.77:0/2486501285 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87b4045ad0 con 0x7f87b4070430 2024-02-10T08:58:35.677 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:35.674+0000 7f87b3fff700 1 --2- 172.21.15.77:0/2486501285 >> [v2:172.21.15.77:6800/3154363712,v1:172.21.15.77:6801/3154363712] conn(0x7f879402ff20 0x7f87940323e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:35.677 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:35.674+0000 7f87b3fff700 1 --2- 172.21.15.77:0/2486501285 >> [v2:172.21.15.77:6800/3154363712,v1:172.21.15.77:6801/3154363712] conn(0x7f879402ff20 0x7f87940323e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f879c0096a0 tx=0x7f879c006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:35.679 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:35.678+0000 7f87b17fa700 1 -- 172.21.15.77:0/2486501285 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f87ac01e070 con 0x7f87b4070430 2024-02-10T08:58:35.853 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:35.852+0000 7f87bb0a1700 1 -- 172.21.15.77:0/2486501285 --> [v2:172.21.15.77:6800/3154363712,v1:172.21.15.77:6801/3154363712] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7f87b40ffea0 con 0x7f879402ff20 2024-02-10T08:58:35.856 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:35.855+0000 7f87b17fa700 1 -- 172.21.15.77:0/2486501285 <== mgr.14142 v2:172.21.15.77:6800/3154363712 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f87b40ffea0 con 0x7f879402ff20 2024-02-10T08:58:35.857 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2024-02-10T08:58:35.859 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:35.857+0000 7f87bb0a1700 1 -- 172.21.15.77:0/2486501285 >> [v2:172.21.15.77:6800/3154363712,v1:172.21.15.77:6801/3154363712] conn(0x7f879402ff20 msgr2=0x7f87940323e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:35.859 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:35.857+0000 7f87bb0a1700 1 --2- 172.21.15.77:0/2486501285 >> [v2:172.21.15.77:6800/3154363712,v1:172.21.15.77:6801/3154363712] conn(0x7f879402ff20 0x7f87940323e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f879c0096a0 tx=0x7f879c006b40).stop 2024-02-10T08:58:35.860 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:35.857+0000 7f87bb0a1700 1 -- 172.21.15.77:0/2486501285 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87b4070430 msgr2=0x7f87b40fc4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:35.860 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:35.857+0000 7f87bb0a1700 1 --2- 172.21.15.77:0/2486501285 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87b4070430 0x7f87b40fc4d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f87ac00b5c0 tx=0x7f87ac011f90).stop 2024-02-10T08:58:35.860 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:35.857+0000 7f87bb0a1700 1 -- 172.21.15.77:0/2486501285 shutdown_connections 2024-02-10T08:58:35.860 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:35.857+0000 7f87bb0a1700 1 --2- 172.21.15.77:0/2486501285 >> [v2:172.21.15.77:6800/3154363712,v1:172.21.15.77:6801/3154363712] conn(0x7f879402ff20 0x7f87940323e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:35.860 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:35.857+0000 7f87bb0a1700 1 --2- 172.21.15.77:0/2486501285 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87b4070430 0x7f87b40fc4d0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:35.860 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:35.857+0000 7f87bb0a1700 1 -- 172.21.15.77:0/2486501285 >> 172.21.15.77:0/2486501285 conn(0x7f87b40f6530 msgr2=0x7f87b40f7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:58:35.860 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:35.857+0000 7f87bb0a1700 1 -- 172.21.15.77:0/2486501285 shutdown_connections 2024-02-10T08:58:35.860 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:35.857+0000 7f87bb0a1700 1 -- 172.21.15.77:0/2486501285 wait complete. 2024-02-10T08:58:35.878 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:35 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:34.482274+0000 mgr.smithi077.dxntvw (mgr.14142) 5 : audit [DBG] from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:58:35.878 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:35 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cephadm 2024-02-10T08:58:34.482893+0000 mgr.smithi077.dxntvw (mgr.14142) 6 : cephadm [INF] Saving service node-exporter spec with placement * 2024-02-10T08:58:36.503 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpzzyygg7o:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpx5m65_qh:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/cephadm/container_init True --force 2024-02-10T08:58:37.085 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:37.064+0000 7f1a6a208700 1 Processor -- start 2024-02-10T08:58:37.085 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:37.064+0000 7f1a6a208700 1 -- start start 2024-02-10T08:58:37.085 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:37.064+0000 7f1a6a208700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a640fbea0 0x7f1a640fc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:37.085 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:37.064+0000 7f1a6a208700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a640fc7b0 con 0x7f1a640fbea0 2024-02-10T08:58:37.085 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:37.064+0000 7f1a637fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a640fbea0 0x7f1a640fc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:37.085 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:37.064+0000 7f1a637fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a640fbea0 0x7f1a640fc2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44368/0 (socket says 172.21.15.77:44368) 2024-02-10T08:58:37.085 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:37.064+0000 7f1a637fe700 1 -- 172.21.15.77:0/1114433784 learned_addr learned my addr 172.21.15.77:0/1114433784 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-10T08:58:37.085 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:37.065+0000 7f1a637fe700 1 -- 172.21.15.77:0/1114433784 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a640fc8f0 con 0x7f1a640fbea0 2024-02-10T08:58:37.086 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:37.065+0000 7f1a637fe700 1 --2- 172.21.15.77:0/1114433784 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a640fbea0 0x7f1a640fc2c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f1a5c00b5f0 tx=0x7f1a5c006ee0).ready entity=mon.0 client_cookie=63e67433ce15a01a server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:37.086 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:37.065+0000 7f1a627fc700 1 -- 172.21.15.77:0/1114433784 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a5c00d600 con 0x7f1a640fbea0 2024-02-10T08:58:37.086 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:37.065+0000 7f1a627fc700 1 -- 172.21.15.77:0/1114433784 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f1a5c00dbf0 con 0x7f1a640fbea0 2024-02-10T08:58:37.086 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:37.065+0000 7f1a627fc700 1 -- 172.21.15.77:0/1114433784 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a5c011cb0 con 0x7f1a640fbea0 2024-02-10T08:58:37.086 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:37.065+0000 7f1a6a208700 1 -- 172.21.15.77:0/1114433784 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a640fbea0 msgr2=0x7f1a640fc2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:37.086 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:37.065+0000 7f1a6a208700 1 --2- 172.21.15.77:0/1114433784 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a640fbea0 0x7f1a640fc2c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f1a5c00b5f0 tx=0x7f1a5c006ee0).stop 2024-02-10T08:58:37.086 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:37.065+0000 7f1a6a208700 1 -- 172.21.15.77:0/1114433784 shutdown_connections 2024-02-10T08:58:37.086 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:37.065+0000 7f1a6a208700 1 --2- 172.21.15.77:0/1114433784 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a640fbea0 0x7f1a640fc2c0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:37.086 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:37.065+0000 7f1a6a208700 1 -- 172.21.15.77:0/1114433784 >> 172.21.15.77:0/1114433784 conn(0x7f1a640f76a0 msgr2=0x7f1a640f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:58:37.086 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:37.065+0000 7f1a6a208700 1 -- 172.21.15.77:0/1114433784 shutdown_connections 2024-02-10T08:58:37.086 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:37.065+0000 7f1a6a208700 1 -- 172.21.15.77:0/1114433784 wait complete. 2024-02-10T08:58:37.086 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:37.066+0000 7f1a6a208700 1 Processor -- start 2024-02-10T08:58:37.086 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:37.066+0000 7f1a6a208700 1 -- start start 2024-02-10T08:58:37.086 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:37.066+0000 7f1a6a208700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a640fbea0 0x7f1a6410ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:37.087 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:37.066+0000 7f1a6a208700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6410b070 con 0x7f1a640fbea0 2024-02-10T08:58:37.087 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:37.066+0000 7f1a637fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a640fbea0 0x7f1a6410ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:37.087 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:37.066+0000 7f1a637fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a640fbea0 0x7f1a6410ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44370/0 (socket says 172.21.15.77:44370) 2024-02-10T08:58:37.087 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:37.066+0000 7f1a637fe700 1 -- 172.21.15.77:0/1398225433 learned_addr learned my addr 172.21.15.77:0/1398225433 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-10T08:58:37.087 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:37.067+0000 7f1a637fe700 1 -- 172.21.15.77:0/1398225433 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a5c00b200 con 0x7f1a640fbea0 2024-02-10T08:58:37.087 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:37.067+0000 7f1a637fe700 1 --2- 172.21.15.77:0/1398225433 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a640fbea0 0x7f1a6410ab80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f1a5c00bf40 tx=0x7f1a5c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:37.087 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:37.067+0000 7f1a4bfff700 1 -- 172.21.15.77:0/1398225433 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a5c01b570 con 0x7f1a640fbea0 2024-02-10T08:58:37.087 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:37.067+0000 7f1a4bfff700 1 -- 172.21.15.77:0/1398225433 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f1a5c01bb60 con 0x7f1a640fbea0 2024-02-10T08:58:37.087 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:37.067+0000 7f1a6a208700 1 -- 172.21.15.77:0/1398225433 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a6410b270 con 0x7f1a640fbea0 2024-02-10T08:58:37.087 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:37.067+0000 7f1a4bfff700 1 -- 172.21.15.77:0/1398225433 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a5c01aa20 con 0x7f1a640fbea0 2024-02-10T08:58:37.087 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:37.067+0000 7f1a6a208700 1 -- 172.21.15.77:0/1398225433 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a6410b700 con 0x7f1a640fbea0 2024-02-10T08:58:37.087 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:37.067+0000 7f1a4bfff700 1 -- 172.21.15.77:0/1398225433 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f1a5c00f070 con 0x7f1a640fbea0 2024-02-10T08:58:37.087 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:37.068+0000 7f1a4bfff700 1 --2- 172.21.15.77:0/1398225433 >> [v2:172.21.15.77:6800/3154363712,v1:172.21.15.77:6801/3154363712] conn(0x7f1a5002ff20 0x7f1a500323e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:37.088 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:37.068+0000 7f1a6a208700 1 -- 172.21.15.77:0/1398225433 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a64045ad0 con 0x7f1a640fbea0 2024-02-10T08:58:37.088 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:37.068+0000 7f1a4bfff700 1 -- 172.21.15.77:0/1398225433 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f1a5c041600 con 0x7f1a640fbea0 2024-02-10T08:58:37.088 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:37.068+0000 7f1a62ffd700 1 --2- 172.21.15.77:0/1398225433 >> [v2:172.21.15.77:6800/3154363712,v1:172.21.15.77:6801/3154363712] conn(0x7f1a5002ff20 0x7f1a500323e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:37.088 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:37.068+0000 7f1a62ffd700 1 --2- 172.21.15.77:0/1398225433 >> [v2:172.21.15.77:6800/3154363712,v1:172.21.15.77:6801/3154363712] conn(0x7f1a5002ff20 0x7f1a500323e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f1a580096a0 tx=0x7f1a58006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:37.088 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:37.072+0000 7f1a4bfff700 1 -- 172.21.15.77:0/1398225433 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1a5c01ab80 con 0x7f1a640fbea0 2024-02-10T08:58:37.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:36 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:35.854026+0000 mgr.smithi077.dxntvw (mgr.14142) 7 : audit [DBG] from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:58:37.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:36 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cephadm 2024-02-10T08:58:35.854540+0000 mgr.smithi077.dxntvw (mgr.14142) 8 : cephadm [INF] Saving service alertmanager spec with placement count:1 2024-02-10T08:58:37.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:36 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:35.856777+0000 mon.smithi077 (mon.0) 81 : audit [INF] from='mgr.14142 172.21.15.77:0/3691491939' entity='mgr.smithi077.dxntvw' 2024-02-10T08:58:37.223 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:37.222+0000 7f1a6a208700 1 -- 172.21.15.77:0/1398225433 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f1a640fdc00 con 0x7f1a640fbea0 2024-02-10T08:58:37.226 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:37.225+0000 7f1a4bfff700 1 -- 172.21.15.77:0/1398225433 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v8)=0 v8) v1 ==== 142+0+0 (secure 0 0 0) 0x7f1a5c019ab0 con 0x7f1a640fbea0 2024-02-10T08:58:37.228 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:37.227+0000 7f1a6a208700 1 -- 172.21.15.77:0/1398225433 >> [v2:172.21.15.77:6800/3154363712,v1:172.21.15.77:6801/3154363712] conn(0x7f1a5002ff20 msgr2=0x7f1a500323e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:37.228 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:37.227+0000 7f1a6a208700 1 --2- 172.21.15.77:0/1398225433 >> [v2:172.21.15.77:6800/3154363712,v1:172.21.15.77:6801/3154363712] conn(0x7f1a5002ff20 0x7f1a500323e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f1a580096a0 tx=0x7f1a58006b40).stop 2024-02-10T08:58:37.228 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:37.227+0000 7f1a6a208700 1 -- 172.21.15.77:0/1398225433 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a640fbea0 msgr2=0x7f1a6410ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:37.228 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:37.227+0000 7f1a6a208700 1 --2- 172.21.15.77:0/1398225433 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a640fbea0 0x7f1a6410ab80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f1a5c00bf40 tx=0x7f1a5c006ee0).stop 2024-02-10T08:58:37.231 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:37.228+0000 7f1a6a208700 1 -- 172.21.15.77:0/1398225433 shutdown_connections 2024-02-10T08:58:37.231 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:37.228+0000 7f1a6a208700 1 --2- 172.21.15.77:0/1398225433 >> [v2:172.21.15.77:6800/3154363712,v1:172.21.15.77:6801/3154363712] conn(0x7f1a5002ff20 0x7f1a500323e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:37.231 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:37.228+0000 7f1a6a208700 1 --2- 172.21.15.77:0/1398225433 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a640fbea0 0x7f1a6410ab80 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:37.231 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:37.228+0000 7f1a6a208700 1 -- 172.21.15.77:0/1398225433 >> 172.21.15.77:0/1398225433 conn(0x7f1a640f76a0 msgr2=0x7f1a640f9020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:58:37.231 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:37.228+0000 7f1a6a208700 1 -- 172.21.15.77:0/1398225433 shutdown_connections 2024-02-10T08:58:37.231 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:37.228+0000 7f1a6a208700 1 -- 172.21.15.77:0/1398225433 wait complete. 2024-02-10T08:58:37.894 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpzzyygg7o:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpx5m65_qh:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/dashboard/ssl_server_port 8443 2024-02-10T08:58:38.648 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:38 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:37.226621+0000 mon.smithi077 (mon.0) 82 : audit [INF] from='client.? 172.21.15.77:0/1398225433' entity='client.admin' 2024-02-10T08:58:38.681 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:38.677+0000 7f84a279e700 1 Processor -- start 2024-02-10T08:58:38.682 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:38.677+0000 7f84a279e700 1 -- start start 2024-02-10T08:58:38.682 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:38.677+0000 7f84a279e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f849c070430 0x7f849c0fea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:38.682 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:38.677+0000 7f84a279e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f849c0fef40 con 0x7f849c070430 2024-02-10T08:58:38.682 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:38.677+0000 7f849bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f849c070430 0x7f849c0fea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:38.682 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:38.677+0000 7f849bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f849c070430 0x7f849c0fea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56388/0 (socket says 172.21.15.77:56388) 2024-02-10T08:58:38.682 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:38.677+0000 7f849bfff700 1 -- 172.21.15.77:0/539724001 learned_addr learned my addr 172.21.15.77:0/539724001 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-10T08:58:38.682 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:38.678+0000 7f849bfff700 1 -- 172.21.15.77:0/539724001 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f849c0ff080 con 0x7f849c070430 2024-02-10T08:58:38.682 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:38.678+0000 7f849bfff700 1 --2- 172.21.15.77:0/539724001 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f849c070430 0x7f849c0fea50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f848c00b5f0 tx=0x7f848c006ee0).ready entity=mon.0 client_cookie=520e9c19b21b86a3 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:38.682 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:38.678+0000 7f849affd700 1 -- 172.21.15.77:0/539724001 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f848c00d600 con 0x7f849c070430 2024-02-10T08:58:38.682 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:38.678+0000 7f849affd700 1 -- 172.21.15.77:0/539724001 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f848c00dbf0 con 0x7f849c070430 2024-02-10T08:58:38.682 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:38.678+0000 7f849affd700 1 -- 172.21.15.77:0/539724001 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f848c011cb0 con 0x7f849c070430 2024-02-10T08:58:38.682 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:38.678+0000 7f84a279e700 1 -- 172.21.15.77:0/539724001 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f849c070430 msgr2=0x7f849c0fea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:38.683 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:38.678+0000 7f84a279e700 1 --2- 172.21.15.77:0/539724001 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f849c070430 0x7f849c0fea50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f848c00b5f0 tx=0x7f848c006ee0).stop 2024-02-10T08:58:38.683 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:38.678+0000 7f84a279e700 1 -- 172.21.15.77:0/539724001 shutdown_connections 2024-02-10T08:58:38.683 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:38.678+0000 7f84a279e700 1 --2- 172.21.15.77:0/539724001 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f849c070430 0x7f849c0fea50 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:38.683 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:38.678+0000 7f84a279e700 1 -- 172.21.15.77:0/539724001 >> 172.21.15.77:0/539724001 conn(0x7f849c0f6530 msgr2=0x7f849c0f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:58:38.683 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:38.678+0000 7f84a279e700 1 -- 172.21.15.77:0/539724001 shutdown_connections 2024-02-10T08:58:38.683 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:38.679+0000 7f84a279e700 1 -- 172.21.15.77:0/539724001 wait complete. 2024-02-10T08:58:38.683 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:38.679+0000 7f84a279e700 1 Processor -- start 2024-02-10T08:58:38.683 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:38.679+0000 7f84a279e700 1 -- start start 2024-02-10T08:58:38.683 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:38.679+0000 7f84a279e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f849c070430 0x7f849c10ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:38.683 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:38.679+0000 7f84a279e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f849c10b020 con 0x7f849c070430 2024-02-10T08:58:38.683 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:38.680+0000 7f849bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f849c070430 0x7f849c10ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:38.683 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:38.680+0000 7f849bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f849c070430 0x7f849c10ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56392/0 (socket says 172.21.15.77:56392) 2024-02-10T08:58:38.683 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:38.680+0000 7f849bfff700 1 -- 172.21.15.77:0/2969454883 learned_addr learned my addr 172.21.15.77:0/2969454883 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-10T08:58:38.683 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:38.680+0000 7f849bfff700 1 -- 172.21.15.77:0/2969454883 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f848c00b200 con 0x7f849c070430 2024-02-10T08:58:38.684 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:38.680+0000 7f849bfff700 1 --2- 172.21.15.77:0/2969454883 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f849c070430 0x7f849c10ab30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f848c006800 tx=0x7f848c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:38.684 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:38.680+0000 7f8498ff9700 1 -- 172.21.15.77:0/2969454883 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f848c01b570 con 0x7f849c070430 2024-02-10T08:58:38.684 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:38.680+0000 7f84a279e700 1 -- 172.21.15.77:0/2969454883 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f849c10b220 con 0x7f849c070430 2024-02-10T08:58:38.684 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:38.680+0000 7f8498ff9700 1 -- 172.21.15.77:0/2969454883 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f848c01bb60 con 0x7f849c070430 2024-02-10T08:58:38.684 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:38.680+0000 7f84a279e700 1 -- 172.21.15.77:0/2969454883 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f849c10b6b0 con 0x7f849c070430 2024-02-10T08:58:38.684 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:38.680+0000 7f8498ff9700 1 -- 172.21.15.77:0/2969454883 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f848c01a9b0 con 0x7f849c070430 2024-02-10T08:58:38.684 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:38.681+0000 7f8498ff9700 1 -- 172.21.15.77:0/2969454883 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f848c00f070 con 0x7f849c070430 2024-02-10T08:58:38.685 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:38.681+0000 7f8498ff9700 1 --2- 172.21.15.77:0/2969454883 >> [v2:172.21.15.77:6800/3154363712,v1:172.21.15.77:6801/3154363712] conn(0x7f848802ff20 0x7f84880323e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:38.685 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:38.681+0000 7f8498ff9700 1 -- 172.21.15.77:0/2969454883 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f848c0424a0 con 0x7f849c070430 2024-02-10T08:58:38.685 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:38.681+0000 7f849b7fe700 1 --2- 172.21.15.77:0/2969454883 >> [v2:172.21.15.77:6800/3154363712,v1:172.21.15.77:6801/3154363712] conn(0x7f848802ff20 0x7f84880323e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:38.685 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:38.681+0000 7f84a279e700 1 -- 172.21.15.77:0/2969454883 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f849c045ad0 con 0x7f849c070430 2024-02-10T08:58:38.685 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:38.682+0000 7f849b7fe700 1 --2- 172.21.15.77:0/2969454883 >> [v2:172.21.15.77:6800/3154363712,v1:172.21.15.77:6801/3154363712] conn(0x7f848802ff20 0x7f84880323e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f84900096a0 tx=0x7f8490006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:38.686 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:38.685+0000 7f8498ff9700 1 -- 172.21.15.77:0/2969454883 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f848c016070 con 0x7f849c070430 2024-02-10T08:58:38.844 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:38.843+0000 7f84a279e700 1 -- 172.21.15.77:0/2969454883 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f849c10cc60 con 0x7f849c070430 2024-02-10T08:58:38.847 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:38.846+0000 7f8498ff9700 1 -- 172.21.15.77:0/2969454883 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v9)=0 v9) v1 ==== 130+0+0 (secure 0 0 0) 0x7f848c025460 con 0x7f849c070430 2024-02-10T08:58:38.850 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:38.848+0000 7f84a279e700 1 -- 172.21.15.77:0/2969454883 >> [v2:172.21.15.77:6800/3154363712,v1:172.21.15.77:6801/3154363712] conn(0x7f848802ff20 msgr2=0x7f84880323e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:38.850 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:38.848+0000 7f84a279e700 1 --2- 172.21.15.77:0/2969454883 >> [v2:172.21.15.77:6800/3154363712,v1:172.21.15.77:6801/3154363712] conn(0x7f848802ff20 0x7f84880323e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f84900096a0 tx=0x7f8490006b40).stop 2024-02-10T08:58:38.850 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:38.848+0000 7f84a279e700 1 -- 172.21.15.77:0/2969454883 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f849c070430 msgr2=0x7f849c10ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:38.850 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:38.848+0000 7f84a279e700 1 --2- 172.21.15.77:0/2969454883 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f849c070430 0x7f849c10ab30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f848c006800 tx=0x7f848c006ee0).stop 2024-02-10T08:58:38.850 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:38.848+0000 7f84a279e700 1 -- 172.21.15.77:0/2969454883 shutdown_connections 2024-02-10T08:58:38.850 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:38.848+0000 7f84a279e700 1 --2- 172.21.15.77:0/2969454883 >> [v2:172.21.15.77:6800/3154363712,v1:172.21.15.77:6801/3154363712] conn(0x7f848802ff20 0x7f84880323e0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:38.850 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:38.848+0000 7f84a279e700 1 --2- 172.21.15.77:0/2969454883 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f849c070430 0x7f849c10ab30 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:38.850 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:38.848+0000 7f84a279e700 1 -- 172.21.15.77:0/2969454883 >> 172.21.15.77:0/2969454883 conn(0x7f849c0f6530 msgr2=0x7f849c0f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:58:38.850 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:38.849+0000 7f84a279e700 1 -- 172.21.15.77:0/2969454883 shutdown_connections 2024-02-10T08:58:38.850 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:38.849+0000 7f84a279e700 1 -- 172.21.15.77:0/2969454883 wait complete. 2024-02-10T08:58:39.899 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:39 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:38.847420+0000 mon.smithi077 (mon.0) 83 : audit [INF] from='client.? 172.21.15.77:0/2969454883' entity='client.admin' 2024-02-10T08:58:40.350 INFO:teuthology.orchestra.run.smithi077.stderr:Enabling the dashboard module... 2024-02-10T08:58:40.350 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpzzyygg7o:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpx5m65_qh:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable dashboard 2024-02-10T08:58:40.946 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:40.916+0000 7ff02c30b700 1 Processor -- start 2024-02-10T08:58:40.946 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:40.916+0000 7ff02c30b700 1 -- start start 2024-02-10T08:58:40.946 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:40.916+0000 7ff02c30b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff024070430 0x7ff0240fea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:40.946 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:40.916+0000 7ff02c30b700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0240fef40 con 0x7ff024070430 2024-02-10T08:58:40.946 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:40.917+0000 7ff02a0a7700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff024070430 0x7ff0240fea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:40.946 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:40.917+0000 7ff02a0a7700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff024070430 0x7ff0240fea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56404/0 (socket says 172.21.15.77:56404) 2024-02-10T08:58:40.946 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:40.917+0000 7ff02a0a7700 1 -- 172.21.15.77:0/3271092746 learned_addr learned my addr 172.21.15.77:0/3271092746 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-10T08:58:40.946 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:40.917+0000 7ff02a0a7700 1 -- 172.21.15.77:0/3271092746 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0240ff080 con 0x7ff024070430 2024-02-10T08:58:40.947 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:40.917+0000 7ff02a0a7700 1 --2- 172.21.15.77:0/3271092746 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff024070430 0x7ff0240fea50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7ff01c00b5f0 tx=0x7ff01c006ee0).ready entity=mon.0 client_cookie=12b31ccc8f834d72 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:40.947 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:40.917+0000 7ff0290a5700 1 -- 172.21.15.77:0/3271092746 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff01c00d600 con 0x7ff024070430 2024-02-10T08:58:40.947 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:40.917+0000 7ff0290a5700 1 -- 172.21.15.77:0/3271092746 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff01c00dc30 con 0x7ff024070430 2024-02-10T08:58:40.947 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:40.917+0000 7ff0290a5700 1 -- 172.21.15.77:0/3271092746 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff01c011d30 con 0x7ff024070430 2024-02-10T08:58:40.947 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:40.918+0000 7ff02c30b700 1 -- 172.21.15.77:0/3271092746 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff024070430 msgr2=0x7ff0240fea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:40.947 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:40.918+0000 7ff02c30b700 1 --2- 172.21.15.77:0/3271092746 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff024070430 0x7ff0240fea50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7ff01c00b5f0 tx=0x7ff01c006ee0).stop 2024-02-10T08:58:40.947 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:40.918+0000 7ff02c30b700 1 -- 172.21.15.77:0/3271092746 shutdown_connections 2024-02-10T08:58:40.947 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:40.918+0000 7ff02c30b700 1 --2- 172.21.15.77:0/3271092746 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff024070430 0x7ff0240fea50 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:40.947 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:40.918+0000 7ff02c30b700 1 -- 172.21.15.77:0/3271092746 >> 172.21.15.77:0/3271092746 conn(0x7ff0240f6530 msgr2=0x7ff0240f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:58:40.947 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:40.918+0000 7ff02c30b700 1 -- 172.21.15.77:0/3271092746 shutdown_connections 2024-02-10T08:58:40.947 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:40.918+0000 7ff02c30b700 1 -- 172.21.15.77:0/3271092746 wait complete. 2024-02-10T08:58:40.947 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:40.918+0000 7ff02c30b700 1 Processor -- start 2024-02-10T08:58:40.947 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:40.918+0000 7ff02c30b700 1 -- start start 2024-02-10T08:58:40.947 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:40.918+0000 7ff02c30b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff024070430 0x7ff0240fc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:40.947 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:40.918+0000 7ff02c30b700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0240fa950 con 0x7ff024070430 2024-02-10T08:58:40.948 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:40.918+0000 7ff02a0a7700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff024070430 0x7ff0240fc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:40.948 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:40.918+0000 7ff02a0a7700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff024070430 0x7ff0240fc4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56418/0 (socket says 172.21.15.77:56418) 2024-02-10T08:58:40.948 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:40.918+0000 7ff02a0a7700 1 -- 172.21.15.77:0/4256159547 learned_addr learned my addr 172.21.15.77:0/4256159547 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-10T08:58:40.948 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:40.919+0000 7ff02a0a7700 1 -- 172.21.15.77:0/4256159547 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff01c00b200 con 0x7ff024070430 2024-02-10T08:58:40.948 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:40.919+0000 7ff02a0a7700 1 --2- 172.21.15.77:0/4256159547 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff024070430 0x7ff0240fc4d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7ff01c00e040 tx=0x7ff01c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:40.948 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:40.919+0000 7ff012ffd700 1 -- 172.21.15.77:0/4256159547 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff01c00da60 con 0x7ff024070430 2024-02-10T08:58:40.948 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:40.919+0000 7ff012ffd700 1 -- 172.21.15.77:0/4256159547 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff01c01b8e0 con 0x7ff024070430 2024-02-10T08:58:40.948 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:40.919+0000 7ff02c30b700 1 -- 172.21.15.77:0/4256159547 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0240faaf0 con 0x7ff024070430 2024-02-10T08:58:40.948 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:40.919+0000 7ff012ffd700 1 -- 172.21.15.77:0/4256159547 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff01c01a900 con 0x7ff024070430 2024-02-10T08:58:40.948 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:40.919+0000 7ff02c30b700 1 -- 172.21.15.77:0/4256159547 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0240faf80 con 0x7ff024070430 2024-02-10T08:58:40.948 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:40.920+0000 7ff012ffd700 1 -- 172.21.15.77:0/4256159547 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7ff01c00f070 con 0x7ff024070430 2024-02-10T08:58:40.948 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:40.920+0000 7ff012ffd700 1 --2- 172.21.15.77:0/4256159547 >> [v2:172.21.15.77:6800/3154363712,v1:172.21.15.77:6801/3154363712] conn(0x7ff00c02ff20 0x7ff00c0323e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:40.948 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:40.920+0000 7ff012ffd700 1 -- 172.21.15.77:0/4256159547 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7ff01c041420 con 0x7ff024070430 2024-02-10T08:58:40.948 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:40.920+0000 7ff0298a6700 1 --2- 172.21.15.77:0/4256159547 >> [v2:172.21.15.77:6800/3154363712,v1:172.21.15.77:6801/3154363712] conn(0x7ff00c02ff20 0x7ff00c0323e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:40.948 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:40.920+0000 7ff02c30b700 1 -- 172.21.15.77:0/4256159547 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff024059100 con 0x7ff024070430 2024-02-10T08:58:40.949 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:40.920+0000 7ff0298a6700 1 --2- 172.21.15.77:0/4256159547 >> [v2:172.21.15.77:6800/3154363712,v1:172.21.15.77:6801/3154363712] conn(0x7ff00c02ff20 0x7ff00c0323e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7ff0140096a0 tx=0x7ff014006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:40.949 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:40.924+0000 7ff012ffd700 1 -- 172.21.15.77:0/4256159547 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff01c016070 con 0x7ff024070430 2024-02-10T08:58:41.133 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:41.132+0000 7ff02c30b700 1 -- 172.21.15.77:0/4256159547 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7ff024045ad0 con 0x7ff024070430 2024-02-10T08:58:41.398 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:40 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:39.927951+0000 mon.smithi077 (mon.0) 84 : audit [INF] from='mgr.14142 172.21.15.77:0/3691491939' entity='mgr.smithi077.dxntvw' 2024-02-10T08:58:41.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:40 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:40.685018+0000 mon.smithi077 (mon.0) 85 : audit [INF] from='mgr.14142 172.21.15.77:0/3691491939' entity='mgr.smithi077.dxntvw' 2024-02-10T08:58:41.930 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:41.929+0000 7ff012ffd700 1 -- 172.21.15.77:0/4256159547 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7ff01c025980 con 0x7ff024070430 2024-02-10T08:58:41.931 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:41.931+0000 7ff012ffd700 1 -- 172.21.15.77:0/4256159547 <== mon.0 v2:172.21.15.77:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v13) v1 ==== 88+0+0 (secure 0 0 0) 0x7ff01c02eb70 con 0x7ff024070430 2024-02-10T08:58:41.935 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:41.932+0000 7ff02c30b700 1 -- 172.21.15.77:0/4256159547 >> [v2:172.21.15.77:6800/3154363712,v1:172.21.15.77:6801/3154363712] conn(0x7ff00c02ff20 msgr2=0x7ff00c0323e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:41.935 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:41.932+0000 7ff02c30b700 1 --2- 172.21.15.77:0/4256159547 >> [v2:172.21.15.77:6800/3154363712,v1:172.21.15.77:6801/3154363712] conn(0x7ff00c02ff20 0x7ff00c0323e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7ff0140096a0 tx=0x7ff014006b40).stop 2024-02-10T08:58:41.935 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:41.932+0000 7ff02c30b700 1 -- 172.21.15.77:0/4256159547 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff024070430 msgr2=0x7ff0240fc4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:41.935 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:41.932+0000 7ff02c30b700 1 --2- 172.21.15.77:0/4256159547 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff024070430 0x7ff0240fc4d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7ff01c00e040 tx=0x7ff01c006ee0).stop 2024-02-10T08:58:41.935 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:41.933+0000 7ff02c30b700 1 -- 172.21.15.77:0/4256159547 shutdown_connections 2024-02-10T08:58:41.935 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:41.933+0000 7ff02c30b700 1 --2- 172.21.15.77:0/4256159547 >> [v2:172.21.15.77:6800/3154363712,v1:172.21.15.77:6801/3154363712] conn(0x7ff00c02ff20 0x7ff00c0323e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:41.935 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:41.933+0000 7ff02c30b700 1 --2- 172.21.15.77:0/4256159547 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff024070430 0x7ff0240fc4d0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:41.935 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:41.933+0000 7ff02c30b700 1 -- 172.21.15.77:0/4256159547 >> 172.21.15.77:0/4256159547 conn(0x7ff0240f6530 msgr2=0x7ff0240f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:58:41.935 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:41.934+0000 7ff02c30b700 1 -- 172.21.15.77:0/4256159547 shutdown_connections 2024-02-10T08:58:41.936 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:41.934+0000 7ff02c30b700 1 -- 172.21.15.77:0/4256159547 wait complete. 2024-02-10T08:58:42.281 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:41 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:41.134427+0000 mon.smithi077 (mon.0) 86 : audit [INF] from='client.? 172.21.15.77:0/4256159547' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-02-10T08:58:42.282 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:41 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:41.835158+0000 mon.smithi077 (mon.0) 87 : audit [INF] from='mgr.14142 172.21.15.77:0/3691491939' entity='mgr.smithi077.dxntvw' 2024-02-10T08:58:43.148 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpzzyygg7o:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpx5m65_qh:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2024-02-10T08:58:43.361 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:42 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:41.932135+0000 mon.smithi077 (mon.0) 88 : audit [INF] from='client.? 172.21.15.77:0/4256159547' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-02-10T08:58:43.361 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:42 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T08:58:41.932198+0000 mon.smithi077 (mon.0) 89 : cluster [DBG] mgrmap e13: smithi077.dxntvw(active, since 11s) 2024-02-10T08:58:44.672 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:44.654+0000 7f40dc694700 1 Processor -- start 2024-02-10T08:58:44.672 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:44.655+0000 7f40dc694700 1 -- start start 2024-02-10T08:58:44.672 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:44.655+0000 7f40dc694700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40d40703d0 0x7f40d4100eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:44.672 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:44.655+0000 7f40dc694700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40d40fc8a0 con 0x7f40d40703d0 2024-02-10T08:58:44.672 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:44.655+0000 7f40da430700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40d40703d0 0x7f40d4100eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:44.672 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:44.655+0000 7f40da430700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40d40703d0 0x7f40d4100eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56448/0 (socket says 172.21.15.77:56448) 2024-02-10T08:58:44.672 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:44.655+0000 7f40da430700 1 -- 172.21.15.77:0/3790167597 learned_addr learned my addr 172.21.15.77:0/3790167597 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-10T08:58:44.672 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:44.655+0000 7f40da430700 1 -- 172.21.15.77:0/3790167597 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40d41013a0 con 0x7f40d40703d0 2024-02-10T08:58:44.672 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:44.655+0000 7f40da430700 1 --2- 172.21.15.77:0/3790167597 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40d40703d0 0x7f40d4100eb0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f40d000b5f0 tx=0x7f40d0006ee0).ready entity=mon.0 client_cookie=8396da403b4c2905 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:44.672 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:44.656+0000 7f40d942e700 1 -- 172.21.15.77:0/3790167597 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40d000d600 con 0x7f40d40703d0 2024-02-10T08:58:44.672 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:44.656+0000 7f40d942e700 1 -- 172.21.15.77:0/3790167597 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f40d000dc30 con 0x7f40d40703d0 2024-02-10T08:58:44.673 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:44.656+0000 7f40d942e700 1 -- 172.21.15.77:0/3790167597 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40d0011d30 con 0x7f40d40703d0 2024-02-10T08:58:44.673 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:44.656+0000 7f40dc694700 1 -- 172.21.15.77:0/3790167597 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40d40703d0 msgr2=0x7f40d4100eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:44.673 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:44.656+0000 7f40dc694700 1 --2- 172.21.15.77:0/3790167597 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40d40703d0 0x7f40d4100eb0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f40d000b5f0 tx=0x7f40d0006ee0).stop 2024-02-10T08:58:44.673 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:44.656+0000 7f40dc694700 1 -- 172.21.15.77:0/3790167597 shutdown_connections 2024-02-10T08:58:44.673 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:44.656+0000 7f40dc694700 1 --2- 172.21.15.77:0/3790167597 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40d40703d0 0x7f40d4100eb0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:44.673 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:44.656+0000 7f40dc694700 1 -- 172.21.15.77:0/3790167597 >> 172.21.15.77:0/3790167597 conn(0x7f40d40f6530 msgr2=0x7f40d40f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:58:44.673 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:44.656+0000 7f40dc694700 1 -- 172.21.15.77:0/3790167597 shutdown_connections 2024-02-10T08:58:44.673 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:44.656+0000 7f40dc694700 1 -- 172.21.15.77:0/3790167597 wait complete. 2024-02-10T08:58:44.673 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:44.657+0000 7f40dc694700 1 Processor -- start 2024-02-10T08:58:44.673 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:44.657+0000 7f40dc694700 1 -- start start 2024-02-10T08:58:44.673 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:44.657+0000 7f40dc694700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40d40703d0 0x7f40d410ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:44.673 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:44.657+0000 7f40dc694700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40d410d330 con 0x7f40d40703d0 2024-02-10T08:58:44.673 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:44.657+0000 7f40da430700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40d40703d0 0x7f40d410ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:44.673 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:44.657+0000 7f40da430700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40d40703d0 0x7f40d410ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56458/0 (socket says 172.21.15.77:56458) 2024-02-10T08:58:44.674 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:44.657+0000 7f40da430700 1 -- 172.21.15.77:0/2131809835 learned_addr learned my addr 172.21.15.77:0/2131809835 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-10T08:58:44.674 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:44.657+0000 7f40da430700 1 -- 172.21.15.77:0/2131809835 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40d000b200 con 0x7f40d40703d0 2024-02-10T08:58:44.674 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:44.657+0000 7f40da430700 1 --2- 172.21.15.77:0/2131809835 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40d40703d0 0x7f40d410ce40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f40d000c010 tx=0x7f40d0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:44.674 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:44.658+0000 7f40c2ffd700 1 -- 172.21.15.77:0/2131809835 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40d000da60 con 0x7f40d40703d0 2024-02-10T08:58:44.674 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:44.658+0000 7f40c2ffd700 1 -- 172.21.15.77:0/2131809835 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f40d001b8e0 con 0x7f40d40703d0 2024-02-10T08:58:44.674 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:44.658+0000 7f40dc694700 1 -- 172.21.15.77:0/2131809835 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40d410d530 con 0x7f40d40703d0 2024-02-10T08:58:44.674 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:44.658+0000 7f40c2ffd700 1 -- 172.21.15.77:0/2131809835 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40d001a990 con 0x7f40d40703d0 2024-02-10T08:58:44.674 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:44.658+0000 7f40dc694700 1 -- 172.21.15.77:0/2131809835 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40d410e140 con 0x7f40d40703d0 2024-02-10T08:58:44.674 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:44.658+0000 7f40dc694700 1 -- 172.21.15.77:0/2131809835 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40d4059100 con 0x7f40d40703d0 2024-02-10T08:58:44.674 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:44.659+0000 7f40c2ffd700 1 -- 172.21.15.77:0/2131809835 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f40d000f070 con 0x7f40d40703d0 2024-02-10T08:58:44.674 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:44.659+0000 7f40c2ffd700 1 --2- 172.21.15.77:0/2131809835 >> [v2:172.21.15.77:6800/3154363712,v1:172.21.15.77:6801/3154363712] conn(0x7f40bc02ff70 0x7f40bc032430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:44.674 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:44.659+0000 7f40c2ffd700 1 -- 172.21.15.77:0/2131809835 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f40d0041430 con 0x7f40d40703d0 2024-02-10T08:58:44.674 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:44.659+0000 7f40d9c2f700 1 -- 172.21.15.77:0/2131809835 >> [v2:172.21.15.77:6800/3154363712,v1:172.21.15.77:6801/3154363712] conn(0x7f40bc02ff70 msgr2=0x7f40bc032430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6800/3154363712 2024-02-10T08:58:44.674 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:44.659+0000 7f40d9c2f700 1 --2- 172.21.15.77:0/2131809835 >> [v2:172.21.15.77:6800/3154363712,v1:172.21.15.77:6801/3154363712] conn(0x7f40bc02ff70 0x7f40bc032430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-10T08:58:44.674 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:44.663+0000 7f40c2ffd700 1 -- 172.21.15.77:0/2131809835 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f40d0010360 con 0x7f40d40703d0 2024-02-10T08:58:44.861 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:44.859+0000 7f40dc694700 1 -- 172.21.15.77:0/2131809835 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f40d41068d0 con 0x7f40d40703d0 2024-02-10T08:58:44.861 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:44.859+0000 7f40c2ffd700 1 -- 172.21.15.77:0/2131809835 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v13) v1 ==== 56+0+104 (secure 0 0 0) 0x7f40d0016070 con 0x7f40d40703d0 2024-02-10T08:58:44.861 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: { 2024-02-10T08:58:44.861 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "epoch": 13, 2024-02-10T08:58:44.861 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "available": true, 2024-02-10T08:58:44.861 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "active_name": "smithi077.dxntvw", 2024-02-10T08:58:44.861 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "num_standby": 0 2024-02-10T08:58:44.861 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: } 2024-02-10T08:58:44.861 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:44.860+0000 7f40d9c2f700 1 -- 172.21.15.77:0/2131809835 >> [v2:172.21.15.77:6800/3154363712,v1:172.21.15.77:6801/3154363712] conn(0x7f40bc02ff70 msgr2=0x7f40bc032430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6800/3154363712 2024-02-10T08:58:44.861 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:44.860+0000 7f40d9c2f700 1 --2- 172.21.15.77:0/2131809835 >> [v2:172.21.15.77:6800/3154363712,v1:172.21.15.77:6801/3154363712] conn(0x7f40bc02ff70 0x7f40bc032430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-10T08:58:44.863 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:44.861+0000 7f40dc694700 1 -- 172.21.15.77:0/2131809835 >> [v2:172.21.15.77:6800/3154363712,v1:172.21.15.77:6801/3154363712] conn(0x7f40bc02ff70 msgr2=0x7f40bc032430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-10T08:58:44.863 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:44.861+0000 7f40dc694700 1 --2- 172.21.15.77:0/2131809835 >> [v2:172.21.15.77:6800/3154363712,v1:172.21.15.77:6801/3154363712] conn(0x7f40bc02ff70 0x7f40bc032430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-10T08:58:44.863 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:44.861+0000 7f40dc694700 1 -- 172.21.15.77:0/2131809835 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40d40703d0 msgr2=0x7f40d410ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:44.863 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:44.861+0000 7f40dc694700 1 --2- 172.21.15.77:0/2131809835 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40d40703d0 0x7f40d410ce40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f40d000c010 tx=0x7f40d0006ee0).stop 2024-02-10T08:58:44.863 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:44.861+0000 7f40dc694700 1 -- 172.21.15.77:0/2131809835 shutdown_connections 2024-02-10T08:58:44.863 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:44.861+0000 7f40dc694700 1 --2- 172.21.15.77:0/2131809835 >> [v2:172.21.15.77:6800/3154363712,v1:172.21.15.77:6801/3154363712] conn(0x7f40bc02ff70 0x7f40bc032430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-10T08:58:44.863 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:44.861+0000 7f40dc694700 1 --2- 172.21.15.77:0/2131809835 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40d40703d0 0x7f40d410ce40 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:44.863 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:44.861+0000 7f40dc694700 1 -- 172.21.15.77:0/2131809835 >> 172.21.15.77:0/2131809835 conn(0x7f40d40f6530 msgr2=0x7f40d40f7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:58:44.863 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:44.861+0000 7f40dc694700 1 -- 172.21.15.77:0/2131809835 shutdown_connections 2024-02-10T08:58:44.863 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:44.861+0000 7f40dc694700 1 -- 172.21.15.77:0/2131809835 wait complete. 2024-02-10T08:58:45.148 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:44 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:44.860595+0000 mon.smithi077 (mon.0) 90 : audit [DBG] from='client.? 172.21.15.77:0/2131809835' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-10T08:58:45.558 INFO:teuthology.orchestra.run.smithi077.stderr:Waiting for the mgr to restart... 2024-02-10T08:58:45.558 INFO:teuthology.orchestra.run.smithi077.stderr:Waiting for mgr epoch 13... 2024-02-10T08:58:45.559 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpzzyygg7o:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpx5m65_qh:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2024-02-10T08:58:46.204 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:46.167+0000 7f3e2d031700 1 Processor -- start 2024-02-10T08:58:46.204 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:46.167+0000 7f3e2d031700 1 -- start start 2024-02-10T08:58:46.204 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:46.167+0000 7f3e2d031700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e280fad90 0x7f3e280fb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:46.204 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:46.167+0000 7f3e2d031700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e280fb6a0 con 0x7f3e280fad90 2024-02-10T08:58:46.204 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:46.167+0000 7f3e26d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e280fad90 0x7f3e280fb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:46.204 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:46.167+0000 7f3e26d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e280fad90 0x7f3e280fb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56474/0 (socket says 172.21.15.77:56474) 2024-02-10T08:58:46.204 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:46.167+0000 7f3e26d9d700 1 -- 172.21.15.77:0/1188896813 learned_addr learned my addr 172.21.15.77:0/1188896813 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-10T08:58:46.204 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:46.168+0000 7f3e26d9d700 1 -- 172.21.15.77:0/1188896813 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e280fb7e0 con 0x7f3e280fad90 2024-02-10T08:58:46.204 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:46.168+0000 7f3e26d9d700 1 --2- 172.21.15.77:0/1188896813 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e280fad90 0x7f3e280fb1b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f3e1800b5f0 tx=0x7f3e18006ee0).ready entity=mon.0 client_cookie=9346df8bd6e3a515 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:46.205 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:46.168+0000 7f3e25d9b700 1 -- 172.21.15.77:0/1188896813 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e1800d600 con 0x7f3e280fad90 2024-02-10T08:58:46.205 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:46.168+0000 7f3e25d9b700 1 -- 172.21.15.77:0/1188896813 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3e1800dc30 con 0x7f3e280fad90 2024-02-10T08:58:46.205 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:46.168+0000 7f3e25d9b700 1 -- 172.21.15.77:0/1188896813 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e18011d30 con 0x7f3e280fad90 2024-02-10T08:58:46.205 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:46.168+0000 7f3e2d031700 1 -- 172.21.15.77:0/1188896813 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e280fad90 msgr2=0x7f3e280fb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:46.205 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:46.168+0000 7f3e2d031700 1 --2- 172.21.15.77:0/1188896813 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e280fad90 0x7f3e280fb1b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f3e1800b5f0 tx=0x7f3e18006ee0).stop 2024-02-10T08:58:46.205 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:46.168+0000 7f3e2d031700 1 -- 172.21.15.77:0/1188896813 shutdown_connections 2024-02-10T08:58:46.205 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:46.168+0000 7f3e2d031700 1 --2- 172.21.15.77:0/1188896813 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e280fad90 0x7f3e280fb1b0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:46.205 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:46.168+0000 7f3e2d031700 1 -- 172.21.15.77:0/1188896813 >> 172.21.15.77:0/1188896813 conn(0x7f3e280f6530 msgr2=0x7f3e280f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:58:46.205 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:46.168+0000 7f3e2d031700 1 -- 172.21.15.77:0/1188896813 shutdown_connections 2024-02-10T08:58:46.205 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:46.168+0000 7f3e2d031700 1 -- 172.21.15.77:0/1188896813 wait complete. 2024-02-10T08:58:46.205 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:46.169+0000 7f3e2d031700 1 Processor -- start 2024-02-10T08:58:46.205 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:46.169+0000 7f3e2d031700 1 -- start start 2024-02-10T08:58:46.205 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:46.170+0000 7f3e2d031700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e280fad90 0x7f3e28108920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:46.205 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:46.170+0000 7f3e2d031700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e28108e10 con 0x7f3e280fad90 2024-02-10T08:58:46.206 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:46.170+0000 7f3e26d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e280fad90 0x7f3e28108920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:46.206 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:46.170+0000 7f3e26d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e280fad90 0x7f3e28108920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56484/0 (socket says 172.21.15.77:56484) 2024-02-10T08:58:46.206 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:46.170+0000 7f3e26d9d700 1 -- 172.21.15.77:0/1857656373 learned_addr learned my addr 172.21.15.77:0/1857656373 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-10T08:58:46.206 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:46.170+0000 7f3e26d9d700 1 -- 172.21.15.77:0/1857656373 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e1800b200 con 0x7f3e280fad90 2024-02-10T08:58:46.206 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:46.170+0000 7f3e26d9d700 1 --2- 172.21.15.77:0/1857656373 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e280fad90 0x7f3e28108920 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f3e18006b80 tx=0x7f3e18006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:46.206 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:46.170+0000 7f3e0f7fe700 1 -- 172.21.15.77:0/1857656373 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e1800da60 con 0x7f3e280fad90 2024-02-10T08:58:46.206 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:46.170+0000 7f3e0f7fe700 1 -- 172.21.15.77:0/1857656373 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3e1801b8e0 con 0x7f3e280fad90 2024-02-10T08:58:46.206 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:46.170+0000 7f3e2d031700 1 -- 172.21.15.77:0/1857656373 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e28109010 con 0x7f3e280fad90 2024-02-10T08:58:46.206 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:46.170+0000 7f3e0f7fe700 1 -- 172.21.15.77:0/1857656373 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e1801a960 con 0x7f3e280fad90 2024-02-10T08:58:46.206 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:46.170+0000 7f3e2d031700 1 -- 172.21.15.77:0/1857656373 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e281094a0 con 0x7f3e280fad90 2024-02-10T08:58:46.206 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:46.171+0000 7f3e0f7fe700 1 -- 172.21.15.77:0/1857656373 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f3e1800f070 con 0x7f3e280fad90 2024-02-10T08:58:46.206 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:46.171+0000 7f3e0f7fe700 1 --2- 172.21.15.77:0/1857656373 >> [v2:172.21.15.77:6800/3154363712,v1:172.21.15.77:6801/3154363712] conn(0x7f3e14032210 0x7f3e140346d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:46.206 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:46.171+0000 7f3e0f7fe700 1 -- 172.21.15.77:0/1857656373 --> [v2:172.21.15.77:6800/3154363712,v1:172.21.15.77:6801/3154363712] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3e14034d90 con 0x7f3e14032210 2024-02-10T08:58:46.207 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:46.171+0000 7f3e0f7fe700 1 -- 172.21.15.77:0/1857656373 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f3e180414c0 con 0x7f3e280fad90 2024-02-10T08:58:46.207 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:46.171+0000 7f3e2659c700 1 -- 172.21.15.77:0/1857656373 >> [v2:172.21.15.77:6800/3154363712,v1:172.21.15.77:6801/3154363712] conn(0x7f3e14032210 msgr2=0x7f3e140346d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6800/3154363712 2024-02-10T08:58:46.207 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:46.171+0000 7f3e2659c700 1 --2- 172.21.15.77:0/1857656373 >> [v2:172.21.15.77:6800/3154363712,v1:172.21.15.77:6801/3154363712] conn(0x7f3e14032210 0x7f3e140346d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-10T08:58:46.372 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:46.372+0000 7f3e2659c700 1 -- 172.21.15.77:0/1857656373 >> [v2:172.21.15.77:6800/3154363712,v1:172.21.15.77:6801/3154363712] conn(0x7f3e14032210 msgr2=0x7f3e140346d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6800/3154363712 2024-02-10T08:58:46.372 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:46.372+0000 7f3e2659c700 1 --2- 172.21.15.77:0/1857656373 >> [v2:172.21.15.77:6800/3154363712,v1:172.21.15.77:6801/3154363712] conn(0x7f3e14032210 0x7f3e140346d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-10T08:58:46.773 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:46.773+0000 7f3e2659c700 1 -- 172.21.15.77:0/1857656373 >> [v2:172.21.15.77:6800/3154363712,v1:172.21.15.77:6801/3154363712] conn(0x7f3e14032210 msgr2=0x7f3e140346d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6800/3154363712 2024-02-10T08:58:46.773 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:46.773+0000 7f3e2659c700 1 --2- 172.21.15.77:0/1857656373 >> [v2:172.21.15.77:6800/3154363712,v1:172.21.15.77:6801/3154363712] conn(0x7f3e14032210 0x7f3e140346d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-02-10T08:58:46.949 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:46.949+0000 7f3e0f7fe700 1 -- 172.21.15.77:0/1857656373 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mgrmap(e 14) v1 ==== 35997+0+0 (secure 0 0 0) 0x7f3e1801aac0 con 0x7f3e280fad90 2024-02-10T08:58:46.950 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:46.949+0000 7f3e0f7fe700 1 -- 172.21.15.77:0/1857656373 >> [v2:172.21.15.77:6800/3154363712,v1:172.21.15.77:6801/3154363712] conn(0x7f3e14032210 msgr2=0x7f3e140346d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-10T08:58:46.950 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:46.949+0000 7f3e0f7fe700 1 --2- 172.21.15.77:0/1857656373 >> [v2:172.21.15.77:6800/3154363712,v1:172.21.15.77:6801/3154363712] conn(0x7f3e14032210 0x7f3e140346d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-10T08:58:47.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:46 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T08:58:46.897089+0000 mon.smithi077 (mon.0) 91 : cluster [INF] Active manager daemon smithi077.dxntvw restarted 2024-02-10T08:58:47.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:46 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T08:58:46.897487+0000 mon.smithi077 (mon.0) 92 : cluster [INF] Activating manager daemon smithi077.dxntvw 2024-02-10T08:58:47.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:46 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T08:58:46.899733+0000 mon.smithi077 (mon.0) 93 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-02-10T08:58:47.954 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:47.953+0000 7f3e0f7fe700 1 -- 172.21.15.77:0/1857656373 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mgrmap(e 15) v1 ==== 36067+0+0 (secure 0 0 0) 0x7f3e18019d70 con 0x7f3e280fad90 2024-02-10T08:58:47.954 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:47.953+0000 7f3e0f7fe700 1 --2- 172.21.15.77:0/1857656373 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f3e14035ea0 0x7f3e14038290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:47.955 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:47.953+0000 7f3e0f7fe700 1 -- 172.21.15.77:0/1857656373 --> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3e14034d90 con 0x7f3e14035ea0 2024-02-10T08:58:47.955 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:47.953+0000 7f3e2659c700 1 --2- 172.21.15.77:0/1857656373 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f3e14035ea0 0x7f3e14038290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:47.955 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:47.954+0000 7f3e2659c700 1 --2- 172.21.15.77:0/1857656373 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f3e14035ea0 0x7f3e14038290 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f3e1c0036d0 tx=0x7f3e1c006c50).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:47.955 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:47.954+0000 7f3e0f7fe700 1 -- 172.21.15.77:0/1857656373 <== mgr.14162 v2:172.21.15.77:6800/81674875 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f3e14034d90 con 0x7f3e14035ea0 2024-02-10T08:58:47.960 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:47.960+0000 7f3e2d031700 1 -- 172.21.15.77:0/1857656373 --> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f3e28059100 con 0x7f3e14035ea0 2024-02-10T08:58:47.961 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:47.961+0000 7f3e0f7fe700 1 -- 172.21.15.77:0/1857656373 <== mgr.14162 v2:172.21.15.77:6800/81674875 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f3e28059100 con 0x7f3e14035ea0 2024-02-10T08:58:47.961 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: { 2024-02-10T08:58:47.962 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "mgrmap_epoch": 15, 2024-02-10T08:58:47.962 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: "initialized": true 2024-02-10T08:58:47.962 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: } 2024-02-10T08:58:47.962 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:47.961+0000 7f3e2d031700 1 -- 172.21.15.77:0/1857656373 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f3e14035ea0 msgr2=0x7f3e14038290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:47.962 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:47.961+0000 7f3e2d031700 1 --2- 172.21.15.77:0/1857656373 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f3e14035ea0 0x7f3e14038290 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f3e1c0036d0 tx=0x7f3e1c006c50).stop 2024-02-10T08:58:47.963 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:47.961+0000 7f3e2d031700 1 -- 172.21.15.77:0/1857656373 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e280fad90 msgr2=0x7f3e28108920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:47.963 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:47.961+0000 7f3e2d031700 1 --2- 172.21.15.77:0/1857656373 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e280fad90 0x7f3e28108920 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f3e18006b80 tx=0x7f3e18006ee0).stop 2024-02-10T08:58:47.963 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:47.961+0000 7f3e2d031700 1 -- 172.21.15.77:0/1857656373 shutdown_connections 2024-02-10T08:58:47.963 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:47.961+0000 7f3e2d031700 1 --2- 172.21.15.77:0/1857656373 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f3e14035ea0 0x7f3e14038290 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:47.963 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:47.961+0000 7f3e2d031700 1 --2- 172.21.15.77:0/1857656373 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e280fad90 0x7f3e28108920 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:47.963 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:47.961+0000 7f3e2d031700 1 --2- 172.21.15.77:0/1857656373 >> [v2:172.21.15.77:6800/3154363712,v1:172.21.15.77:6801/3154363712] conn(0x7f3e14032210 0x7f3e140346d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-10T08:58:47.963 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:47.961+0000 7f3e2d031700 1 -- 172.21.15.77:0/1857656373 >> 172.21.15.77:0/1857656373 conn(0x7f3e280f6530 msgr2=0x7f3e280ff9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:58:47.963 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:47.962+0000 7f3e2d031700 1 -- 172.21.15.77:0/1857656373 shutdown_connections 2024-02-10T08:58:47.963 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:47.962+0000 7f3e2d031700 1 -- 172.21.15.77:0/1857656373 wait complete. 2024-02-10T08:58:48.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T08:58:46.950817+0000 mon.smithi077 (mon.0) 94 : cluster [DBG] mgrmap e14: smithi077.dxntvw(active, starting, since 0.0534113s) 2024-02-10T08:58:48.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:46.953707+0000 mon.smithi077 (mon.0) 95 : audit [DBG] from='mgr.14162 172.21.15.77:0/2029106299' entity='mgr.smithi077.dxntvw' cmd=[{"prefix": "mon metadata", "id": "smithi077"}]: dispatch 2024-02-10T08:58:48.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:46.954390+0000 mon.smithi077 (mon.0) 96 : audit [DBG] from='mgr.14162 172.21.15.77:0/2029106299' entity='mgr.smithi077.dxntvw' cmd=[{"prefix": "mgr metadata", "who": "smithi077.dxntvw", "id": "smithi077.dxntvw"}]: dispatch 2024-02-10T08:58:48.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:46.955086+0000 mon.smithi077 (mon.0) 97 : audit [DBG] from='mgr.14162 172.21.15.77:0/2029106299' entity='mgr.smithi077.dxntvw' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T08:58:48.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:46.955257+0000 mon.smithi077 (mon.0) 98 : audit [DBG] from='mgr.14162 172.21.15.77:0/2029106299' entity='mgr.smithi077.dxntvw' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T08:58:48.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:46.955432+0000 mon.smithi077 (mon.0) 99 : audit [DBG] from='mgr.14162 172.21.15.77:0/2029106299' entity='mgr.smithi077.dxntvw' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T08:58:48.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T08:58:46.970132+0000 mon.smithi077 (mon.0) 100 : cluster [INF] Manager daemon smithi077.dxntvw is now available 2024-02-10T08:58:48.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:47.067681+0000 mon.smithi077 (mon.0) 101 : audit [DBG] from='mgr.14162 172.21.15.77:0/2029106299' entity='mgr.smithi077.dxntvw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:58:48.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:47.069208+0000 mon.smithi077 (mon.0) 102 : audit [DBG] from='mgr.14162 172.21.15.77:0/2029106299' entity='mgr.smithi077.dxntvw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T08:58:48.150 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:47.085524+0000 mon.smithi077 (mon.0) 103 : audit [INF] from='mgr.14162 172.21.15.77:0/2029106299' entity='mgr.smithi077.dxntvw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi077.dxntvw/mirror_snapshot_schedule"}]: dispatch 2024-02-10T08:58:48.150 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:47.088322+0000 mon.smithi077 (mon.0) 104 : audit [INF] from='mgr.14162 172.21.15.77:0/2029106299' entity='mgr.smithi077.dxntvw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi077.dxntvw/trash_purge_schedule"}]: dispatch 2024-02-10T08:58:48.936 INFO:teuthology.orchestra.run.smithi077.stderr:mgr epoch 13 is available 2024-02-10T08:58:48.936 INFO:teuthology.orchestra.run.smithi077.stderr:Generating a dashboard self-signed certificate... 2024-02-10T08:58:48.936 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpzzyygg7o:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpx5m65_qh:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 dashboard create-self-signed-cert 2024-02-10T08:58:49.201 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:47.954873+0000 mgr.smithi077.dxntvw (mgr.14162) 1 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-10T08:58:49.201 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T08:58:47.955590+0000 mon.smithi077 (mon.0) 105 : cluster [DBG] mgrmap e15: smithi077.dxntvw(active, since 1.05818s) 2024-02-10T08:58:49.202 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:47.961185+0000 mgr.smithi077.dxntvw (mgr.14162) 2 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-10T08:58:49.202 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:48.477059+0000 mon.smithi077 (mon.0) 106 : audit [INF] from='mgr.14162 172.21.15.77:0/2029106299' entity='mgr.smithi077.dxntvw' 2024-02-10T08:58:49.768 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:49.701+0000 7f0d5a32a700 1 Processor -- start 2024-02-10T08:58:49.768 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:49.701+0000 7f0d5a32a700 1 -- start start 2024-02-10T08:58:49.768 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:49.701+0000 7f0d5a32a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d540703f0 0x7f0d540fea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:49.768 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:49.701+0000 7f0d5a32a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d540fef80 con 0x7f0d540703f0 2024-02-10T08:58:49.768 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:49.702+0000 7f0d53fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d540703f0 0x7f0d540fea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:49.768 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:49.702+0000 7f0d53fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d540703f0 0x7f0d540fea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55162/0 (socket says 172.21.15.77:55162) 2024-02-10T08:58:49.768 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:49.702+0000 7f0d53fff700 1 -- 172.21.15.77:0/2343103579 learned_addr learned my addr 172.21.15.77:0/2343103579 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-10T08:58:49.768 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:49.702+0000 7f0d53fff700 1 -- 172.21.15.77:0/2343103579 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d540ff0c0 con 0x7f0d540703f0 2024-02-10T08:58:49.768 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:49.702+0000 7f0d53fff700 1 --2- 172.21.15.77:0/2343103579 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d540703f0 0x7f0d540fea90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f0d4400b5f0 tx=0x7f0d44006ee0).ready entity=mon.0 client_cookie=aa5a8af4934a041b server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:49.768 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:49.702+0000 7f0d52ffd700 1 -- 172.21.15.77:0/2343103579 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d4400d600 con 0x7f0d540703f0 2024-02-10T08:58:49.769 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:49.702+0000 7f0d52ffd700 1 -- 172.21.15.77:0/2343103579 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0d4400dc30 con 0x7f0d540703f0 2024-02-10T08:58:49.769 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:49.702+0000 7f0d52ffd700 1 -- 172.21.15.77:0/2343103579 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d44011d60 con 0x7f0d540703f0 2024-02-10T08:58:49.769 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:49.703+0000 7f0d5a32a700 1 -- 172.21.15.77:0/2343103579 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d540703f0 msgr2=0x7f0d540fea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:49.769 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:49.703+0000 7f0d5a32a700 1 --2- 172.21.15.77:0/2343103579 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d540703f0 0x7f0d540fea90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f0d4400b5f0 tx=0x7f0d44006ee0).stop 2024-02-10T08:58:49.769 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:49.703+0000 7f0d5a32a700 1 -- 172.21.15.77:0/2343103579 shutdown_connections 2024-02-10T08:58:49.769 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:49.703+0000 7f0d5a32a700 1 --2- 172.21.15.77:0/2343103579 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d540703f0 0x7f0d540fea90 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:49.769 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:49.703+0000 7f0d5a32a700 1 -- 172.21.15.77:0/2343103579 >> 172.21.15.77:0/2343103579 conn(0x7f0d540f6530 msgr2=0x7f0d540f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:58:49.769 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:49.703+0000 7f0d5a32a700 1 -- 172.21.15.77:0/2343103579 shutdown_connections 2024-02-10T08:58:49.769 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:49.703+0000 7f0d5a32a700 1 -- 172.21.15.77:0/2343103579 wait complete. 2024-02-10T08:58:49.769 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:49.703+0000 7f0d5a32a700 1 Processor -- start 2024-02-10T08:58:49.769 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:49.704+0000 7f0d5a32a700 1 -- start start 2024-02-10T08:58:49.769 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:49.704+0000 7f0d5a32a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d540703f0 0x7f0d5410ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:49.769 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:49.704+0000 7f0d5a32a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d5410b060 con 0x7f0d540703f0 2024-02-10T08:58:49.769 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:49.704+0000 7f0d53fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d540703f0 0x7f0d5410ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:49.770 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:49.704+0000 7f0d53fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d540703f0 0x7f0d5410ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55166/0 (socket says 172.21.15.77:55166) 2024-02-10T08:58:49.770 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:49.704+0000 7f0d53fff700 1 -- 172.21.15.77:0/1479319732 learned_addr learned my addr 172.21.15.77:0/1479319732 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-10T08:58:49.770 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:49.704+0000 7f0d53fff700 1 -- 172.21.15.77:0/1479319732 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d4400b200 con 0x7f0d540703f0 2024-02-10T08:58:49.770 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:49.704+0000 7f0d53fff700 1 --2- 172.21.15.77:0/1479319732 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d540703f0 0x7f0d5410ab70 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f0d44006b80 tx=0x7f0d44006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:49.770 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:49.704+0000 7f0d50ff9700 1 -- 172.21.15.77:0/1479319732 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d44011e10 con 0x7f0d540703f0 2024-02-10T08:58:49.770 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:49.704+0000 7f0d5a32a700 1 -- 172.21.15.77:0/1479319732 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d5410b260 con 0x7f0d540703f0 2024-02-10T08:58:49.770 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:49.704+0000 7f0d50ff9700 1 -- 172.21.15.77:0/1479319732 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0d44011610 con 0x7f0d540703f0 2024-02-10T08:58:49.770 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:49.704+0000 7f0d50ff9700 1 -- 172.21.15.77:0/1479319732 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d4401a7b0 con 0x7f0d540703f0 2024-02-10T08:58:49.770 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:49.704+0000 7f0d5a32a700 1 -- 172.21.15.77:0/1479319732 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d5410b6f0 con 0x7f0d540703f0 2024-02-10T08:58:49.770 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:49.705+0000 7f0d50ff9700 1 -- 172.21.15.77:0/1479319732 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f0d4400f070 con 0x7f0d540703f0 2024-02-10T08:58:49.770 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:49.705+0000 7f0d50ff9700 1 --2- 172.21.15.77:0/1479319732 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f0d4002ff70 0x7f0d40032430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:49.770 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:49.705+0000 7f0d5a32a700 1 -- 172.21.15.77:0/1479319732 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d54045ad0 con 0x7f0d540703f0 2024-02-10T08:58:49.770 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:49.705+0000 7f0d50ff9700 1 -- 172.21.15.77:0/1479319732 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0d44041010 con 0x7f0d540703f0 2024-02-10T08:58:49.770 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:49.705+0000 7f0d537fe700 1 --2- 172.21.15.77:0/1479319732 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f0d4002ff70 0x7f0d40032430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:49.771 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:49.706+0000 7f0d537fe700 1 --2- 172.21.15.77:0/1479319732 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f0d4002ff70 0x7f0d40032430 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f0d480096a0 tx=0x7f0d48006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:49.771 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:49.710+0000 7f0d50ff9700 1 -- 172.21.15.77:0/1479319732 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0d4406d2b0 con 0x7f0d540703f0 2024-02-10T08:58:49.869 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:49.869+0000 7f0d5a32a700 1 -- 172.21.15.77:0/1479319732 --> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f0d5410cdf0 con 0x7f0d4002ff70 2024-02-10T08:58:50.024 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:50.024+0000 7f0d50ff9700 1 -- 172.21.15.77:0/1479319732 <== mgr.14162 v2:172.21.15.77:6800/81674875 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f0d5410cdf0 con 0x7f0d4002ff70 2024-02-10T08:58:50.024 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: Self-signed certificate created 2024-02-10T08:58:50.027 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:50.026+0000 7f0d5a32a700 1 -- 172.21.15.77:0/1479319732 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f0d4002ff70 msgr2=0x7f0d40032430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:50.027 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:50.026+0000 7f0d5a32a700 1 --2- 172.21.15.77:0/1479319732 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f0d4002ff70 0x7f0d40032430 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f0d480096a0 tx=0x7f0d48006b40).stop 2024-02-10T08:58:50.027 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:50.026+0000 7f0d5a32a700 1 -- 172.21.15.77:0/1479319732 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d540703f0 msgr2=0x7f0d5410ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:50.027 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:50.026+0000 7f0d5a32a700 1 --2- 172.21.15.77:0/1479319732 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d540703f0 0x7f0d5410ab70 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f0d44006b80 tx=0x7f0d44006ee0).stop 2024-02-10T08:58:50.027 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:50.026+0000 7f0d5a32a700 1 -- 172.21.15.77:0/1479319732 shutdown_connections 2024-02-10T08:58:50.027 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:50.026+0000 7f0d5a32a700 1 --2- 172.21.15.77:0/1479319732 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f0d4002ff70 0x7f0d40032430 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:50.027 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:50.026+0000 7f0d5a32a700 1 --2- 172.21.15.77:0/1479319732 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d540703f0 0x7f0d5410ab70 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:50.027 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:50.026+0000 7f0d5a32a700 1 -- 172.21.15.77:0/1479319732 >> 172.21.15.77:0/1479319732 conn(0x7f0d540f6530 msgr2=0x7f0d540f7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:58:50.027 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:50.026+0000 7f0d5a32a700 1 -- 172.21.15.77:0/1479319732 shutdown_connections 2024-02-10T08:58:50.027 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:50.026+0000 7f0d5a32a700 1 -- 172.21.15.77:0/1479319732 wait complete. 2024-02-10T08:58:50.648 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:50 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T08:58:49.478635+0000 mon.smithi077 (mon.0) 107 : cluster [DBG] mgrmap e16: smithi077.dxntvw(active, since 2s) 2024-02-10T08:58:50.649 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:50 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:49.869887+0000 mgr.smithi077.dxntvw (mgr.14162) 3 : audit [DBG] from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:58:50.649 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:50 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:50.022607+0000 mon.smithi077 (mon.0) 108 : audit [INF] from='mgr.14162 172.21.15.77:0/2029106299' entity='mgr.smithi077.dxntvw' 2024-02-10T08:58:50.649 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:50 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:50.024396+0000 mon.smithi077 (mon.0) 109 : audit [INF] from='mgr.14162 172.21.15.77:0/2029106299' entity='mgr.smithi077.dxntvw' 2024-02-10T08:58:50.721 INFO:teuthology.orchestra.run.smithi077.stderr:Creating initial admin user... 2024-02-10T08:58:50.721 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpzzyygg7o:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpx5m65_qh:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp_8rxnps1:/tmp/dashboard.pw:z docker.io/ceph/ceph:v16.2.4 dashboard ac-user-create admin -i /tmp/dashboard.pw administrator --force-password --pwd-update-required 2024-02-10T08:58:51.846 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:51.842+0000 7f07a643b700 1 Processor -- start 2024-02-10T08:58:51.846 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:51.842+0000 7f07a643b700 1 -- start start 2024-02-10T08:58:51.846 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:51.842+0000 7f07a643b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07a00703b0 0x7f07a0100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:51.846 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:51.842+0000 7f07a643b700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a00fc8c0 con 0x7f07a00703b0 2024-02-10T08:58:51.846 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:51.842+0000 7f079ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07a00703b0 0x7f07a0100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:51.846 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:51.842+0000 7f079ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07a00703b0 0x7f07a0100ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55172/0 (socket says 172.21.15.77:55172) 2024-02-10T08:58:51.846 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:51.842+0000 7f079ffff700 1 -- 172.21.15.77:0/540099305 learned_addr learned my addr 172.21.15.77:0/540099305 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-10T08:58:51.846 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:51.843+0000 7f079ffff700 1 -- 172.21.15.77:0/540099305 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07a01013c0 con 0x7f07a00703b0 2024-02-10T08:58:51.846 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:51.843+0000 7f079ffff700 1 --2- 172.21.15.77:0/540099305 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07a00703b0 0x7f07a0100ed0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f079400b5f0 tx=0x7f0794006ee0).ready entity=mon.0 client_cookie=f22cbea3e680d1f4 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:51.847 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:51.843+0000 7f079effd700 1 -- 172.21.15.77:0/540099305 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f079400d600 con 0x7f07a00703b0 2024-02-10T08:58:51.847 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:51.843+0000 7f079effd700 1 -- 172.21.15.77:0/540099305 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f079400dc30 con 0x7f07a00703b0 2024-02-10T08:58:51.847 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:51.843+0000 7f079effd700 1 -- 172.21.15.77:0/540099305 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0794011d30 con 0x7f07a00703b0 2024-02-10T08:58:51.847 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:51.843+0000 7f07a643b700 1 -- 172.21.15.77:0/540099305 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07a00703b0 msgr2=0x7f07a0100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:51.847 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:51.843+0000 7f07a643b700 1 --2- 172.21.15.77:0/540099305 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07a00703b0 0x7f07a0100ed0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f079400b5f0 tx=0x7f0794006ee0).stop 2024-02-10T08:58:51.847 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:51.843+0000 7f07a643b700 1 -- 172.21.15.77:0/540099305 shutdown_connections 2024-02-10T08:58:51.847 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:51.843+0000 7f07a643b700 1 --2- 172.21.15.77:0/540099305 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07a00703b0 0x7f07a0100ed0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:51.847 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:51.843+0000 7f07a643b700 1 -- 172.21.15.77:0/540099305 >> 172.21.15.77:0/540099305 conn(0x7f07a00f6530 msgr2=0x7f07a00f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:58:51.847 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:51.843+0000 7f07a643b700 1 -- 172.21.15.77:0/540099305 shutdown_connections 2024-02-10T08:58:51.847 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:51.844+0000 7f07a643b700 1 -- 172.21.15.77:0/540099305 wait complete. 2024-02-10T08:58:51.847 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:51.844+0000 7f07a643b700 1 Processor -- start 2024-02-10T08:58:51.847 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:51.844+0000 7f07a643b700 1 -- start start 2024-02-10T08:58:51.847 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:51.844+0000 7f07a643b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07a00703b0 0x7f07a010ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:51.847 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:51.845+0000 7f07a643b700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a010d2f0 con 0x7f07a00703b0 2024-02-10T08:58:51.848 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:51.845+0000 7f079ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07a00703b0 0x7f07a010ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:51.848 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:51.845+0000 7f079ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07a00703b0 0x7f07a010ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55186/0 (socket says 172.21.15.77:55186) 2024-02-10T08:58:51.848 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:51.845+0000 7f079ffff700 1 -- 172.21.15.77:0/3947518601 learned_addr learned my addr 172.21.15.77:0/3947518601 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-10T08:58:51.848 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:51.845+0000 7f079ffff700 1 -- 172.21.15.77:0/3947518601 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f079400b200 con 0x7f07a00703b0 2024-02-10T08:58:51.848 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:51.845+0000 7f079ffff700 1 --2- 172.21.15.77:0/3947518601 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07a00703b0 0x7f07a010ce00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f079400e040 tx=0x7f0794006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:51.848 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:51.845+0000 7f079cff9700 1 -- 172.21.15.77:0/3947518601 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f079400da60 con 0x7f07a00703b0 2024-02-10T08:58:51.848 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:51.845+0000 7f079cff9700 1 -- 172.21.15.77:0/3947518601 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f079401b8e0 con 0x7f07a00703b0 2024-02-10T08:58:51.848 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:51.845+0000 7f07a643b700 1 -- 172.21.15.77:0/3947518601 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07a010d4f0 con 0x7f07a00703b0 2024-02-10T08:58:51.848 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:51.845+0000 7f079cff9700 1 -- 172.21.15.77:0/3947518601 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f079401a900 con 0x7f07a00703b0 2024-02-10T08:58:51.848 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:51.845+0000 7f07a643b700 1 -- 172.21.15.77:0/3947518601 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07a010e100 con 0x7f07a00703b0 2024-02-10T08:58:51.848 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:51.846+0000 7f079cff9700 1 -- 172.21.15.77:0/3947518601 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f079400f070 con 0x7f07a00703b0 2024-02-10T08:58:51.849 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:51.846+0000 7f079cff9700 1 --2- 172.21.15.77:0/3947518601 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f078c02ff70 0x7f078c032430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:51.849 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:51.846+0000 7f079cff9700 1 -- 172.21.15.77:0/3947518601 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f07940414e0 con 0x7f07a00703b0 2024-02-10T08:58:51.849 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:51.846+0000 7f079f7fe700 1 --2- 172.21.15.77:0/3947518601 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f078c02ff70 0x7f078c032430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:51.849 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:51.847+0000 7f079f7fe700 1 --2- 172.21.15.77:0/3947518601 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f078c02ff70 0x7f078c032430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f07900096a0 tx=0x7f0790006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:51.849 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:51.847+0000 7f07a643b700 1 -- 172.21.15.77:0/3947518601 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07a0059100 con 0x7f07a00703b0 2024-02-10T08:58:51.851 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:51.851+0000 7f079cff9700 1 -- 172.21.15.77:0/3947518601 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0794016070 con 0x7f07a00703b0 2024-02-10T08:58:52.016 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:52.016+0000 7f07a643b700 1 -- 172.21.15.77:0/3947518601 --> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f07a0058f20 con 0x7f078c02ff70 2024-02-10T08:58:52.241 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:52.241+0000 7f079cff9700 1 -- 172.21.15.77:0/3947518601 <== mgr.14162 v2:172.21.15.77:6800/81674875 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f07a0058f20 con 0x7f078c02ff70 2024-02-10T08:58:52.241 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$yPteSrCfIMo/5oA7upL10OgEYxRogPTPMTXMsxQo4ZpH8y5USC9ii", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1707555532, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-02-10T08:58:52.243 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:52.242+0000 7f07a643b700 1 -- 172.21.15.77:0/3947518601 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f078c02ff70 msgr2=0x7f078c032430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:52.243 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:52.242+0000 7f07a643b700 1 --2- 172.21.15.77:0/3947518601 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f078c02ff70 0x7f078c032430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f07900096a0 tx=0x7f0790006b40).stop 2024-02-10T08:58:52.243 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:52.242+0000 7f07a643b700 1 -- 172.21.15.77:0/3947518601 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07a00703b0 msgr2=0x7f07a010ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:52.243 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:52.242+0000 7f07a643b700 1 --2- 172.21.15.77:0/3947518601 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07a00703b0 0x7f07a010ce00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f079400e040 tx=0x7f0794006ee0).stop 2024-02-10T08:58:52.244 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:52.242+0000 7f07a643b700 1 -- 172.21.15.77:0/3947518601 shutdown_connections 2024-02-10T08:58:52.244 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:52.242+0000 7f07a643b700 1 --2- 172.21.15.77:0/3947518601 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f078c02ff70 0x7f078c032430 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:52.244 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:52.242+0000 7f07a643b700 1 --2- 172.21.15.77:0/3947518601 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07a00703b0 0x7f07a010ce00 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:52.244 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:52.242+0000 7f07a643b700 1 -- 172.21.15.77:0/3947518601 >> 172.21.15.77:0/3947518601 conn(0x7f07a00f6530 msgr2=0x7f07a00f7f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:58:52.244 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:52.243+0000 7f07a643b700 1 -- 172.21.15.77:0/3947518601 shutdown_connections 2024-02-10T08:58:52.244 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:52.243+0000 7f07a643b700 1 -- 172.21.15.77:0/3947518601 wait complete. 2024-02-10T08:58:52.956 INFO:teuthology.orchestra.run.smithi077.stderr:Fetching dashboard port number... 2024-02-10T08:58:52.957 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpzzyygg7o:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpx5m65_qh:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config get mgr mgr/dashboard/ssl_server_port 2024-02-10T08:58:53.636 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:53.630+0000 7f8efffff700 1 Processor -- start 2024-02-10T08:58:53.637 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:53.630+0000 7f8efffff700 1 -- start start 2024-02-10T08:58:53.637 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:53.630+0000 7f8efffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f00070430 0x7f8f000fe890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:53.637 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:53.630+0000 7f8efffff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f000fed80 con 0x7f8f00070430 2024-02-10T08:58:53.637 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:53.630+0000 7f8efeffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f00070430 0x7f8f000fe890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:53.637 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:53.630+0000 7f8efeffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f00070430 0x7f8f000fe890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55192/0 (socket says 172.21.15.77:55192) 2024-02-10T08:58:53.637 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:53.630+0000 7f8efeffd700 1 -- 172.21.15.77:0/1856964450 learned_addr learned my addr 172.21.15.77:0/1856964450 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-10T08:58:53.637 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:53.630+0000 7f8efeffd700 1 -- 172.21.15.77:0/1856964450 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f000feec0 con 0x7f8f00070430 2024-02-10T08:58:53.637 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:53.630+0000 7f8efeffd700 1 --2- 172.21.15.77:0/1856964450 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f00070430 0x7f8f000fe890 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f8ef000b5f0 tx=0x7f8ef0006ee0).ready entity=mon.0 client_cookie=121d365baf42dd2e server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:53.637 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:53.631+0000 7f8efdffb700 1 -- 172.21.15.77:0/1856964450 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ef000d600 con 0x7f8f00070430 2024-02-10T08:58:53.637 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:53.631+0000 7f8efdffb700 1 -- 172.21.15.77:0/1856964450 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8ef000dc30 con 0x7f8f00070430 2024-02-10T08:58:53.637 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:53.631+0000 7f8efdffb700 1 -- 172.21.15.77:0/1856964450 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ef0011d30 con 0x7f8f00070430 2024-02-10T08:58:53.637 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:53.631+0000 7f8efffff700 1 -- 172.21.15.77:0/1856964450 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f00070430 msgr2=0x7f8f000fe890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:53.637 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:53.631+0000 7f8efffff700 1 --2- 172.21.15.77:0/1856964450 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f00070430 0x7f8f000fe890 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f8ef000b5f0 tx=0x7f8ef0006ee0).stop 2024-02-10T08:58:53.637 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:53.631+0000 7f8efffff700 1 -- 172.21.15.77:0/1856964450 shutdown_connections 2024-02-10T08:58:53.638 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:53.631+0000 7f8efffff700 1 --2- 172.21.15.77:0/1856964450 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f00070430 0x7f8f000fe890 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:53.638 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:53.631+0000 7f8efffff700 1 -- 172.21.15.77:0/1856964450 >> 172.21.15.77:0/1856964450 conn(0x7f8f000f6390 msgr2=0x7f8f000f87b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:58:53.638 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:53.631+0000 7f8efffff700 1 -- 172.21.15.77:0/1856964450 shutdown_connections 2024-02-10T08:58:53.638 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:53.631+0000 7f8efffff700 1 -- 172.21.15.77:0/1856964450 wait complete. 2024-02-10T08:58:53.638 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:53.632+0000 7f8efffff700 1 Processor -- start 2024-02-10T08:58:53.638 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:53.632+0000 7f8efffff700 1 -- start start 2024-02-10T08:58:53.638 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:53.632+0000 7f8efffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f00070430 0x7f8f0010cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:53.638 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:53.632+0000 7f8efffff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f0010d0d0 con 0x7f8f00070430 2024-02-10T08:58:53.638 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:53.632+0000 7f8efeffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f00070430 0x7f8f0010cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:53.638 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:53.632+0000 7f8efeffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f00070430 0x7f8f0010cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55204/0 (socket says 172.21.15.77:55204) 2024-02-10T08:58:53.638 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:53.632+0000 7f8efeffd700 1 -- 172.21.15.77:0/76798377 learned_addr learned my addr 172.21.15.77:0/76798377 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-10T08:58:53.638 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:53.632+0000 7f8efeffd700 1 -- 172.21.15.77:0/76798377 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ef000b200 con 0x7f8f00070430 2024-02-10T08:58:53.638 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:53.632+0000 7f8efeffd700 1 --2- 172.21.15.77:0/76798377 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f00070430 0x7f8f0010cbe0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f8ef000e040 tx=0x7f8ef0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:53.638 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:53.633+0000 7f8ee77fe700 1 -- 172.21.15.77:0/76798377 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ef000da60 con 0x7f8f00070430 2024-02-10T08:58:53.638 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:53.633+0000 7f8efffff700 1 -- 172.21.15.77:0/76798377 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f0010d2d0 con 0x7f8f00070430 2024-02-10T08:58:53.639 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:53.633+0000 7f8ee77fe700 1 -- 172.21.15.77:0/76798377 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8ef001b8e0 con 0x7f8f00070430 2024-02-10T08:58:53.639 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:53.633+0000 7f8ee77fe700 1 -- 172.21.15.77:0/76798377 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ef001a900 con 0x7f8f00070430 2024-02-10T08:58:53.639 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:53.633+0000 7f8efffff700 1 -- 172.21.15.77:0/76798377 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f0010d760 con 0x7f8f00070430 2024-02-10T08:58:53.639 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:53.634+0000 7f8ee77fe700 1 -- 172.21.15.77:0/76798377 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f8ef000f070 con 0x7f8f00070430 2024-02-10T08:58:53.639 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:53.634+0000 7f8efffff700 1 -- 172.21.15.77:0/76798377 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f00045ad0 con 0x7f8f00070430 2024-02-10T08:58:53.639 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:53.634+0000 7f8ee77fe700 1 --2- 172.21.15.77:0/76798377 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f8eec02ff70 0x7f8eec032430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:53.639 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:53.634+0000 7f8ee77fe700 1 -- 172.21.15.77:0/76798377 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8ef00414f0 con 0x7f8f00070430 2024-02-10T08:58:53.639 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:53.634+0000 7f8efe7fc700 1 --2- 172.21.15.77:0/76798377 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f8eec02ff70 0x7f8eec032430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:53.639 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:53.634+0000 7f8efe7fc700 1 --2- 172.21.15.77:0/76798377 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f8eec02ff70 0x7f8eec032430 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f8ef40096a0 tx=0x7f8ef4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:53.640 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:53.638+0000 7f8ee77fe700 1 -- 172.21.15.77:0/76798377 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8ef0025980 con 0x7f8f00070430 2024-02-10T08:58:53.648 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:53 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:52.017454+0000 mgr.smithi077.dxntvw (mgr.14162) 4 : audit [DBG] from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:58:53.648 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:53 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:52.241212+0000 mon.smithi077 (mon.0) 110 : audit [INF] from='mgr.14162 172.21.15.77:0/2029106299' entity='mgr.smithi077.dxntvw' 2024-02-10T08:58:53.678 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:53.678+0000 7f8ee77fe700 1 -- 172.21.15.77:0/76798377 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8ef0025460 con 0x7f8f00070430 2024-02-10T08:58:53.792 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:53.791+0000 7f8efffff700 1 -- 172.21.15.77:0/76798377 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f8f000ffc70 con 0x7f8f00070430 2024-02-10T08:58:53.792 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:53.792+0000 7f8ee77fe700 1 -- 172.21.15.77:0/76798377 <== mon.0 v2:172.21.15.77:3300/0 8 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v10) v1 ==== 112+0+5 (secure 0 0 0) 0x7f8ef00160b0 con 0x7f8f00070430 2024-02-10T08:58:53.792 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 8443 2024-02-10T08:58:53.794 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:53.793+0000 7f8efffff700 1 -- 172.21.15.77:0/76798377 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f8eec02ff70 msgr2=0x7f8eec032430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:53.794 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:53.793+0000 7f8efffff700 1 --2- 172.21.15.77:0/76798377 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f8eec02ff70 0x7f8eec032430 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f8ef40096a0 tx=0x7f8ef4006b40).stop 2024-02-10T08:58:53.794 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:53.793+0000 7f8efffff700 1 -- 172.21.15.77:0/76798377 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f00070430 msgr2=0x7f8f0010cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:53.794 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:53.793+0000 7f8efffff700 1 --2- 172.21.15.77:0/76798377 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f00070430 0x7f8f0010cbe0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f8ef000e040 tx=0x7f8ef0006ee0).stop 2024-02-10T08:58:53.794 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:53.793+0000 7f8efffff700 1 -- 172.21.15.77:0/76798377 shutdown_connections 2024-02-10T08:58:53.794 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:53.793+0000 7f8efffff700 1 --2- 172.21.15.77:0/76798377 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f8eec02ff70 0x7f8eec032430 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:53.794 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:53.793+0000 7f8efffff700 1 --2- 172.21.15.77:0/76798377 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8f00070430 0x7f8f0010cbe0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:53.795 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:53.793+0000 7f8efffff700 1 -- 172.21.15.77:0/76798377 >> 172.21.15.77:0/76798377 conn(0x7f8f000f6390 msgr2=0x7f8f000f7d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:58:53.795 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:53.794+0000 7f8efffff700 1 -- 172.21.15.77:0/76798377 shutdown_connections 2024-02-10T08:58:53.795 INFO:teuthology.orchestra.run.smithi077.stderr:/usr/bin/ceph: 2024-02-10T08:58:53.794+0000 7f8efffff700 1 -- 172.21.15.77:0/76798377 wait complete. 2024-02-10T08:58:54.488 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: systemctl is-enabled firewalld.service 2024-02-10T08:58:54.496 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: disabled 2024-02-10T08:58:54.497 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: systemctl is-active firewalld.service 2024-02-10T08:58:54.504 INFO:teuthology.orchestra.run.smithi077.stderr:systemctl: inactive 2024-02-10T08:58:54.505 INFO:teuthology.orchestra.run.smithi077.stderr:firewalld.service is not enabled 2024-02-10T08:58:54.505 INFO:teuthology.orchestra.run.smithi077.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-02-10T08:58:54.508 INFO:teuthology.orchestra.run.smithi077.stderr:Ceph Dashboard is now available at: 2024-02-10T08:58:54.508 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-02-10T08:58:54.508 INFO:teuthology.orchestra.run.smithi077.stderr: URL: https://smithi077.front.sepia.ceph.com:8443/ 2024-02-10T08:58:54.508 INFO:teuthology.orchestra.run.smithi077.stderr: User: admin 2024-02-10T08:58:54.509 INFO:teuthology.orchestra.run.smithi077.stderr: Password: 97w8e0dg3p 2024-02-10T08:58:54.509 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-02-10T08:58:54.509 INFO:teuthology.orchestra.run.smithi077.stderr:You can access the Ceph CLI with: 2024-02-10T08:58:54.509 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-02-10T08:58:54.509 INFO:teuthology.orchestra.run.smithi077.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-02-10T08:58:54.509 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-02-10T08:58:54.509 INFO:teuthology.orchestra.run.smithi077.stderr:Please consider enabling telemetry to help improve Ceph: 2024-02-10T08:58:54.509 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-02-10T08:58:54.509 INFO:teuthology.orchestra.run.smithi077.stderr: ceph telemetry on 2024-02-10T08:58:54.509 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-02-10T08:58:54.509 INFO:teuthology.orchestra.run.smithi077.stderr:For more information see: 2024-02-10T08:58:54.509 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-02-10T08:58:54.509 INFO:teuthology.orchestra.run.smithi077.stderr: https://docs.ceph.com/docs/pacific/mgr/telemetry/ 2024-02-10T08:58:54.509 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-02-10T08:58:54.510 INFO:teuthology.orchestra.run.smithi077.stderr:Bootstrap complete. 2024-02-10T08:58:54.510 INFO:teuthology.orchestra.run.smithi077.stderr:Releasing lock 139803068135912 on /run/cephadm/6081b54a-c7f2-11ee-95b8-87774f69a715.lock 2024-02-10T08:58:54.510 INFO:teuthology.orchestra.run.smithi077.stderr:Lock 139803068135912 released on /run/cephadm/6081b54a-c7f2-11ee-95b8-87774f69a715.lock 2024-02-10T08:58:54.544 INFO:tasks.cephadm:Fetching config... 2024-02-10T08:58:54.545 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-02-10T08:58:54.545 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-02-10T08:58:54.603 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-02-10T08:58:54.603 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-02-10T08:58:54.604 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-02-10T08:58:54.661 INFO:tasks.cephadm:Fetching mon keyring... 2024-02-10T08:58:54.661 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-02-10T08:58:54.661 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/mon.smithi077/keyring of=/dev/stdout 2024-02-10T08:58:54.728 INFO:tasks.cephadm:Fetching pub ssh key... 2024-02-10T08:58:54.729 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-02-10T08:58:54.729 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-02-10T08:58:54.786 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-02-10T08:58:54.786 DEBUG:teuthology.orchestra.run.smithi077:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDV3XILkUclmDRAU4yCQDRVwgwrV6pZ3hfbzzMuXfddDui8gFX8bu+wpWRC74vg9ZkO0KKWbpD3qWql9QgtyhCZTOJnmQyF/69EwbX5I7xLEs1UW2Rh2P639kMfhCJWdIMYieAqCSKZl+GbfdIh5ZORxYDnDQxSWI6i0VvnhVnlURFdHCSAAjlkRxF2qf6EhfzJzrr9cjZv6nDabYoYButG5G7rhMxQDddFqoQANwB6w7xHk3Ec+1eyC8L/uAEwhNXObj94W4js38qQj/pFpasf9ctVl4PyFJ0f9CqMgd2MssEqGlbAvga+bFVDfE3Jzn8MH1kiOasSQiaZ6iPU32Xn7MbksFD/n+a6CDQawAMhaKjVaDInGvNTE22imoZFq+RUfKJKDlBEXM/0g8ddRNwxNjA32WTIovpa7DnW7b5mZUoZ4r+kJHTSpVu7f5GcHPkkeqljIUlUwRLd5wCtEgwOE7NBDfSi1uD0BVFCryays2EJ4g8NlhAbS3dyQFHSTXk= ceph-6081b54a-c7f2-11ee-95b8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-10T08:58:54.832 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:54 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T08:58:53.680521+0000 mon.smithi077 (mon.0) 111 : cluster [DBG] mgrmap e17: smithi077.dxntvw(active, since 6s) 2024-02-10T08:58:54.832 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:54 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:53.792175+0000 mon.smithi077 (mon.0) 112 : audit [DBG] from='client.? 172.21.15.77:0/76798377' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-02-10T08:58:54.865 INFO:teuthology.orchestra.run.smithi077.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDV3XILkUclmDRAU4yCQDRVwgwrV6pZ3hfbzzMuXfddDui8gFX8bu+wpWRC74vg9ZkO0KKWbpD3qWql9QgtyhCZTOJnmQyF/69EwbX5I7xLEs1UW2Rh2P639kMfhCJWdIMYieAqCSKZl+GbfdIh5ZORxYDnDQxSWI6i0VvnhVnlURFdHCSAAjlkRxF2qf6EhfzJzrr9cjZv6nDabYoYButG5G7rhMxQDddFqoQANwB6w7xHk3Ec+1eyC8L/uAEwhNXObj94W4js38qQj/pFpasf9ctVl4PyFJ0f9CqMgd2MssEqGlbAvga+bFVDfE3Jzn8MH1kiOasSQiaZ6iPU32Xn7MbksFD/n+a6CDQawAMhaKjVaDInGvNTE22imoZFq+RUfKJKDlBEXM/0g8ddRNwxNjA32WTIovpa7DnW7b5mZUoZ4r+kJHTSpVu7f5GcHPkkeqljIUlUwRLd5wCtEgwOE7NBDfSi1uD0BVFCryays2EJ4g8NlhAbS3dyQFHSTXk= ceph-6081b54a-c7f2-11ee-95b8-87774f69a715 2024-02-10T08:58:54.876 DEBUG:teuthology.orchestra.run.smithi155:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDV3XILkUclmDRAU4yCQDRVwgwrV6pZ3hfbzzMuXfddDui8gFX8bu+wpWRC74vg9ZkO0KKWbpD3qWql9QgtyhCZTOJnmQyF/69EwbX5I7xLEs1UW2Rh2P639kMfhCJWdIMYieAqCSKZl+GbfdIh5ZORxYDnDQxSWI6i0VvnhVnlURFdHCSAAjlkRxF2qf6EhfzJzrr9cjZv6nDabYoYButG5G7rhMxQDddFqoQANwB6w7xHk3Ec+1eyC8L/uAEwhNXObj94W4js38qQj/pFpasf9ctVl4PyFJ0f9CqMgd2MssEqGlbAvga+bFVDfE3Jzn8MH1kiOasSQiaZ6iPU32Xn7MbksFD/n+a6CDQawAMhaKjVaDInGvNTE22imoZFq+RUfKJKDlBEXM/0g8ddRNwxNjA32WTIovpa7DnW7b5mZUoZ4r+kJHTSpVu7f5GcHPkkeqljIUlUwRLd5wCtEgwOE7NBDfSi1uD0BVFCryays2EJ4g8NlhAbS3dyQFHSTXk= ceph-6081b54a-c7f2-11ee-95b8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-10T08:58:54.920 INFO:teuthology.orchestra.run.smithi155.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDV3XILkUclmDRAU4yCQDRVwgwrV6pZ3hfbzzMuXfddDui8gFX8bu+wpWRC74vg9ZkO0KKWbpD3qWql9QgtyhCZTOJnmQyF/69EwbX5I7xLEs1UW2Rh2P639kMfhCJWdIMYieAqCSKZl+GbfdIh5ZORxYDnDQxSWI6i0VvnhVnlURFdHCSAAjlkRxF2qf6EhfzJzrr9cjZv6nDabYoYButG5G7rhMxQDddFqoQANwB6w7xHk3Ec+1eyC8L/uAEwhNXObj94W4js38qQj/pFpasf9ctVl4PyFJ0f9CqMgd2MssEqGlbAvga+bFVDfE3Jzn8MH1kiOasSQiaZ6iPU32Xn7MbksFD/n+a6CDQawAMhaKjVaDInGvNTE22imoZFq+RUfKJKDlBEXM/0g8ddRNwxNjA32WTIovpa7DnW7b5mZUoZ4r+kJHTSpVu7f5GcHPkkeqljIUlUwRLd5wCtEgwOE7NBDfSi1uD0BVFCryays2EJ4g8NlhAbS3dyQFHSTXk= ceph-6081b54a-c7f2-11ee-95b8-87774f69a715 2024-02-10T08:58:54.932 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-02-10T08:58:57.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:58:57.863+0000 7f343d2dd700 1 -- 172.21.15.77:0/3160241561 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34380f51a0 msgr2=0x7f34380f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:57.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:58:57.863+0000 7f343d2dd700 1 --2- 172.21.15.77:0/3160241561 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34380f51a0 0x7f34380f55c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f3428009fb0 tx=0x7f3428009b30).stop 2024-02-10T08:58:57.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:58:57.863+0000 7f343d2dd700 1 -- 172.21.15.77:0/3160241561 shutdown_connections 2024-02-10T08:58:57.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:58:57.863+0000 7f343d2dd700 1 --2- 172.21.15.77:0/3160241561 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34380f51a0 0x7f34380f55c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:57.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:58:57.863+0000 7f343d2dd700 1 -- 172.21.15.77:0/3160241561 >> 172.21.15.77:0/3160241561 conn(0x7f34380f09a0 msgr2=0x7f34380f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:58:57.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:58:57.863+0000 7f343d2dd700 1 -- 172.21.15.77:0/3160241561 shutdown_connections 2024-02-10T08:58:57.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:58:57.863+0000 7f343d2dd700 1 -- 172.21.15.77:0/3160241561 wait complete. 2024-02-10T08:58:57.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:58:57.864+0000 7f343d2dd700 1 Processor -- start 2024-02-10T08:58:57.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:58:57.864+0000 7f343d2dd700 1 -- start start 2024-02-10T08:58:57.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:58:57.864+0000 7f343d2dd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34380f51a0 0x7f34380fd270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:57.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:58:57.864+0000 7f343d2dd700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34380fb6f0 con 0x7f34380f51a0 2024-02-10T08:58:57.885 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:58:57.864+0000 7f3437fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34380f51a0 0x7f34380fd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:57.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:58:57.864+0000 7f3437fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34380f51a0 0x7f34380fd270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36638/0 (socket says 172.21.15.77:36638) 2024-02-10T08:58:57.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:58:57.864+0000 7f3437fff700 1 -- 172.21.15.77:0/4012996631 learned_addr learned my addr 172.21.15.77:0/4012996631 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-10T08:58:57.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:58:57.864+0000 7f3437fff700 1 -- 172.21.15.77:0/4012996631 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3428005040 con 0x7f34380f51a0 2024-02-10T08:58:57.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:58:57.864+0000 7f3437fff700 1 --2- 172.21.15.77:0/4012996631 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34380f51a0 0x7f34380fd270 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f34280089f0 tx=0x7f342800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:57.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:58:57.865+0000 7f3434ff9700 1 -- 172.21.15.77:0/4012996631 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3428009320 con 0x7f34380f51a0 2024-02-10T08:58:57.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:58:57.865+0000 7f3434ff9700 1 -- 172.21.15.77:0/4012996631 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3428013380 con 0x7f34380f51a0 2024-02-10T08:58:57.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:58:57.865+0000 7f343d2dd700 1 -- 172.21.15.77:0/4012996631 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34380fb890 con 0x7f34380f51a0 2024-02-10T08:58:57.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:58:57.865+0000 7f3434ff9700 1 -- 172.21.15.77:0/4012996631 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3428014e20 con 0x7f34380f51a0 2024-02-10T08:58:57.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:58:57.865+0000 7f343d2dd700 1 -- 172.21.15.77:0/4012996631 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34380fbd20 con 0x7f34380f51a0 2024-02-10T08:58:57.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:58:57.865+0000 7f3434ff9700 1 -- 172.21.15.77:0/4012996631 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f34280263a0 con 0x7f34380f51a0 2024-02-10T08:58:57.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:58:57.865+0000 7f3434ff9700 1 --2- 172.21.15.77:0/4012996631 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f342002ff10 0x7f34200323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:58:57.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:58:57.866+0000 7f3434ff9700 1 -- 172.21.15.77:0/4012996631 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3428041fe0 con 0x7f34380f51a0 2024-02-10T08:58:57.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:58:57.866+0000 7f34377fe700 1 --2- 172.21.15.77:0/4012996631 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f342002ff10 0x7f34200323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:58:57.886 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:58:57.866+0000 7f343d2dd700 1 -- 172.21.15.77:0/4012996631 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3438059100 con 0x7f34380f51a0 2024-02-10T08:58:57.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:58:57.866+0000 7f34377fe700 1 --2- 172.21.15.77:0/4012996631 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f342002ff10 0x7f34200323d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f342c0096a0 tx=0x7f342c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:58:57.887 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:58:57.870+0000 7f3434ff9700 1 -- 172.21.15.77:0/4012996631 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3428012920 con 0x7f34380f51a0 2024-02-10T08:58:58.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:58:58.021+0000 7f343d2dd700 1 -- 172.21.15.77:0/4012996631 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f3438045ad0 con 0x7f34380f51a0 2024-02-10T08:58:58.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:58:58.024+0000 7f3434ff9700 1 -- 172.21.15.77:0/4012996631 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v11)=0 v11) v1 ==== 125+0+0 (secure 0 0 0) 0x7f342802b020 con 0x7f34380f51a0 2024-02-10T08:58:58.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:58:58.026+0000 7f343d2dd700 1 -- 172.21.15.77:0/4012996631 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f342002ff10 msgr2=0x7f34200323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:58.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:58:58.026+0000 7f343d2dd700 1 --2- 172.21.15.77:0/4012996631 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f342002ff10 0x7f34200323d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f342c0096a0 tx=0x7f342c006b40).stop 2024-02-10T08:58:58.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:58:58.026+0000 7f343d2dd700 1 -- 172.21.15.77:0/4012996631 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34380f51a0 msgr2=0x7f34380fd270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:58:58.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:58:58.026+0000 7f343d2dd700 1 --2- 172.21.15.77:0/4012996631 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34380f51a0 0x7f34380fd270 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f34280089f0 tx=0x7f342800b040).stop 2024-02-10T08:58:58.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:58:58.026+0000 7f343d2dd700 1 -- 172.21.15.77:0/4012996631 shutdown_connections 2024-02-10T08:58:58.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:58:58.026+0000 7f343d2dd700 1 --2- 172.21.15.77:0/4012996631 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f342002ff10 0x7f34200323d0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:58.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:58:58.026+0000 7f343d2dd700 1 --2- 172.21.15.77:0/4012996631 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f34380f51a0 0x7f34380fd270 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:58:58.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:58:58.026+0000 7f343d2dd700 1 -- 172.21.15.77:0/4012996631 >> 172.21.15.77:0/4012996631 conn(0x7f34380f09a0 msgr2=0x7f343805fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:58:58.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:58:58.026+0000 7f343d2dd700 1 -- 172.21.15.77:0/4012996631 shutdown_connections 2024-02-10T08:58:58.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:58:58.026+0000 7f343d2dd700 1 -- 172.21.15.77:0/4012996631 wait complete. 2024-02-10T08:58:58.800 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi155 2024-02-10T08:58:58.800 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-02-10T08:58:58.800 DEBUG:teuthology.orchestra.run.smithi155:> dd of=/etc/ceph/ceph.conf 2024-02-10T08:58:58.821 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-02-10T08:58:58.822 DEBUG:teuthology.orchestra.run.smithi155:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-10T08:58:58.882 INFO:tasks.cephadm:Adding host smithi155 to orchestrator... 2024-02-10T08:58:58.883 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph orch host add smithi155 2024-02-10T08:58:59.148 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:58:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:58.025604+0000 mon.smithi077 (mon.0) 113 : audit [INF] from='client.? 172.21.15.77:0/4012996631' entity='client.admin' 2024-02-10T08:59:01.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:59:00 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:58:59.797590+0000 mon.smithi077 (mon.0) 114 : audit [INF] from='mgr.14162 172.21.15.77:0/2029106299' entity='mgr.smithi077.dxntvw' 2024-02-10T08:59:01.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:59:00 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:59:00.055281+0000 mon.smithi077 (mon.0) 115 : audit [INF] from='mgr.14162 172.21.15.77:0/2029106299' entity='mgr.smithi077.dxntvw' 2024-02-10T08:59:02.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:59:01 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cephadm 2024-02-10T08:59:00.061431+0000 mgr.smithi077.dxntvw (mgr.14162) 5 : cephadm [INF] Deploying daemon alertmanager.smithi077 on smithi077 2024-02-10T08:59:08.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:59:07 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T08:59:06.956305+0000 mgr.smithi077.dxntvw (mgr.14162) 6 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:59:10.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:59:09 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T08:59:08.956658+0000 mgr.smithi077.dxntvw (mgr.14162) 7 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:59:12.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:59:11 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T08:59:10.956938+0000 mgr.smithi077.dxntvw (mgr.14162) 8 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:59:14.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:59:13 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T08:59:12.957361+0000 mgr.smithi077.dxntvw (mgr.14162) 9 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:59:15.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:59:15 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T08:59:14.957740+0000 mgr.smithi077.dxntvw (mgr.14162) 10 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:59:17.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:59:17 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T08:59:16.958097+0000 mgr.smithi077.dxntvw (mgr.14162) 11 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:59:19.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:59:19 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T08:59:18.958453+0000 mgr.smithi077.dxntvw (mgr.14162) 12 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:59:20.931 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:20.917+0000 7fa8f8a89700 1 -- 172.21.15.155:0/1083844263 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8f40f5410 msgr2=0x7fa8f40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:59:20.931 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:20.917+0000 7fa8f8a89700 1 --2- 172.21.15.155:0/1083844263 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8f40f5410 0x7fa8f40f5830 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fa8e4009fb0 tx=0x7fa8e4009b30).stop 2024-02-10T08:59:20.931 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:20.917+0000 7fa8f8a89700 1 -- 172.21.15.155:0/1083844263 shutdown_connections 2024-02-10T08:59:20.931 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:20.917+0000 7fa8f8a89700 1 --2- 172.21.15.155:0/1083844263 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8f40f5410 0x7fa8f40f5830 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:59:20.931 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:20.917+0000 7fa8f8a89700 1 -- 172.21.15.155:0/1083844263 >> 172.21.15.155:0/1083844263 conn(0x7fa8f40f0b70 msgr2=0x7fa8f40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:59:20.931 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:20.917+0000 7fa8f8a89700 1 -- 172.21.15.155:0/1083844263 shutdown_connections 2024-02-10T08:59:20.931 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:20.917+0000 7fa8f8a89700 1 -- 172.21.15.155:0/1083844263 wait complete. 2024-02-10T08:59:20.932 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:20.917+0000 7fa8f8a89700 1 Processor -- start 2024-02-10T08:59:20.932 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:20.917+0000 7fa8f8a89700 1 -- start start 2024-02-10T08:59:20.932 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:20.918+0000 7fa8f8a89700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8f40f5410 0x7fa8f41094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:59:20.932 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:20.918+0000 7fa8f8a89700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f4060240 con 0x7fa8f40f5410 2024-02-10T08:59:20.932 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:20.918+0000 7fa8f259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8f40f5410 0x7fa8f41094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:59:20.932 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:20.918+0000 7fa8f259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8f40f5410 0x7fa8f41094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:44878/0 (socket says 172.21.15.155:44878) 2024-02-10T08:59:20.932 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:20.918+0000 7fa8f259c700 1 -- 172.21.15.155:0/2189835035 learned_addr learned my addr 172.21.15.155:0/2189835035 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T08:59:20.932 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:20.918+0000 7fa8f259c700 1 -- 172.21.15.155:0/2189835035 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8e4005040 con 0x7fa8f40f5410 2024-02-10T08:59:20.932 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:20.918+0000 7fa8f259c700 1 --2- 172.21.15.155:0/2189835035 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8f40f5410 0x7fa8f41094e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fa8e4005010 tx=0x7fa8e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:59:20.932 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:20.918+0000 7fa8e2ffd700 1 -- 172.21.15.155:0/2189835035 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8e4009320 con 0x7fa8f40f5410 2024-02-10T08:59:20.932 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:20.919+0000 7fa8f8a89700 1 -- 172.21.15.155:0/2189835035 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8f4060440 con 0x7fa8f40f5410 2024-02-10T08:59:20.932 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:20.919+0000 7fa8e2ffd700 1 -- 172.21.15.155:0/2189835035 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa8e4013380 con 0x7fa8f40f5410 2024-02-10T08:59:20.932 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:20.919+0000 7fa8e2ffd700 1 -- 172.21.15.155:0/2189835035 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8e4014e50 con 0x7fa8f40f5410 2024-02-10T08:59:20.932 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:20.919+0000 7fa8f8a89700 1 -- 172.21.15.155:0/2189835035 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8f4109b90 con 0x7fa8f40f5410 2024-02-10T08:59:20.933 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:20.919+0000 7fa8e2ffd700 1 -- 172.21.15.155:0/2189835035 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa8e40095f0 con 0x7fa8f40f5410 2024-02-10T08:59:20.933 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:20.919+0000 7fa8f8a89700 1 -- 172.21.15.155:0/2189835035 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8f4045ad0 con 0x7fa8f40f5410 2024-02-10T08:59:20.933 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:20.919+0000 7fa8e2ffd700 1 --2- 172.21.15.155:0/2189835035 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fa8dc02ff10 0x7fa8dc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:59:20.933 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:20.920+0000 7fa8e2ffd700 1 -- 172.21.15.155:0/2189835035 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa8e4040e50 con 0x7fa8f40f5410 2024-02-10T08:59:20.933 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:20.920+0000 7fa8f1d9b700 1 --2- 172.21.15.155:0/2189835035 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fa8dc02ff10 0x7fa8dc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:59:20.933 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:20.920+0000 7fa8f1d9b700 1 --2- 172.21.15.155:0/2189835035 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fa8dc02ff10 0x7fa8dc0323d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fa8e80096a0 tx=0x7fa8e8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:59:20.933 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:20.924+0000 7fa8e2ffd700 1 -- 172.21.15.155:0/2189835035 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa8e4014420 con 0x7fa8f40f5410 2024-02-10T08:59:21.097 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:21.097+0000 7fa8f8a89700 1 -- 172.21.15.155:0/2189835035 --> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi155", "target": ["mon-mgr", ""]}) v1 -- 0x7fa8f40f1d20 con 0x7fa8dc02ff10 2024-02-10T08:59:21.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:59:21 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T08:59:20.958800+0000 mgr.smithi077.dxntvw (mgr.14162) 13 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:59:21.703 INFO:teuthology.orchestra.run.smithi155.stdout:Added host 'smithi155' 2024-02-10T08:59:21.704 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:21.702+0000 7fa8e2ffd700 1 -- 172.21.15.155:0/2189835035 <== mgr.14162 v2:172.21.15.77:6800/81674875 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7fa8f40f1d20 con 0x7fa8dc02ff10 2024-02-10T08:59:21.705 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:21.705+0000 7fa8f8a89700 1 -- 172.21.15.155:0/2189835035 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fa8dc02ff10 msgr2=0x7fa8dc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:59:21.705 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:21.705+0000 7fa8f8a89700 1 --2- 172.21.15.155:0/2189835035 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fa8dc02ff10 0x7fa8dc0323d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fa8e80096a0 tx=0x7fa8e8006b40).stop 2024-02-10T08:59:21.705 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:21.705+0000 7fa8f8a89700 1 -- 172.21.15.155:0/2189835035 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8f40f5410 msgr2=0x7fa8f41094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:59:21.705 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:21.705+0000 7fa8f8a89700 1 --2- 172.21.15.155:0/2189835035 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8f40f5410 0x7fa8f41094e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fa8e4005010 tx=0x7fa8e400b040).stop 2024-02-10T08:59:21.705 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:21.705+0000 7fa8f8a89700 1 -- 172.21.15.155:0/2189835035 shutdown_connections 2024-02-10T08:59:21.705 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:21.705+0000 7fa8f8a89700 1 --2- 172.21.15.155:0/2189835035 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fa8dc02ff10 0x7fa8dc0323d0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:59:21.705 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:21.705+0000 7fa8f8a89700 1 --2- 172.21.15.155:0/2189835035 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8f40f5410 0x7fa8f41094e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:59:21.705 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:21.705+0000 7fa8f8a89700 1 -- 172.21.15.155:0/2189835035 >> 172.21.15.155:0/2189835035 conn(0x7fa8f40f0b70 msgr2=0x7fa8f40f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:59:21.705 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:21.705+0000 7fa8f8a89700 1 -- 172.21.15.155:0/2189835035 shutdown_connections 2024-02-10T08:59:21.705 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:21.705+0000 7fa8f8a89700 1 -- 172.21.15.155:0/2189835035 wait complete. 2024-02-10T08:59:22.405 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph orch host ls --format=json 2024-02-10T08:59:23.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:59:22 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:59:21.098418+0000 mgr.smithi077.dxntvw (mgr.14162) 14 : audit [DBG] from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi155", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:59:23.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:59:22 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:59:21.701957+0000 mon.smithi077 (mon.0) 116 : audit [INF] from='mgr.14162 172.21.15.77:0/2029106299' entity='mgr.smithi077.dxntvw' 2024-02-10T08:59:23.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:59:22 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cephadm 2024-02-10T08:59:21.702379+0000 mgr.smithi077.dxntvw (mgr.14162) 15 : cephadm [INF] Added host smithi155 2024-02-10T08:59:24.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:59:23 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T08:59:22.959196+0000 mgr.smithi077.dxntvw (mgr.14162) 16 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:59:24.422 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:24.420+0000 7f63f948a700 1 -- 172.21.15.155:0/1337770893 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63f40f5410 msgr2=0x7f63f40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:59:24.422 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:24.420+0000 7f63f948a700 1 --2- 172.21.15.155:0/1337770893 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63f40f5410 0x7f63f40f5830 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f63e8009fb0 tx=0x7f63e8009b30).stop 2024-02-10T08:59:24.422 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:24.420+0000 7f63f948a700 1 -- 172.21.15.155:0/1337770893 shutdown_connections 2024-02-10T08:59:24.422 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:24.420+0000 7f63f948a700 1 --2- 172.21.15.155:0/1337770893 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63f40f5410 0x7f63f40f5830 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:59:24.422 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:24.420+0000 7f63f948a700 1 -- 172.21.15.155:0/1337770893 >> 172.21.15.155:0/1337770893 conn(0x7f63f40f0b70 msgr2=0x7f63f40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:59:24.422 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:24.420+0000 7f63f948a700 1 -- 172.21.15.155:0/1337770893 shutdown_connections 2024-02-10T08:59:24.422 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:24.420+0000 7f63f948a700 1 -- 172.21.15.155:0/1337770893 wait complete. 2024-02-10T08:59:24.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:24.421+0000 7f63f948a700 1 Processor -- start 2024-02-10T08:59:24.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:24.421+0000 7f63f948a700 1 -- start start 2024-02-10T08:59:24.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:24.421+0000 7f63f948a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63f40f5410 0x7f63f40fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:59:24.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:24.421+0000 7f63f948a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63f40fb960 con 0x7f63f40f5410 2024-02-10T08:59:24.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:24.421+0000 7f63f2ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63f40f5410 0x7f63f40fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:59:24.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:24.421+0000 7f63f2ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63f40f5410 0x7f63f40fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:42092/0 (socket says 172.21.15.155:42092) 2024-02-10T08:59:24.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:24.421+0000 7f63f2ffd700 1 -- 172.21.15.155:0/251667209 learned_addr learned my addr 172.21.15.155:0/251667209 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T08:59:24.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:24.422+0000 7f63f2ffd700 1 -- 172.21.15.155:0/251667209 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63e8005040 con 0x7f63f40f5410 2024-02-10T08:59:24.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:24.422+0000 7f63f2ffd700 1 --2- 172.21.15.155:0/251667209 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63f40f5410 0x7f63f40fd4e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f63e8005010 tx=0x7f63e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:59:24.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:24.422+0000 7f63db7fe700 1 -- 172.21.15.155:0/251667209 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f63e8009320 con 0x7f63f40f5410 2024-02-10T08:59:24.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:24.422+0000 7f63db7fe700 1 -- 172.21.15.155:0/251667209 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f63e8013380 con 0x7f63f40f5410 2024-02-10T08:59:24.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:24.422+0000 7f63f948a700 1 -- 172.21.15.155:0/251667209 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63f40fbb00 con 0x7f63f40f5410 2024-02-10T08:59:24.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:24.422+0000 7f63db7fe700 1 -- 172.21.15.155:0/251667209 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f63e8014e50 con 0x7f63f40f5410 2024-02-10T08:59:24.424 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:24.422+0000 7f63f948a700 1 -- 172.21.15.155:0/251667209 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63f40fbf90 con 0x7f63f40f5410 2024-02-10T08:59:24.424 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:24.423+0000 7f63db7fe700 1 -- 172.21.15.155:0/251667209 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f63e8014420 con 0x7f63f40f5410 2024-02-10T08:59:24.425 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:24.423+0000 7f63f948a700 1 -- 172.21.15.155:0/251667209 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63f4059100 con 0x7f63f40f5410 2024-02-10T08:59:24.425 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:24.423+0000 7f63db7fe700 1 --2- 172.21.15.155:0/251667209 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f63dc02ff10 0x7f63dc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:59:24.425 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:24.423+0000 7f63db7fe700 1 -- 172.21.15.155:0/251667209 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f63e8040e30 con 0x7f63f40f5410 2024-02-10T08:59:24.425 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:24.423+0000 7f63f27fc700 1 --2- 172.21.15.155:0/251667209 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f63dc02ff10 0x7f63dc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:59:24.425 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:24.423+0000 7f63f27fc700 1 --2- 172.21.15.155:0/251667209 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f63dc02ff10 0x7f63dc0323d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f63e40096a0 tx=0x7f63e4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:59:24.427 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:24.428+0000 7f63db7fe700 1 -- 172.21.15.155:0/251667209 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f63e80095f0 con 0x7f63f40f5410 2024-02-10T08:59:24.599 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:24.599+0000 7f63f948a700 1 -- 172.21.15.155:0/251667209 --> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f63f40fc7e0 con 0x7f63dc02ff10 2024-02-10T08:59:24.601 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:24.601+0000 7f63db7fe700 1 -- 172.21.15.155:0/251667209 <== mgr.14162 v2:172.21.15.77:6800/81674875 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+156 (secure 0 0 0) 0x7f63f40fc7e0 con 0x7f63dc02ff10 2024-02-10T08:59:24.601 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:59:24.601 INFO:teuthology.orchestra.run.smithi155.stdout:[{"addr": "172.21.15.77", "hostname": "smithi077", "labels": [], "status": ""}, {"addr": "smithi155", "hostname": "smithi155", "labels": [], "status": ""}] 2024-02-10T08:59:24.603 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:24.602+0000 7f63f948a700 1 -- 172.21.15.155:0/251667209 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f63dc02ff10 msgr2=0x7f63dc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:59:24.603 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:24.602+0000 7f63f948a700 1 --2- 172.21.15.155:0/251667209 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f63dc02ff10 0x7f63dc0323d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f63e40096a0 tx=0x7f63e4006b40).stop 2024-02-10T08:59:24.603 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:24.602+0000 7f63f948a700 1 -- 172.21.15.155:0/251667209 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63f40f5410 msgr2=0x7f63f40fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:59:24.603 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:24.602+0000 7f63f948a700 1 --2- 172.21.15.155:0/251667209 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63f40f5410 0x7f63f40fd4e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f63e8005010 tx=0x7f63e800b040).stop 2024-02-10T08:59:24.603 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:24.602+0000 7f63f948a700 1 -- 172.21.15.155:0/251667209 shutdown_connections 2024-02-10T08:59:24.603 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:24.602+0000 7f63f948a700 1 --2- 172.21.15.155:0/251667209 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f63dc02ff10 0x7f63dc0323d0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:59:24.603 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:24.602+0000 7f63f948a700 1 --2- 172.21.15.155:0/251667209 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f63f40f5410 0x7f63f40fd4e0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:59:24.603 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:24.603+0000 7f63f948a700 1 -- 172.21.15.155:0/251667209 >> 172.21.15.155:0/251667209 conn(0x7f63f40f0b70 msgr2=0x7f63f405fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:59:24.603 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:24.603+0000 7f63f948a700 1 -- 172.21.15.155:0/251667209 shutdown_connections 2024-02-10T08:59:24.603 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:24.603+0000 7f63f948a700 1 -- 172.21.15.155:0/251667209 wait complete. 2024-02-10T08:59:25.226 INFO:tasks.cephadm:Setting crush tunables to default 2024-02-10T08:59:25.227 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph osd crush tunables default 2024-02-10T08:59:25.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:59:25 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:59:24.600785+0000 mgr.smithi077.dxntvw (mgr.14162) 17 : audit [DBG] from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T08:59:25.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:59:25 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T08:59:24.959527+0000 mgr.smithi077.dxntvw (mgr.14162) 18 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:59:27.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:59:27.103+0000 7f440c908700 1 -- 172.21.15.77:0/2937163331 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44080601e0 msgr2=0x7f44080f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:59:27.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:59:27.103+0000 7f440c908700 1 --2- 172.21.15.77:0/2937163331 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44080601e0 0x7f44080f8f20 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f43f8009fb0 tx=0x7f43f8009b30).stop 2024-02-10T08:59:27.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:59:27.103+0000 7f440c908700 1 -- 172.21.15.77:0/2937163331 shutdown_connections 2024-02-10T08:59:27.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:59:27.103+0000 7f440c908700 1 --2- 172.21.15.77:0/2937163331 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44080601e0 0x7f44080f8f20 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:59:27.147 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:59:27.103+0000 7f440c908700 1 -- 172.21.15.77:0/2937163331 >> 172.21.15.77:0/2937163331 conn(0x7f44080f09a0 msgr2=0x7f44080f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:59:27.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:59:27.103+0000 7f440c908700 1 -- 172.21.15.77:0/2937163331 shutdown_connections 2024-02-10T08:59:27.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:59:27.103+0000 7f440c908700 1 -- 172.21.15.77:0/2937163331 wait complete. 2024-02-10T08:59:27.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:59:27.104+0000 7f440c908700 1 Processor -- start 2024-02-10T08:59:27.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:59:27.104+0000 7f440c908700 1 -- start start 2024-02-10T08:59:27.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:59:27.104+0000 7f440c908700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44080601e0 0x7f4408106600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:59:27.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:59:27.104+0000 7f440c908700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4408106af0 con 0x7f44080601e0 2024-02-10T08:59:27.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:59:27.104+0000 7f44077fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44080601e0 0x7f4408106600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:59:27.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:59:27.104+0000 7f44077fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44080601e0 0x7f4408106600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46042/0 (socket says 172.21.15.77:46042) 2024-02-10T08:59:27.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:59:27.104+0000 7f44077fe700 1 -- 172.21.15.77:0/1654539033 learned_addr learned my addr 172.21.15.77:0/1654539033 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-10T08:59:27.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:59:27.104+0000 7f44077fe700 1 -- 172.21.15.77:0/1654539033 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43f8005040 con 0x7f44080601e0 2024-02-10T08:59:27.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:59:27.105+0000 7f44077fe700 1 --2- 172.21.15.77:0/1654539033 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44080601e0 0x7f4408106600 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f43f80089f0 tx=0x7f43f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:59:27.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:59:27.105+0000 7f43effff700 1 -- 172.21.15.77:0/1654539033 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f43f8009320 con 0x7f44080601e0 2024-02-10T08:59:27.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:59:27.105+0000 7f43effff700 1 -- 172.21.15.77:0/1654539033 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f43f8013380 con 0x7f44080601e0 2024-02-10T08:59:27.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:59:27.105+0000 7f440c908700 1 -- 172.21.15.77:0/1654539033 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4408106cf0 con 0x7f44080601e0 2024-02-10T08:59:27.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:59:27.105+0000 7f43effff700 1 -- 172.21.15.77:0/1654539033 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f43f801ed80 con 0x7f44080601e0 2024-02-10T08:59:27.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:59:27.105+0000 7f440c908700 1 -- 172.21.15.77:0/1654539033 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4408107180 con 0x7f44080601e0 2024-02-10T08:59:27.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:59:27.105+0000 7f43effff700 1 -- 172.21.15.77:0/1654539033 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f43f8014420 con 0x7f44080601e0 2024-02-10T08:59:27.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:59:27.105+0000 7f43effff700 1 --2- 172.21.15.77:0/1654539033 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f43f002ff60 0x7f43f0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:59:27.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:59:27.106+0000 7f43effff700 1 -- 172.21.15.77:0/1654539033 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f43f801e4b0 con 0x7f44080601e0 2024-02-10T08:59:27.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:59:27.106+0000 7f4406ffd700 1 --2- 172.21.15.77:0/1654539033 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f43f002ff60 0x7f43f0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:59:27.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:59:27.106+0000 7f440c908700 1 -- 172.21.15.77:0/1654539033 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4408059100 con 0x7f44080601e0 2024-02-10T08:59:27.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:59:27.106+0000 7f4406ffd700 1 --2- 172.21.15.77:0/1654539033 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f43f002ff60 0x7f43f0032420 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f43fc0096a0 tx=0x7f43fc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:59:27.149 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:59:27.110+0000 7f43effff700 1 -- 172.21.15.77:0/1654539033 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f43f801c910 con 0x7f44080601e0 2024-02-10T08:59:27.150 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:59:27 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T08:59:26.959922+0000 mgr.smithi077.dxntvw (mgr.14162) 19 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:59:27.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:59:27.263+0000 7f440c908700 1 -- 172.21.15.77:0/1654539033 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f44080fbe10 con 0x7f44080601e0 2024-02-10T08:59:28.032 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:59:28.031+0000 7f43effff700 1 -- 172.21.15.77:0/1654539033 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v5) v1 ==== 124+0+0 (secure 0 0 0) 0x7f43f801c320 con 0x7f44080601e0 2024-02-10T08:59:28.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:59:28.033+0000 7f440c908700 1 -- 172.21.15.77:0/1654539033 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f43f002ff60 msgr2=0x7f43f0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:59:28.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:59:28.033+0000 7f440c908700 1 --2- 172.21.15.77:0/1654539033 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f43f002ff60 0x7f43f0032420 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f43fc0096a0 tx=0x7f43fc006b40).stop 2024-02-10T08:59:28.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:59:28.033+0000 7f440c908700 1 -- 172.21.15.77:0/1654539033 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44080601e0 msgr2=0x7f4408106600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:59:28.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:59:28.033+0000 7f440c908700 1 --2- 172.21.15.77:0/1654539033 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44080601e0 0x7f4408106600 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f43f80089f0 tx=0x7f43f800b040).stop 2024-02-10T08:59:28.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:59:28.033+0000 7f440c908700 1 -- 172.21.15.77:0/1654539033 shutdown_connections 2024-02-10T08:59:28.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:59:28.033+0000 7f440c908700 1 --2- 172.21.15.77:0/1654539033 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f43f002ff60 0x7f43f0032420 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:59:28.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:59:28.033+0000 7f440c908700 1 --2- 172.21.15.77:0/1654539033 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44080601e0 0x7f4408106600 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:59:28.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:59:28.033+0000 7f440c908700 1 -- 172.21.15.77:0/1654539033 >> 172.21.15.77:0/1654539033 conn(0x7f44080f09a0 msgr2=0x7f44080f13d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:59:28.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:59:28.033+0000 7f440c908700 1 -- 172.21.15.77:0/1654539033 shutdown_connections 2024-02-10T08:59:28.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-10T08:59:28.033+0000 7f440c908700 1 -- 172.21.15.77:0/1654539033 wait complete. 2024-02-10T08:59:28.035 INFO:teuthology.orchestra.run.smithi077.stderr:adjusted tunables profile to default 2024-02-10T08:59:28.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:59:28 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:59:27.264837+0000 mon.smithi077 (mon.0) 117 : audit [INF] from='client.? 172.21.15.77:0/1654539033' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-10T08:59:28.754 INFO:tasks.cephadm:Adding mon.smithi077 on smithi077 2024-02-10T08:59:28.754 INFO:tasks.cephadm:Adding mon.smithi155 on smithi155 2024-02-10T08:59:28.754 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph orch apply mon '2;smithi077:172.21.15.77=smithi077;smithi155:172.21.15.155=smithi155' 2024-02-10T08:59:29.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:59:29 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:59:28.032293+0000 mon.smithi077 (mon.0) 118 : audit [INF] from='client.? 172.21.15.77:0/1654539033' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-10T08:59:29.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:59:29 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T08:59:28.032352+0000 mon.smithi077 (mon.0) 119 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2024-02-10T08:59:29.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:59:29 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T08:59:28.960258+0000 mgr.smithi077.dxntvw (mgr.14162) 20 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:59:30.717 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:30.714+0000 7fc0a3957700 1 -- 172.21.15.155:0/218832866 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc09c060220 msgr2=0x7fc09c0fb550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:59:30.717 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:30.714+0000 7fc0a3957700 1 --2- 172.21.15.155:0/218832866 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc09c060220 0x7fc09c0fb550 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fc08c009fb0 tx=0x7fc08c009b30).stop 2024-02-10T08:59:30.717 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:30.714+0000 7fc0a3957700 1 -- 172.21.15.155:0/218832866 shutdown_connections 2024-02-10T08:59:30.717 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:30.714+0000 7fc0a3957700 1 --2- 172.21.15.155:0/218832866 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc09c060220 0x7fc09c0fb550 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:59:30.717 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:30.714+0000 7fc0a3957700 1 -- 172.21.15.155:0/218832866 >> 172.21.15.155:0/218832866 conn(0x7fc09c0f0b70 msgr2=0x7fc09c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:59:30.717 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:30.714+0000 7fc0a3957700 1 -- 172.21.15.155:0/218832866 shutdown_connections 2024-02-10T08:59:30.718 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:30.714+0000 7fc0a3957700 1 -- 172.21.15.155:0/218832866 wait complete. 2024-02-10T08:59:30.718 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:30.715+0000 7fc0a3957700 1 Processor -- start 2024-02-10T08:59:30.718 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:30.715+0000 7fc0a3957700 1 -- start start 2024-02-10T08:59:30.718 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:30.715+0000 7fc0a3957700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc09c060220 0x7fc09c1089d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:59:30.718 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:30.715+0000 7fc0a3957700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc09c108ec0 con 0x7fc09c060220 2024-02-10T08:59:30.718 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:30.715+0000 7fc0a16f3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc09c060220 0x7fc09c1089d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:59:30.718 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:30.715+0000 7fc0a16f3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc09c060220 0x7fc09c1089d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:42122/0 (socket says 172.21.15.155:42122) 2024-02-10T08:59:30.718 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:30.715+0000 7fc0a16f3700 1 -- 172.21.15.155:0/4168352044 learned_addr learned my addr 172.21.15.155:0/4168352044 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T08:59:30.718 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:30.716+0000 7fc0a16f3700 1 -- 172.21.15.155:0/4168352044 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc08c005040 con 0x7fc09c060220 2024-02-10T08:59:30.718 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:30.716+0000 7fc0a16f3700 1 --2- 172.21.15.155:0/4168352044 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc09c060220 0x7fc09c1089d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fc08c0089f0 tx=0x7fc08c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:59:30.718 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:30.716+0000 7fc091ffb700 1 -- 172.21.15.155:0/4168352044 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc08c009320 con 0x7fc09c060220 2024-02-10T08:59:30.718 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:30.716+0000 7fc091ffb700 1 -- 172.21.15.155:0/4168352044 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc08c013380 con 0x7fc09c060220 2024-02-10T08:59:30.718 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:30.716+0000 7fc0a3957700 1 -- 172.21.15.155:0/4168352044 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc09c1090c0 con 0x7fc09c060220 2024-02-10T08:59:30.718 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:30.716+0000 7fc091ffb700 1 -- 172.21.15.155:0/4168352044 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc08c014e20 con 0x7fc09c060220 2024-02-10T08:59:30.718 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:30.716+0000 7fc0a3957700 1 -- 172.21.15.155:0/4168352044 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc09c109550 con 0x7fc09c060220 2024-02-10T08:59:30.719 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:30.717+0000 7fc091ffb700 1 -- 172.21.15.155:0/4168352044 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc08c0263a0 con 0x7fc09c060220 2024-02-10T08:59:30.719 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:30.717+0000 7fc091ffb700 1 --2- 172.21.15.155:0/4168352044 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fc08802ff10 0x7fc0880323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:59:30.720 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:30.717+0000 7fc091ffb700 1 -- 172.21.15.155:0/4168352044 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc08c040f70 con 0x7fc09c060220 2024-02-10T08:59:30.720 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:30.717+0000 7fc0a3957700 1 -- 172.21.15.155:0/4168352044 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc09c045ad0 con 0x7fc09c060220 2024-02-10T08:59:30.720 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:30.717+0000 7fc0a0ef2700 1 --2- 172.21.15.155:0/4168352044 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fc08802ff10 0x7fc0880323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:59:30.720 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:30.718+0000 7fc0a0ef2700 1 --2- 172.21.15.155:0/4168352044 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fc08802ff10 0x7fc0880323d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fc0980096a0 tx=0x7fc098006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:59:30.723 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:30.722+0000 7fc091ffb700 1 -- 172.21.15.155:0/4168352044 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc08c012920 con 0x7fc09c060220 2024-02-10T08:59:30.898 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:30.897+0000 7fc0a3957700 1 -- 172.21.15.155:0/4168352044 --> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi077:172.21.15.77=smithi077;smithi155:172.21.15.155=smithi155", "target": ["mon-mgr", ""]}) v1 -- 0x7fc09c0f1df0 con 0x7fc08802ff10 2024-02-10T08:59:30.904 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:30.903+0000 7fc091ffb700 1 -- 172.21.15.155:0/4168352044 <== mgr.14162 v2:172.21.15.77:6800/81674875 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fc09c0f1df0 con 0x7fc08802ff10 2024-02-10T08:59:30.904 INFO:teuthology.orchestra.run.smithi155.stdout:Scheduled mon update... 2024-02-10T08:59:30.905 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:30.904+0000 7fc0a3957700 1 -- 172.21.15.155:0/4168352044 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fc08802ff10 msgr2=0x7fc0880323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:59:30.906 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:30.904+0000 7fc0a3957700 1 --2- 172.21.15.155:0/4168352044 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fc08802ff10 0x7fc0880323d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fc0980096a0 tx=0x7fc098006b40).stop 2024-02-10T08:59:30.906 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:30.904+0000 7fc0a3957700 1 -- 172.21.15.155:0/4168352044 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc09c060220 msgr2=0x7fc09c1089d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:59:30.906 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:30.904+0000 7fc0a3957700 1 --2- 172.21.15.155:0/4168352044 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc09c060220 0x7fc09c1089d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fc08c0089f0 tx=0x7fc08c00b040).stop 2024-02-10T08:59:30.906 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:30.904+0000 7fc0a3957700 1 -- 172.21.15.155:0/4168352044 shutdown_connections 2024-02-10T08:59:30.906 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:30.904+0000 7fc0a3957700 1 --2- 172.21.15.155:0/4168352044 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fc08802ff10 0x7fc0880323d0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:59:30.906 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:30.904+0000 7fc0a3957700 1 --2- 172.21.15.155:0/4168352044 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc09c060220 0x7fc09c1089d0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:59:30.906 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:30.904+0000 7fc0a3957700 1 -- 172.21.15.155:0/4168352044 >> 172.21.15.155:0/4168352044 conn(0x7fc09c0f0b70 msgr2=0x7fc09c0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:59:30.906 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:30.904+0000 7fc0a3957700 1 -- 172.21.15.155:0/4168352044 shutdown_connections 2024-02-10T08:59:30.906 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:30.905+0000 7fc0a3957700 1 -- 172.21.15.155:0/4168352044 wait complete. 2024-02-10T08:59:31.571 DEBUG:teuthology.orchestra.run.smithi155:mon.smithi155> sudo journalctl -f -n 0 -u ceph-6081b54a-c7f2-11ee-95b8-87774f69a715@mon.smithi155.service 2024-02-10T08:59:31.575 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T08:59:31.576 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T08:59:31.603 INFO:journalctl@ceph.mon.smithi155.smithi155.stdout:-- Logs begin at Sat 2024-02-10 08:38:53 UTC. -- 2024-02-10T08:59:32.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:59:31 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:59:30.899928+0000 mgr.smithi077.dxntvw (mgr.14162) 21 : audit [DBG] from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi077:172.21.15.77=smithi077;smithi155:172.21.15.155=smithi155", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T08:59:32.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:59:31 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cephadm 2024-02-10T08:59:30.901293+0000 mgr.smithi077.dxntvw (mgr.14162) 22 : cephadm [INF] Saving service mon spec with placement smithi077:172.21.15.77=smithi077;smithi155:172.21.15.155=smithi155;count:2 2024-02-10T08:59:32.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:59:31 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:59:30.904042+0000 mon.smithi077 (mon.0) 120 : audit [INF] from='mgr.14162 172.21.15.77:0/2029106299' entity='mgr.smithi077.dxntvw' 2024-02-10T08:59:32.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:59:31 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T08:59:30.960552+0000 mgr.smithi077.dxntvw (mgr.14162) 23 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:59:33.675 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:33.658+0000 7fd9ea4b5700 1 -- 172.21.15.155:0/3049472895 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9e40601e0 msgr2=0x7fd9e40f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:59:33.676 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:33.658+0000 7fd9ea4b5700 1 --2- 172.21.15.155:0/3049472895 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9e40601e0 0x7fd9e40f9190 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fd9d4009fb0 tx=0x7fd9d4009b30).stop 2024-02-10T08:59:33.676 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:33.658+0000 7fd9ea4b5700 1 -- 172.21.15.155:0/3049472895 shutdown_connections 2024-02-10T08:59:33.676 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:33.658+0000 7fd9ea4b5700 1 --2- 172.21.15.155:0/3049472895 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9e40601e0 0x7fd9e40f9190 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:59:33.676 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:33.658+0000 7fd9ea4b5700 1 -- 172.21.15.155:0/3049472895 >> 172.21.15.155:0/3049472895 conn(0x7fd9e40f0b70 msgr2=0x7fd9e40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:59:33.676 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:33.658+0000 7fd9ea4b5700 1 -- 172.21.15.155:0/3049472895 shutdown_connections 2024-02-10T08:59:33.676 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:33.658+0000 7fd9ea4b5700 1 -- 172.21.15.155:0/3049472895 wait complete. 2024-02-10T08:59:33.676 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:33.659+0000 7fd9ea4b5700 1 Processor -- start 2024-02-10T08:59:33.676 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:33.659+0000 7fd9ea4b5700 1 -- start start 2024-02-10T08:59:33.676 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:33.659+0000 7fd9ea4b5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9e40601e0 0x7fd9e40f6c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:59:33.676 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:33.659+0000 7fd9ea4b5700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e40f5090 con 0x7fd9e40601e0 2024-02-10T08:59:33.676 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:33.659+0000 7fd9e3fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9e40601e0 0x7fd9e40f6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:59:33.677 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:33.659+0000 7fd9e3fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9e40601e0 0x7fd9e40f6c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:48980/0 (socket says 172.21.15.155:48980) 2024-02-10T08:59:33.677 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:33.659+0000 7fd9e3fff700 1 -- 172.21.15.155:0/3903832788 learned_addr learned my addr 172.21.15.155:0/3903832788 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T08:59:33.677 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:33.660+0000 7fd9e3fff700 1 -- 172.21.15.155:0/3903832788 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d4005040 con 0x7fd9e40601e0 2024-02-10T08:59:33.677 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:33.660+0000 7fd9e3fff700 1 --2- 172.21.15.155:0/3903832788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9e40601e0 0x7fd9e40f6c10 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fd9d4007a10 tx=0x7fd9d4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:59:33.677 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:33.660+0000 7fd9e0ff9700 1 -- 172.21.15.155:0/3903832788 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd9d4018070 con 0x7fd9e40601e0 2024-02-10T08:59:33.677 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:33.660+0000 7fd9ea4b5700 1 -- 172.21.15.155:0/3903832788 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9e40f5230 con 0x7fd9e40601e0 2024-02-10T08:59:33.677 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:33.660+0000 7fd9e0ff9700 1 -- 172.21.15.155:0/3903832788 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd9d4013410 con 0x7fd9e40601e0 2024-02-10T08:59:33.677 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:33.660+0000 7fd9e0ff9700 1 -- 172.21.15.155:0/3903832788 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd9d401c3c0 con 0x7fd9e40601e0 2024-02-10T08:59:33.677 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:33.660+0000 7fd9ea4b5700 1 -- 172.21.15.155:0/3903832788 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9e40f56c0 con 0x7fd9e40601e0 2024-02-10T08:59:33.677 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:33.661+0000 7fd9e0ff9700 1 -- 172.21.15.155:0/3903832788 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd9d401d420 con 0x7fd9e40601e0 2024-02-10T08:59:33.677 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:33.661+0000 7fd9ea4b5700 1 -- 172.21.15.155:0/3903832788 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9e4059100 con 0x7fd9e40601e0 2024-02-10T08:59:33.677 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:33.661+0000 7fd9e0ff9700 1 --2- 172.21.15.155:0/3903832788 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fd9cc02ff60 0x7fd9cc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:59:33.677 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:33.661+0000 7fd9e0ff9700 1 -- 172.21.15.155:0/3903832788 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd9d4041570 con 0x7fd9e40601e0 2024-02-10T08:59:33.677 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:33.661+0000 7fd9e37fe700 1 --2- 172.21.15.155:0/3903832788 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fd9cc02ff60 0x7fd9cc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:59:33.677 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:33.662+0000 7fd9e37fe700 1 --2- 172.21.15.155:0/3903832788 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fd9cc02ff60 0x7fd9cc032420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fd9d80096a0 tx=0x7fd9d8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:59:33.678 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:33.666+0000 7fd9e0ff9700 1 -- 172.21.15.155:0/3903832788 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd9d401d820 con 0x7fd9e40601e0 2024-02-10T08:59:33.871 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:33.870+0000 7fd9ea4b5700 1 -- 172.21.15.155:0/3903832788 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd9e40fc0a0 con 0x7fd9e40601e0 2024-02-10T08:59:33.872 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:33.871+0000 7fd9e0ff9700 1 -- 172.21.15.155:0/3903832788 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd9d401b370 con 0x7fd9e40601e0 2024-02-10T08:59:33.872 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:59:33.872 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T08:59:33.874 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:33.873+0000 7fd9ea4b5700 1 -- 172.21.15.155:0/3903832788 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fd9cc02ff60 msgr2=0x7fd9cc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:59:33.874 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:33.873+0000 7fd9ea4b5700 1 --2- 172.21.15.155:0/3903832788 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fd9cc02ff60 0x7fd9cc032420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fd9d80096a0 tx=0x7fd9d8006b40).stop 2024-02-10T08:59:33.874 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:33.873+0000 7fd9ea4b5700 1 -- 172.21.15.155:0/3903832788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9e40601e0 msgr2=0x7fd9e40f6c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:59:33.874 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:33.873+0000 7fd9ea4b5700 1 --2- 172.21.15.155:0/3903832788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9e40601e0 0x7fd9e40f6c10 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fd9d4007a10 tx=0x7fd9d4009b30).stop 2024-02-10T08:59:33.874 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:33.873+0000 7fd9ea4b5700 1 -- 172.21.15.155:0/3903832788 shutdown_connections 2024-02-10T08:59:33.874 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:33.873+0000 7fd9ea4b5700 1 --2- 172.21.15.155:0/3903832788 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fd9cc02ff60 0x7fd9cc032420 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:59:33.874 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:33.873+0000 7fd9ea4b5700 1 --2- 172.21.15.155:0/3903832788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9e40601e0 0x7fd9e40f6c10 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:59:33.874 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:33.873+0000 7fd9ea4b5700 1 -- 172.21.15.155:0/3903832788 >> 172.21.15.155:0/3903832788 conn(0x7fd9e40f0b70 msgr2=0x7fd9e40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:59:33.874 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:33.873+0000 7fd9ea4b5700 1 -- 172.21.15.155:0/3903832788 shutdown_connections 2024-02-10T08:59:33.874 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:33.873+0000 7fd9ea4b5700 1 -- 172.21.15.155:0/3903832788 wait complete. 2024-02-10T08:59:33.875 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T08:59:34.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:59:33 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T08:59:32.960905+0000 mgr.smithi077.dxntvw (mgr.14162) 24 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:59:35.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:59:34 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:59:33.872322+0000 mon.smithi077 (mon.0) 121 : audit [DBG] from='client.? 172.21.15.155:0/3903832788' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T08:59:35.499 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T08:59:35.499 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T08:59:36.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:59:35 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T08:59:34.961313+0000 mgr.smithi077.dxntvw (mgr.14162) 25 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:59:37.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:59:37 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T08:59:36.961541+0000 mgr.smithi077.dxntvw (mgr.14162) 26 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:59:38.077 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:38.074+0000 7ff6dc844700 1 -- 172.21.15.155:0/563227574 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6d80f51a0 msgr2=0x7ff6d80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:59:38.077 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:38.074+0000 7ff6dc844700 1 --2- 172.21.15.155:0/563227574 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6d80f51a0 0x7ff6d80f55c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7ff6c8009fb0 tx=0x7ff6c8009b30).stop 2024-02-10T08:59:38.077 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:38.075+0000 7ff6dc844700 1 -- 172.21.15.155:0/563227574 shutdown_connections 2024-02-10T08:59:38.077 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:38.075+0000 7ff6dc844700 1 --2- 172.21.15.155:0/563227574 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6d80f51a0 0x7ff6d80f55c0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:59:38.077 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:38.075+0000 7ff6dc844700 1 -- 172.21.15.155:0/563227574 >> 172.21.15.155:0/563227574 conn(0x7ff6d80f09a0 msgr2=0x7ff6d80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:59:38.077 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:38.075+0000 7ff6dc844700 1 -- 172.21.15.155:0/563227574 shutdown_connections 2024-02-10T08:59:38.077 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:38.075+0000 7ff6dc844700 1 -- 172.21.15.155:0/563227574 wait complete. 2024-02-10T08:59:38.077 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:38.075+0000 7ff6dc844700 1 Processor -- start 2024-02-10T08:59:38.078 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:38.075+0000 7ff6dc844700 1 -- start start 2024-02-10T08:59:38.078 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:38.075+0000 7ff6dc844700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6d80f51a0 0x7ff6d8109290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:59:38.078 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:38.075+0000 7ff6dc844700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d8060280 con 0x7ff6d80f51a0 2024-02-10T08:59:38.078 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:38.076+0000 7ff6d77fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6d80f51a0 0x7ff6d8109290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:59:38.078 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:38.076+0000 7ff6d77fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6d80f51a0 0x7ff6d8109290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:49004/0 (socket says 172.21.15.155:49004) 2024-02-10T08:59:38.078 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:38.076+0000 7ff6d77fe700 1 -- 172.21.15.155:0/2121209170 learned_addr learned my addr 172.21.15.155:0/2121209170 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T08:59:38.078 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:38.076+0000 7ff6d77fe700 1 -- 172.21.15.155:0/2121209170 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6c8005040 con 0x7ff6d80f51a0 2024-02-10T08:59:38.078 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:38.076+0000 7ff6d77fe700 1 --2- 172.21.15.155:0/2121209170 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6d80f51a0 0x7ff6d8109290 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7ff6c80089f0 tx=0x7ff6c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:59:38.079 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:38.076+0000 7ff6bffff700 1 -- 172.21.15.155:0/2121209170 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff6c8009320 con 0x7ff6d80f51a0 2024-02-10T08:59:38.080 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:38.076+0000 7ff6bffff700 1 -- 172.21.15.155:0/2121209170 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff6c8013380 con 0x7ff6d80f51a0 2024-02-10T08:59:38.080 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:38.076+0000 7ff6dc844700 1 -- 172.21.15.155:0/2121209170 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6d8060480 con 0x7ff6d80f51a0 2024-02-10T08:59:38.080 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:38.076+0000 7ff6bffff700 1 -- 172.21.15.155:0/2121209170 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff6c8014e20 con 0x7ff6d80f51a0 2024-02-10T08:59:38.080 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:38.076+0000 7ff6dc844700 1 -- 172.21.15.155:0/2121209170 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6d81099c0 con 0x7ff6d80f51a0 2024-02-10T08:59:38.080 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:38.077+0000 7ff6bffff700 1 -- 172.21.15.155:0/2121209170 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff6c80095f0 con 0x7ff6d80f51a0 2024-02-10T08:59:38.080 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:38.077+0000 7ff6dc844700 1 -- 172.21.15.155:0/2121209170 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6d8045ad0 con 0x7ff6d80f51a0 2024-02-10T08:59:38.080 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:38.077+0000 7ff6bffff700 1 --2- 172.21.15.155:0/2121209170 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7ff6c002ff10 0x7ff6c00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:59:38.080 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:38.077+0000 7ff6bffff700 1 -- 172.21.15.155:0/2121209170 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff6c8040e90 con 0x7ff6d80f51a0 2024-02-10T08:59:38.080 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:38.078+0000 7ff6d6ffd700 1 --2- 172.21.15.155:0/2121209170 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7ff6c002ff10 0x7ff6c00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:59:38.080 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:38.078+0000 7ff6d6ffd700 1 --2- 172.21.15.155:0/2121209170 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7ff6c002ff10 0x7ff6c00323d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7ff6cc0096a0 tx=0x7ff6cc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:59:38.083 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:38.082+0000 7ff6bffff700 1 -- 172.21.15.155:0/2121209170 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff6c8014420 con 0x7ff6d80f51a0 2024-02-10T08:59:38.287 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:38.286+0000 7ff6dc844700 1 -- 172.21.15.155:0/2121209170 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff6d80fad30 con 0x7ff6d80f51a0 2024-02-10T08:59:38.287 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:38.286+0000 7ff6bffff700 1 -- 172.21.15.155:0/2121209170 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff6c801d980 con 0x7ff6d80f51a0 2024-02-10T08:59:38.287 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:59:38.288 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T08:59:38.289 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:38.288+0000 7ff6dc844700 1 -- 172.21.15.155:0/2121209170 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7ff6c002ff10 msgr2=0x7ff6c00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:59:38.289 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:38.288+0000 7ff6dc844700 1 --2- 172.21.15.155:0/2121209170 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7ff6c002ff10 0x7ff6c00323d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7ff6cc0096a0 tx=0x7ff6cc006b40).stop 2024-02-10T08:59:38.289 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:38.288+0000 7ff6dc844700 1 -- 172.21.15.155:0/2121209170 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6d80f51a0 msgr2=0x7ff6d8109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:59:38.289 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:38.288+0000 7ff6dc844700 1 --2- 172.21.15.155:0/2121209170 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6d80f51a0 0x7ff6d8109290 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7ff6c80089f0 tx=0x7ff6c800b040).stop 2024-02-10T08:59:38.289 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:38.288+0000 7ff6dc844700 1 -- 172.21.15.155:0/2121209170 shutdown_connections 2024-02-10T08:59:38.289 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:38.288+0000 7ff6dc844700 1 --2- 172.21.15.155:0/2121209170 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7ff6c002ff10 0x7ff6c00323d0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:59:38.289 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:38.288+0000 7ff6dc844700 1 --2- 172.21.15.155:0/2121209170 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6d80f51a0 0x7ff6d8109290 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:59:38.289 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:38.288+0000 7ff6dc844700 1 -- 172.21.15.155:0/2121209170 >> 172.21.15.155:0/2121209170 conn(0x7ff6d80f09a0 msgr2=0x7ff6d80f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:59:38.289 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:38.288+0000 7ff6dc844700 1 -- 172.21.15.155:0/2121209170 shutdown_connections 2024-02-10T08:59:38.289 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:38.288+0000 7ff6dc844700 1 -- 172.21.15.155:0/2121209170 wait complete. 2024-02-10T08:59:38.290 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T08:59:39.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:59:38 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:59:38.287725+0000 mon.smithi077 (mon.0) 122 : audit [DBG] from='client.? 172.21.15.155:0/2121209170' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T08:59:39.918 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T08:59:39.919 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T08:59:40.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:59:39 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T08:59:38.961906+0000 mgr.smithi077.dxntvw (mgr.14162) 27 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:59:41.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:59:41 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T08:59:40.962312+0000 mgr.smithi077.dxntvw (mgr.14162) 28 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:59:44.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:59:43 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T08:59:42.962606+0000 mgr.smithi077.dxntvw (mgr.14162) 29 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:59:45.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:59:45 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T08:59:44.962895+0000 mgr.smithi077.dxntvw (mgr.14162) 30 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:59:45.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:59:45 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:59:45.019322+0000 mon.smithi077 (mon.0) 123 : audit [DBG] from='client.? 172.21.15.155:0/1966882349' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T08:59:46.022 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:44.802+0000 7ff24a05f700 1 -- 172.21.15.155:0/3838996624 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2440f6480 msgr2=0x7ff2440f68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:59:46.022 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:44.802+0000 7ff24a05f700 1 --2- 172.21.15.155:0/3838996624 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2440f6480 0x7ff2440f68a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7ff238009fb0 tx=0x7ff238009b30).stop 2024-02-10T08:59:46.022 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:44.802+0000 7ff24a05f700 1 -- 172.21.15.155:0/3838996624 shutdown_connections 2024-02-10T08:59:46.023 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:44.802+0000 7ff24a05f700 1 --2- 172.21.15.155:0/3838996624 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2440f6480 0x7ff2440f68a0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:59:46.023 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:44.802+0000 7ff24a05f700 1 -- 172.21.15.155:0/3838996624 >> 172.21.15.155:0/3838996624 conn(0x7ff2440f1c00 msgr2=0x7ff2440f4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:59:46.023 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:44.802+0000 7ff24a05f700 1 -- 172.21.15.155:0/3838996624 shutdown_connections 2024-02-10T08:59:46.023 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:44.802+0000 7ff24a05f700 1 -- 172.21.15.155:0/3838996624 wait complete. 2024-02-10T08:59:46.023 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:44.803+0000 7ff24a05f700 1 Processor -- start 2024-02-10T08:59:46.023 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:44.803+0000 7ff24a05f700 1 -- start start 2024-02-10T08:59:46.023 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:44.804+0000 7ff24a05f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2440f6480 0x7ff2441067f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:59:46.023 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:44.804+0000 7ff24a05f700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff244106ce0 con 0x7ff2440f6480 2024-02-10T08:59:46.023 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:44.804+0000 7ff2437fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2440f6480 0x7ff2441067f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:59:46.023 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:44.804+0000 7ff2437fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2440f6480 0x7ff2441067f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:59546/0 (socket says 172.21.15.155:59546) 2024-02-10T08:59:46.023 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:44.804+0000 7ff2437fe700 1 -- 172.21.15.155:0/1966882349 learned_addr learned my addr 172.21.15.155:0/1966882349 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T08:59:46.023 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:44.804+0000 7ff2437fe700 1 -- 172.21.15.155:0/1966882349 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff238005040 con 0x7ff2440f6480 2024-02-10T08:59:46.024 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:44.804+0000 7ff2437fe700 1 --2- 172.21.15.155:0/1966882349 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2440f6480 0x7ff2441067f0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7ff238005010 tx=0x7ff23800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:59:46.024 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:44.805+0000 7ff22bfff700 1 -- 172.21.15.155:0/1966882349 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff238009320 con 0x7ff2440f6480 2024-02-10T08:59:46.024 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:44.805+0000 7ff24a05f700 1 -- 172.21.15.155:0/1966882349 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff244106ee0 con 0x7ff2440f6480 2024-02-10T08:59:46.024 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:44.805+0000 7ff24a05f700 1 -- 172.21.15.155:0/1966882349 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff244107370 con 0x7ff2440f6480 2024-02-10T08:59:46.024 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:44.805+0000 7ff22bfff700 1 -- 172.21.15.155:0/1966882349 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff238013380 con 0x7ff2440f6480 2024-02-10T08:59:46.024 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:44.805+0000 7ff22bfff700 1 -- 172.21.15.155:0/1966882349 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff238014df0 con 0x7ff2440f6480 2024-02-10T08:59:46.024 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:44.805+0000 7ff22bfff700 1 -- 172.21.15.155:0/1966882349 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff2380095f0 con 0x7ff2440f6480 2024-02-10T08:59:46.024 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:44.805+0000 7ff24a05f700 1 -- 172.21.15.155:0/1966882349 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff244045ad0 con 0x7ff2440f6480 2024-02-10T08:59:46.024 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:44.806+0000 7ff22bfff700 1 --2- 172.21.15.155:0/1966882349 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7ff22c02ff10 0x7ff22c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:59:46.024 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:44.806+0000 7ff22bfff700 1 -- 172.21.15.155:0/1966882349 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff238040e60 con 0x7ff2440f6480 2024-02-10T08:59:46.024 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:44.806+0000 7ff242ffd700 1 --2- 172.21.15.155:0/1966882349 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7ff22c02ff10 0x7ff22c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:59:46.024 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:44.806+0000 7ff242ffd700 1 --2- 172.21.15.155:0/1966882349 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7ff22c02ff10 0x7ff22c0323d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7ff2340096a0 tx=0x7ff234006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:59:46.025 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:44.810+0000 7ff22bfff700 1 -- 172.21.15.155:0/1966882349 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff238014420 con 0x7ff2440f6480 2024-02-10T08:59:46.025 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:45.017+0000 7ff24a05f700 1 -- 172.21.15.155:0/1966882349 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff2440fc090 con 0x7ff2440f6480 2024-02-10T08:59:46.025 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:45.018+0000 7ff22bfff700 1 -- 172.21.15.155:0/1966882349 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff238010230 con 0x7ff2440f6480 2024-02-10T08:59:46.025 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:59:46.025 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T08:59:46.025 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:45.019+0000 7ff24a05f700 1 -- 172.21.15.155:0/1966882349 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7ff22c02ff10 msgr2=0x7ff22c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:59:46.025 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:45.019+0000 7ff24a05f700 1 --2- 172.21.15.155:0/1966882349 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7ff22c02ff10 0x7ff22c0323d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7ff2340096a0 tx=0x7ff234006b40).stop 2024-02-10T08:59:46.025 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:45.019+0000 7ff24a05f700 1 -- 172.21.15.155:0/1966882349 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2440f6480 msgr2=0x7ff2441067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:59:46.026 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:45.019+0000 7ff24a05f700 1 --2- 172.21.15.155:0/1966882349 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2440f6480 0x7ff2441067f0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7ff238005010 tx=0x7ff23800b040).stop 2024-02-10T08:59:46.026 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:45.019+0000 7ff24a05f700 1 -- 172.21.15.155:0/1966882349 shutdown_connections 2024-02-10T08:59:46.026 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:45.019+0000 7ff24a05f700 1 --2- 172.21.15.155:0/1966882349 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7ff22c02ff10 0x7ff22c0323d0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:59:46.026 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:45.019+0000 7ff24a05f700 1 --2- 172.21.15.155:0/1966882349 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2440f6480 0x7ff2441067f0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:59:46.026 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:45.019+0000 7ff24a05f700 1 -- 172.21.15.155:0/1966882349 >> 172.21.15.155:0/1966882349 conn(0x7ff2440f1c00 msgr2=0x7ff2440f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:59:46.026 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:45.020+0000 7ff24a05f700 1 -- 172.21.15.155:0/1966882349 shutdown_connections 2024-02-10T08:59:46.026 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:45.020+0000 7ff24a05f700 1 -- 172.21.15.155:0/1966882349 wait complete. 2024-02-10T08:59:46.026 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T08:59:47.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:59:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T08:59:46.963168+0000 mgr.smithi077.dxntvw (mgr.14162) 31 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:59:48.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:59:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:59:47.087317+0000 mon.smithi077 (mon.0) 124 : audit [INF] from='mgr.14162 172.21.15.77:0/2029106299' entity='mgr.smithi077.dxntvw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi077.dxntvw/mirror_snapshot_schedule"}]: dispatch 2024-02-10T08:59:48.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:59:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:59:47.095418+0000 mon.smithi077 (mon.0) 125 : audit [INF] from='mgr.14162 172.21.15.77:0/2029106299' entity='mgr.smithi077.dxntvw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi077.dxntvw/trash_purge_schedule"}]: dispatch 2024-02-10T08:59:49.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:59:49 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T08:59:48.963423+0000 mgr.smithi077.dxntvw (mgr.14162) 32 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:59:51.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:59:51 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T08:59:50.963745+0000 mgr.smithi077.dxntvw (mgr.14162) 33 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:59:52.041 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T08:59:52.041 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T08:59:54.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:59:53 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T08:59:52.964096+0000 mgr.smithi077.dxntvw (mgr.14162) 34 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:59:55.070 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:55.067+0000 7fd188482700 1 -- 172.21.15.155:0/1409359569 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1800f6510 msgr2=0x7fd1800f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:59:55.071 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:55.067+0000 7fd188482700 1 --2- 172.21.15.155:0/1409359569 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1800f6510 0x7fd1800f6930 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fd170009fb0 tx=0x7fd170009b30).stop 2024-02-10T08:59:55.071 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:55.067+0000 7fd188482700 1 -- 172.21.15.155:0/1409359569 shutdown_connections 2024-02-10T08:59:55.071 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:55.067+0000 7fd188482700 1 --2- 172.21.15.155:0/1409359569 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1800f6510 0x7fd1800f6930 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:59:55.071 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:55.067+0000 7fd188482700 1 -- 172.21.15.155:0/1409359569 >> 172.21.15.155:0/1409359569 conn(0x7fd1800f1c70 msgr2=0x7fd1800f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:59:55.071 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:55.067+0000 7fd188482700 1 -- 172.21.15.155:0/1409359569 shutdown_connections 2024-02-10T08:59:55.071 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:55.067+0000 7fd188482700 1 -- 172.21.15.155:0/1409359569 wait complete. 2024-02-10T08:59:55.071 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:55.068+0000 7fd188482700 1 Processor -- start 2024-02-10T08:59:55.071 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:55.068+0000 7fd188482700 1 -- start start 2024-02-10T08:59:55.071 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:55.068+0000 7fd188482700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1800f6510 0x7fd1801067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:59:55.071 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:55.068+0000 7fd188482700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd180106cd0 con 0x7fd1800f6510 2024-02-10T08:59:55.071 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:55.068+0000 7fd18621e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1800f6510 0x7fd1801067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:59:55.071 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:55.068+0000 7fd18621e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1800f6510 0x7fd1801067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:56286/0 (socket says 172.21.15.155:56286) 2024-02-10T08:59:55.071 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:55.068+0000 7fd18621e700 1 -- 172.21.15.155:0/2053170884 learned_addr learned my addr 172.21.15.155:0/2053170884 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T08:59:55.072 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:55.069+0000 7fd18621e700 1 -- 172.21.15.155:0/2053170884 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd170005040 con 0x7fd1800f6510 2024-02-10T08:59:55.072 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:55.069+0000 7fd18621e700 1 --2- 172.21.15.155:0/2053170884 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1800f6510 0x7fd1801067e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fd1700089f0 tx=0x7fd17000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:59:55.072 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:55.069+0000 7fd176ffd700 1 -- 172.21.15.155:0/2053170884 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd170009320 con 0x7fd1800f6510 2024-02-10T08:59:55.072 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:55.069+0000 7fd176ffd700 1 -- 172.21.15.155:0/2053170884 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd170013380 con 0x7fd1800f6510 2024-02-10T08:59:55.072 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:55.069+0000 7fd176ffd700 1 -- 172.21.15.155:0/2053170884 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd17001ed80 con 0x7fd1800f6510 2024-02-10T08:59:55.072 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:55.069+0000 7fd188482700 1 -- 172.21.15.155:0/2053170884 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd180106ed0 con 0x7fd1800f6510 2024-02-10T08:59:55.072 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:55.069+0000 7fd188482700 1 -- 172.21.15.155:0/2053170884 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd180107360 con 0x7fd1800f6510 2024-02-10T08:59:55.073 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:55.070+0000 7fd176ffd700 1 -- 172.21.15.155:0/2053170884 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd17001e420 con 0x7fd1800f6510 2024-02-10T08:59:55.073 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:55.070+0000 7fd188482700 1 -- 172.21.15.155:0/2053170884 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd180045ad0 con 0x7fd1800f6510 2024-02-10T08:59:55.073 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:55.071+0000 7fd176ffd700 1 --2- 172.21.15.155:0/2053170884 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fd16c02ffb0 0x7fd16c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:59:55.073 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:55.071+0000 7fd176ffd700 1 -- 172.21.15.155:0/2053170884 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd170040f30 con 0x7fd1800f6510 2024-02-10T08:59:55.073 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:55.071+0000 7fd185a1d700 1 --2- 172.21.15.155:0/2053170884 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fd16c02ffb0 0x7fd16c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:59:55.073 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:55.071+0000 7fd185a1d700 1 --2- 172.21.15.155:0/2053170884 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fd16c02ffb0 0x7fd16c032470 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fd17c0096a0 tx=0x7fd17c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:59:55.076 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:55.076+0000 7fd176ffd700 1 -- 172.21.15.155:0/2053170884 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd1700095f0 con 0x7fd1800f6510 2024-02-10T08:59:55.281 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:55.280+0000 7fd188482700 1 -- 172.21.15.155:0/2053170884 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd180108210 con 0x7fd1800f6510 2024-02-10T08:59:55.281 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:55.281+0000 7fd176ffd700 1 -- 172.21.15.155:0/2053170884 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd17001cb10 con 0x7fd1800f6510 2024-02-10T08:59:55.282 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:59:55.282 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T08:59:55.284 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:55.282+0000 7fd188482700 1 -- 172.21.15.155:0/2053170884 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fd16c02ffb0 msgr2=0x7fd16c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:59:55.284 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:55.282+0000 7fd188482700 1 --2- 172.21.15.155:0/2053170884 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fd16c02ffb0 0x7fd16c032470 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fd17c0096a0 tx=0x7fd17c006b40).stop 2024-02-10T08:59:55.284 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:55.282+0000 7fd188482700 1 -- 172.21.15.155:0/2053170884 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1800f6510 msgr2=0x7fd1801067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:59:55.284 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:55.282+0000 7fd188482700 1 --2- 172.21.15.155:0/2053170884 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1800f6510 0x7fd1801067e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fd1700089f0 tx=0x7fd17000b040).stop 2024-02-10T08:59:55.284 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:55.282+0000 7fd188482700 1 -- 172.21.15.155:0/2053170884 shutdown_connections 2024-02-10T08:59:55.284 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:55.282+0000 7fd188482700 1 --2- 172.21.15.155:0/2053170884 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fd16c02ffb0 0x7fd16c032470 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:59:55.284 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:55.282+0000 7fd188482700 1 --2- 172.21.15.155:0/2053170884 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd1800f6510 0x7fd1801067e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:59:55.284 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:55.282+0000 7fd188482700 1 -- 172.21.15.155:0/2053170884 >> 172.21.15.155:0/2053170884 conn(0x7fd1800f1c70 msgr2=0x7fd1800f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:59:55.284 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:55.282+0000 7fd188482700 1 -- 172.21.15.155:0/2053170884 shutdown_connections 2024-02-10T08:59:55.284 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:55.282+0000 7fd188482700 1 -- 172.21.15.155:0/2053170884 wait complete. 2024-02-10T08:59:55.284 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T08:59:55.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:59:55 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T08:59:54.964448+0000 mgr.smithi077.dxntvw (mgr.14162) 35 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:59:56.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:59:56 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:59:55.282164+0000 mon.smithi077 (mon.0) 126 : audit [DBG] from='client.? 172.21.15.155:0/2053170884' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T08:59:56.899 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T08:59:56.900 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T08:59:57.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:59:57 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T08:59:56.964751+0000 mgr.smithi077.dxntvw (mgr.14162) 36 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:59:58.809 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:58.809+0000 7f0f666a2700 1 -- 172.21.15.155:0/4017579592 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f600f62a0 msgr2=0x7f0f600f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:59:58.809 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:58.809+0000 7f0f666a2700 1 --2- 172.21.15.155:0/4017579592 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f600f62a0 0x7f0f600f66c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f0f5c009fb0 tx=0x7f0f5c009b30).stop 2024-02-10T08:59:58.809 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:58.809+0000 7f0f666a2700 1 -- 172.21.15.155:0/4017579592 shutdown_connections 2024-02-10T08:59:58.809 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:58.809+0000 7f0f666a2700 1 --2- 172.21.15.155:0/4017579592 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f600f62a0 0x7f0f600f66c0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:59:58.809 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:58.809+0000 7f0f666a2700 1 -- 172.21.15.155:0/4017579592 >> 172.21.15.155:0/4017579592 conn(0x7f0f600f1aa0 msgr2=0x7f0f600f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:59:58.809 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:58.809+0000 7f0f666a2700 1 -- 172.21.15.155:0/4017579592 shutdown_connections 2024-02-10T08:59:58.809 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:58.809+0000 7f0f666a2700 1 -- 172.21.15.155:0/4017579592 wait complete. 2024-02-10T08:59:58.811 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:58.809+0000 7f0f666a2700 1 Processor -- start 2024-02-10T08:59:58.811 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:58.810+0000 7f0f666a2700 1 -- start start 2024-02-10T08:59:58.811 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:58.810+0000 7f0f666a2700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f600f62a0 0x7f0f601065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:59:58.811 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:58.810+0000 7f0f666a2700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f60106ac0 con 0x7f0f600f62a0 2024-02-10T08:59:58.811 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:58.810+0000 7f0f656a0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f600f62a0 0x7f0f601065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:59:58.811 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:58.810+0000 7f0f656a0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f600f62a0 0x7f0f601065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:56310/0 (socket says 172.21.15.155:56310) 2024-02-10T08:59:58.812 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:58.810+0000 7f0f656a0700 1 -- 172.21.15.155:0/3120101206 learned_addr learned my addr 172.21.15.155:0/3120101206 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T08:59:58.812 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:58.810+0000 7f0f656a0700 1 -- 172.21.15.155:0/3120101206 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f5c005040 con 0x7f0f600f62a0 2024-02-10T08:59:58.812 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:58.810+0000 7f0f656a0700 1 --2- 172.21.15.155:0/3120101206 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f600f62a0 0x7f0f601065d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f0f5c0089f0 tx=0x7f0f5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:59:58.812 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:58.811+0000 7f0f51ffb700 1 -- 172.21.15.155:0/3120101206 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f5c009320 con 0x7f0f600f62a0 2024-02-10T08:59:58.812 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:58.811+0000 7f0f666a2700 1 -- 172.21.15.155:0/3120101206 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f60106cc0 con 0x7f0f600f62a0 2024-02-10T08:59:58.812 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:58.811+0000 7f0f51ffb700 1 -- 172.21.15.155:0/3120101206 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0f5c013380 con 0x7f0f600f62a0 2024-02-10T08:59:58.812 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:58.811+0000 7f0f51ffb700 1 -- 172.21.15.155:0/3120101206 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f5c014e20 con 0x7f0f600f62a0 2024-02-10T08:59:58.812 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:58.811+0000 7f0f666a2700 1 -- 172.21.15.155:0/3120101206 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f60107150 con 0x7f0f600f62a0 2024-02-10T08:59:58.814 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:58.812+0000 7f0f51ffb700 1 -- 172.21.15.155:0/3120101206 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0f5c0095f0 con 0x7f0f600f62a0 2024-02-10T08:59:58.814 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:58.812+0000 7f0f666a2700 1 -- 172.21.15.155:0/3120101206 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f60045ad0 con 0x7f0f600f62a0 2024-02-10T08:59:58.814 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:58.812+0000 7f0f51ffb700 1 --2- 172.21.15.155:0/3120101206 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f0f4c02ff10 0x7f0f4c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T08:59:58.814 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:58.812+0000 7f0f51ffb700 1 -- 172.21.15.155:0/3120101206 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0f5c040e90 con 0x7f0f600f62a0 2024-02-10T08:59:58.814 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:58.812+0000 7f0f64e9f700 1 --2- 172.21.15.155:0/3120101206 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f0f4c02ff10 0x7f0f4c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T08:59:58.814 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:58.812+0000 7f0f64e9f700 1 --2- 172.21.15.155:0/3120101206 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f0f4c02ff10 0x7f0f4c0323d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f0f540096a0 tx=0x7f0f54006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T08:59:58.816 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:58.816+0000 7f0f51ffb700 1 -- 172.21.15.155:0/3120101206 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0f5c014420 con 0x7f0f600f62a0 2024-02-10T08:59:59.022 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:59.022+0000 7f0f666a2700 1 -- 172.21.15.155:0/3120101206 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0f60108000 con 0x7f0f600f62a0 2024-02-10T08:59:59.022 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:59.022+0000 7f0f51ffb700 1 -- 172.21.15.155:0/3120101206 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0f5c01d980 con 0x7f0f600f62a0 2024-02-10T08:59:59.022 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T08:59:59.023 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T08:59:59.024 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:59.024+0000 7f0f666a2700 1 -- 172.21.15.155:0/3120101206 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f0f4c02ff10 msgr2=0x7f0f4c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:59:59.025 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:59.024+0000 7f0f666a2700 1 --2- 172.21.15.155:0/3120101206 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f0f4c02ff10 0x7f0f4c0323d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f0f540096a0 tx=0x7f0f54006b40).stop 2024-02-10T08:59:59.025 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:59.024+0000 7f0f666a2700 1 -- 172.21.15.155:0/3120101206 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f600f62a0 msgr2=0x7f0f601065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T08:59:59.025 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:59.024+0000 7f0f666a2700 1 --2- 172.21.15.155:0/3120101206 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f600f62a0 0x7f0f601065d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f0f5c0089f0 tx=0x7f0f5c00b040).stop 2024-02-10T08:59:59.025 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:59.024+0000 7f0f666a2700 1 -- 172.21.15.155:0/3120101206 shutdown_connections 2024-02-10T08:59:59.025 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:59.024+0000 7f0f666a2700 1 --2- 172.21.15.155:0/3120101206 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f0f4c02ff10 0x7f0f4c0323d0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:59:59.025 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:59.024+0000 7f0f666a2700 1 --2- 172.21.15.155:0/3120101206 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0f600f62a0 0x7f0f601065d0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T08:59:59.025 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:59.024+0000 7f0f666a2700 1 -- 172.21.15.155:0/3120101206 >> 172.21.15.155:0/3120101206 conn(0x7f0f600f1aa0 msgr2=0x7f0f600f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T08:59:59.025 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:59.024+0000 7f0f666a2700 1 -- 172.21.15.155:0/3120101206 shutdown_connections 2024-02-10T08:59:59.025 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T08:59:59.024+0000 7f0f666a2700 1 -- 172.21.15.155:0/3120101206 wait complete. 2024-02-10T08:59:59.025 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T08:59:59.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:59:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T08:59:58.965118+0000 mgr.smithi077.dxntvw (mgr.14162) 37 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T08:59:59.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 08:59:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T08:59:59.022876+0000 mon.smithi077 (mon.0) 127 : audit [DBG] from='client.? 172.21.15.155:0/3120101206' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:00:00.714 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:00:00.715 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:00:01.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:00:01 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:00:00.965479+0000 mgr.smithi077.dxntvw (mgr.14162) 38 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:00:02.828 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:02.812+0000 7f8549da4700 1 -- 172.21.15.155:0/119128757 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8544060200 msgr2=0x7f85440fb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:00:02.829 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:02.812+0000 7f8549da4700 1 --2- 172.21.15.155:0/119128757 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8544060200 0x7f85440fb570 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f8534009fb0 tx=0x7f8534009b30).stop 2024-02-10T09:00:02.829 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:02.812+0000 7f8549da4700 1 -- 172.21.15.155:0/119128757 shutdown_connections 2024-02-10T09:00:02.829 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:02.812+0000 7f8549da4700 1 --2- 172.21.15.155:0/119128757 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8544060200 0x7f85440fb570 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:00:02.829 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:02.812+0000 7f8549da4700 1 -- 172.21.15.155:0/119128757 >> 172.21.15.155:0/119128757 conn(0x7f85440f0b70 msgr2=0x7f85440f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:00:02.829 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:02.813+0000 7f8549da4700 1 -- 172.21.15.155:0/119128757 shutdown_connections 2024-02-10T09:00:02.829 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:02.813+0000 7f8549da4700 1 -- 172.21.15.155:0/119128757 wait complete. 2024-02-10T09:00:02.829 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:02.813+0000 7f8549da4700 1 Processor -- start 2024-02-10T09:00:02.829 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:02.813+0000 7f8549da4700 1 -- start start 2024-02-10T09:00:02.829 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:02.813+0000 7f8549da4700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8544060200 0x7f8544108970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:00:02.829 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:02.813+0000 7f8549da4700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8544108e60 con 0x7f8544060200 2024-02-10T09:00:02.830 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:02.814+0000 7f85437fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8544060200 0x7f8544108970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:00:02.830 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:02.814+0000 7f85437fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8544060200 0x7f8544108970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:33864/0 (socket says 172.21.15.155:33864) 2024-02-10T09:00:02.830 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:02.814+0000 7f85437fe700 1 -- 172.21.15.155:0/494927084 learned_addr learned my addr 172.21.15.155:0/494927084 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:00:02.830 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:02.814+0000 7f85437fe700 1 -- 172.21.15.155:0/494927084 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8534005040 con 0x7f8544060200 2024-02-10T09:00:02.830 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:02.814+0000 7f85437fe700 1 --2- 172.21.15.155:0/494927084 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8544060200 0x7f8544108970 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f85340089f0 tx=0x7f853400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:00:02.830 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:02.814+0000 7f853bfff700 1 -- 172.21.15.155:0/494927084 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8534009320 con 0x7f8544060200 2024-02-10T09:00:02.830 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:02.814+0000 7f8549da4700 1 -- 172.21.15.155:0/494927084 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8544109060 con 0x7f8544060200 2024-02-10T09:00:02.830 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:02.814+0000 7f853bfff700 1 -- 172.21.15.155:0/494927084 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8534013380 con 0x7f8544060200 2024-02-10T09:00:02.830 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:02.814+0000 7f853bfff700 1 -- 172.21.15.155:0/494927084 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8534014e20 con 0x7f8544060200 2024-02-10T09:00:02.830 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:02.814+0000 7f8549da4700 1 -- 172.21.15.155:0/494927084 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85441094f0 con 0x7f8544060200 2024-02-10T09:00:02.830 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:02.815+0000 7f853bfff700 1 -- 172.21.15.155:0/494927084 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f85340095f0 con 0x7f8544060200 2024-02-10T09:00:02.830 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:02.815+0000 7f8549da4700 1 -- 172.21.15.155:0/494927084 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8544045ad0 con 0x7f8544060200 2024-02-10T09:00:02.830 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:02.815+0000 7f853bfff700 1 --2- 172.21.15.155:0/494927084 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f853002ff10 0x7f85300323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:00:02.831 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:02.815+0000 7f853bfff700 1 -- 172.21.15.155:0/494927084 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8534040e90 con 0x7f8544060200 2024-02-10T09:00:02.831 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:02.815+0000 7f8542ffd700 1 --2- 172.21.15.155:0/494927084 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f853002ff10 0x7f85300323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:00:02.831 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:02.816+0000 7f8542ffd700 1 --2- 172.21.15.155:0/494927084 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f853002ff10 0x7f85300323d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f852c0096a0 tx=0x7f852c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:00:02.831 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:02.820+0000 7f853bfff700 1 -- 172.21.15.155:0/494927084 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8534014420 con 0x7f8544060200 2024-02-10T09:00:03.031 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:03.031+0000 7f8549da4700 1 -- 172.21.15.155:0/494927084 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f854410a3a0 con 0x7f8544060200 2024-02-10T09:00:03.031 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:03.032+0000 7f853bfff700 1 -- 172.21.15.155:0/494927084 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f853401d980 con 0x7f8544060200 2024-02-10T09:00:03.032 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:00:03.032 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:00:03.033 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:03.033+0000 7f8549da4700 1 -- 172.21.15.155:0/494927084 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f853002ff10 msgr2=0x7f85300323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:00:03.033 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:03.033+0000 7f8549da4700 1 --2- 172.21.15.155:0/494927084 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f853002ff10 0x7f85300323d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f852c0096a0 tx=0x7f852c006b40).stop 2024-02-10T09:00:03.034 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:03.033+0000 7f8549da4700 1 -- 172.21.15.155:0/494927084 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8544060200 msgr2=0x7f8544108970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:00:03.034 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:03.033+0000 7f8549da4700 1 --2- 172.21.15.155:0/494927084 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8544060200 0x7f8544108970 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f85340089f0 tx=0x7f853400b040).stop 2024-02-10T09:00:03.034 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:03.033+0000 7f8549da4700 1 -- 172.21.15.155:0/494927084 shutdown_connections 2024-02-10T09:00:03.034 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:03.033+0000 7f8549da4700 1 --2- 172.21.15.155:0/494927084 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f853002ff10 0x7f85300323d0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:00:03.034 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:03.033+0000 7f8549da4700 1 --2- 172.21.15.155:0/494927084 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8544060200 0x7f8544108970 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:00:03.034 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:03.033+0000 7f8549da4700 1 -- 172.21.15.155:0/494927084 >> 172.21.15.155:0/494927084 conn(0x7f85440f0b70 msgr2=0x7f85440f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:00:03.034 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:03.033+0000 7f8549da4700 1 -- 172.21.15.155:0/494927084 shutdown_connections 2024-02-10T09:00:03.034 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:03.033+0000 7f8549da4700 1 -- 172.21.15.155:0/494927084 wait complete. 2024-02-10T09:00:03.034 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:00:04.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:00:03 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:00:02.965848+0000 mgr.smithi077.dxntvw (mgr.14162) 39 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:00:04.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:00:03 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:00:03.032028+0000 mon.smithi077 (mon.0) 128 : audit [DBG] from='client.? 172.21.15.155:0/494927084' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:00:04.662 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:00:04.663 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:00:05.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:00:05 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:00:04.966217+0000 mgr.smithi077.dxntvw (mgr.14162) 40 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:00:06.782 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:06.731+0000 7fee3c6ef700 1 -- 172.21.15.155:0/319357100 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee340f6510 msgr2=0x7fee340f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:00:06.782 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:06.731+0000 7fee3c6ef700 1 --2- 172.21.15.155:0/319357100 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee340f6510 0x7fee340f6930 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7fee24009fb0 tx=0x7fee24009b30).stop 2024-02-10T09:00:06.783 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:06.731+0000 7fee3c6ef700 1 -- 172.21.15.155:0/319357100 shutdown_connections 2024-02-10T09:00:06.783 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:06.731+0000 7fee3c6ef700 1 --2- 172.21.15.155:0/319357100 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee340f6510 0x7fee340f6930 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:00:06.783 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:06.731+0000 7fee3c6ef700 1 -- 172.21.15.155:0/319357100 >> 172.21.15.155:0/319357100 conn(0x7fee340f1c70 msgr2=0x7fee340f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:00:06.783 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:06.731+0000 7fee3c6ef700 1 -- 172.21.15.155:0/319357100 shutdown_connections 2024-02-10T09:00:06.783 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:06.731+0000 7fee3c6ef700 1 -- 172.21.15.155:0/319357100 wait complete. 2024-02-10T09:00:06.783 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:06.731+0000 7fee3c6ef700 1 Processor -- start 2024-02-10T09:00:06.783 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:06.731+0000 7fee3c6ef700 1 -- start start 2024-02-10T09:00:06.783 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:06.732+0000 7fee3c6ef700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee340f6510 0x7fee341067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:00:06.783 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:06.732+0000 7fee3c6ef700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee34106cd0 con 0x7fee340f6510 2024-02-10T09:00:06.783 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:06.732+0000 7fee3a48b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee340f6510 0x7fee341067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:00:06.783 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:06.732+0000 7fee3a48b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee340f6510 0x7fee341067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:33886/0 (socket says 172.21.15.155:33886) 2024-02-10T09:00:06.783 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:06.732+0000 7fee3a48b700 1 -- 172.21.15.155:0/2686771944 learned_addr learned my addr 172.21.15.155:0/2686771944 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:00:06.783 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:06.732+0000 7fee3a48b700 1 -- 172.21.15.155:0/2686771944 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee24005040 con 0x7fee340f6510 2024-02-10T09:00:06.784 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:06.732+0000 7fee3a48b700 1 --2- 172.21.15.155:0/2686771944 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee340f6510 0x7fee341067e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fee240089f0 tx=0x7fee2400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:00:06.784 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:06.732+0000 7fee2affd700 1 -- 172.21.15.155:0/2686771944 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee24009320 con 0x7fee340f6510 2024-02-10T09:00:06.784 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:06.732+0000 7fee3c6ef700 1 -- 172.21.15.155:0/2686771944 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee34106ed0 con 0x7fee340f6510 2024-02-10T09:00:06.784 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:06.732+0000 7fee2affd700 1 -- 172.21.15.155:0/2686771944 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fee24013380 con 0x7fee340f6510 2024-02-10T09:00:06.784 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:06.732+0000 7fee2affd700 1 -- 172.21.15.155:0/2686771944 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee2401ed80 con 0x7fee340f6510 2024-02-10T09:00:06.784 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:06.732+0000 7fee3c6ef700 1 -- 172.21.15.155:0/2686771944 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee34107360 con 0x7fee340f6510 2024-02-10T09:00:06.784 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:06.733+0000 7fee2affd700 1 -- 172.21.15.155:0/2686771944 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fee2401e420 con 0x7fee340f6510 2024-02-10T09:00:06.784 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:06.733+0000 7fee3c6ef700 1 -- 172.21.15.155:0/2686771944 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee34045ad0 con 0x7fee340f6510 2024-02-10T09:00:06.784 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:06.733+0000 7fee2affd700 1 --2- 172.21.15.155:0/2686771944 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fee2002ff60 0x7fee20032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:00:06.784 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:06.733+0000 7fee2affd700 1 -- 172.21.15.155:0/2686771944 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fee24041130 con 0x7fee340f6510 2024-02-10T09:00:06.784 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:06.734+0000 7fee39c8a700 1 --2- 172.21.15.155:0/2686771944 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fee2002ff60 0x7fee20032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:00:06.784 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:06.734+0000 7fee39c8a700 1 --2- 172.21.15.155:0/2686771944 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fee2002ff60 0x7fee20032420 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fee300096a0 tx=0x7fee30006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:00:06.784 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:06.738+0000 7fee2affd700 1 -- 172.21.15.155:0/2686771944 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fee240095f0 con 0x7fee340f6510 2024-02-10T09:00:06.946 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:06.946+0000 7fee3c6ef700 1 -- 172.21.15.155:0/2686771944 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fee34108210 con 0x7fee340f6510 2024-02-10T09:00:06.946 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:06.946+0000 7fee2affd700 1 -- 172.21.15.155:0/2686771944 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fee2402c2d0 con 0x7fee340f6510 2024-02-10T09:00:06.947 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:00:06.947 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:00:06.948 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:06.948+0000 7fee3c6ef700 1 -- 172.21.15.155:0/2686771944 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fee2002ff60 msgr2=0x7fee20032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:00:06.948 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:06.948+0000 7fee3c6ef700 1 --2- 172.21.15.155:0/2686771944 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fee2002ff60 0x7fee20032420 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fee300096a0 tx=0x7fee30006b40).stop 2024-02-10T09:00:06.948 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:06.948+0000 7fee3c6ef700 1 -- 172.21.15.155:0/2686771944 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee340f6510 msgr2=0x7fee341067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:00:06.948 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:06.948+0000 7fee3c6ef700 1 --2- 172.21.15.155:0/2686771944 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee340f6510 0x7fee341067e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fee240089f0 tx=0x7fee2400b040).stop 2024-02-10T09:00:06.948 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:06.948+0000 7fee3c6ef700 1 -- 172.21.15.155:0/2686771944 shutdown_connections 2024-02-10T09:00:06.949 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:06.948+0000 7fee3c6ef700 1 --2- 172.21.15.155:0/2686771944 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fee2002ff60 0x7fee20032420 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:00:06.949 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:06.948+0000 7fee3c6ef700 1 --2- 172.21.15.155:0/2686771944 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee340f6510 0x7fee341067e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:00:06.949 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:06.948+0000 7fee3c6ef700 1 -- 172.21.15.155:0/2686771944 >> 172.21.15.155:0/2686771944 conn(0x7fee340f1c70 msgr2=0x7fee340f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:00:06.949 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:06.948+0000 7fee3c6ef700 1 -- 172.21.15.155:0/2686771944 shutdown_connections 2024-02-10T09:00:06.949 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:06.948+0000 7fee3c6ef700 1 -- 172.21.15.155:0/2686771944 wait complete. 2024-02-10T09:00:06.949 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:00:07.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:00:06 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:00:06.946696+0000 mon.smithi077 (mon.0) 129 : audit [DBG] from='client.? 172.21.15.155:0/2686771944' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:00:07.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:00:06 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:00:06.966491+0000 mgr.smithi077.dxntvw (mgr.14162) 41 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:00:08.672 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:00:08.675 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:00:09.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:00:09 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:00:08.966842+0000 mgr.smithi077.dxntvw (mgr.14162) 42 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:00:10.891 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:10.889+0000 7f99ddb69700 1 -- 172.21.15.155:0/3714904543 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99d80f6510 msgr2=0x7f99d80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:00:10.891 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:10.890+0000 7f99ddb69700 1 --2- 172.21.15.155:0/3714904543 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99d80f6510 0x7f99d80f6930 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f99cc009fb0 tx=0x7f99cc009b30).stop 2024-02-10T09:00:10.891 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:10.890+0000 7f99ddb69700 1 -- 172.21.15.155:0/3714904543 shutdown_connections 2024-02-10T09:00:10.892 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:10.890+0000 7f99ddb69700 1 --2- 172.21.15.155:0/3714904543 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99d80f6510 0x7f99d80f6930 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:00:10.892 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:10.890+0000 7f99ddb69700 1 -- 172.21.15.155:0/3714904543 >> 172.21.15.155:0/3714904543 conn(0x7f99d80f1c70 msgr2=0x7f99d80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:00:10.892 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:10.890+0000 7f99ddb69700 1 -- 172.21.15.155:0/3714904543 shutdown_connections 2024-02-10T09:00:10.892 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:10.890+0000 7f99ddb69700 1 -- 172.21.15.155:0/3714904543 wait complete. 2024-02-10T09:00:10.892 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:10.890+0000 7f99ddb69700 1 Processor -- start 2024-02-10T09:00:10.892 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:10.890+0000 7f99ddb69700 1 -- start start 2024-02-10T09:00:10.892 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:10.890+0000 7f99ddb69700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99d80f6510 0x7f99d81067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:00:10.892 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:10.890+0000 7f99ddb69700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99d8106cd0 con 0x7f99d80f6510 2024-02-10T09:00:10.892 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:10.891+0000 7f99d77fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99d80f6510 0x7f99d81067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:00:10.892 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:10.891+0000 7f99d77fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99d80f6510 0x7f99d81067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:33900/0 (socket says 172.21.15.155:33900) 2024-02-10T09:00:10.892 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:10.891+0000 7f99d77fe700 1 -- 172.21.15.155:0/1724918651 learned_addr learned my addr 172.21.15.155:0/1724918651 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:00:10.892 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:10.891+0000 7f99d77fe700 1 -- 172.21.15.155:0/1724918651 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99cc005040 con 0x7f99d80f6510 2024-02-10T09:00:10.892 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:10.891+0000 7f99d77fe700 1 --2- 172.21.15.155:0/1724918651 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99d80f6510 0x7f99d81067e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f99cc005010 tx=0x7f99cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:00:10.893 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:10.891+0000 7f99bffff700 1 -- 172.21.15.155:0/1724918651 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f99cc009320 con 0x7f99d80f6510 2024-02-10T09:00:10.893 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:10.891+0000 7f99bffff700 1 -- 172.21.15.155:0/1724918651 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f99cc013380 con 0x7f99d80f6510 2024-02-10T09:00:10.893 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:10.891+0000 7f99ddb69700 1 -- 172.21.15.155:0/1724918651 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99d8106ed0 con 0x7f99d80f6510 2024-02-10T09:00:10.893 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:10.891+0000 7f99bffff700 1 -- 172.21.15.155:0/1724918651 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f99cc014e50 con 0x7f99d80f6510 2024-02-10T09:00:10.893 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:10.891+0000 7f99ddb69700 1 -- 172.21.15.155:0/1724918651 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99d8107360 con 0x7f99d80f6510 2024-02-10T09:00:10.894 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:10.892+0000 7f99bffff700 1 -- 172.21.15.155:0/1724918651 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f99cc0263a0 con 0x7f99d80f6510 2024-02-10T09:00:10.894 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:10.892+0000 7f99bffff700 1 --2- 172.21.15.155:0/1724918651 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f99c002ff10 0x7f99c00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:00:10.894 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:10.892+0000 7f99bffff700 1 -- 172.21.15.155:0/1724918651 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f99cc040f90 con 0x7f99d80f6510 2024-02-10T09:00:10.894 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:10.892+0000 7f99ddb69700 1 -- 172.21.15.155:0/1724918651 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99d8045ad0 con 0x7f99d80f6510 2024-02-10T09:00:10.894 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:10.893+0000 7f99d6ffd700 1 --2- 172.21.15.155:0/1724918651 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f99c002ff10 0x7f99c00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:00:10.894 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:10.893+0000 7f99d6ffd700 1 --2- 172.21.15.155:0/1724918651 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f99c002ff10 0x7f99c00323d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f99c80096a0 tx=0x7f99c8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:00:10.897 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:10.897+0000 7f99bffff700 1 -- 172.21.15.155:0/1724918651 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f99cc012920 con 0x7f99d80f6510 2024-02-10T09:00:11.104 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:11.104+0000 7f99ddb69700 1 -- 172.21.15.155:0/1724918651 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f99d8108210 con 0x7f99d80f6510 2024-02-10T09:00:11.105 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:11.105+0000 7f99bffff700 1 -- 172.21.15.155:0/1724918651 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f99cc02b020 con 0x7f99d80f6510 2024-02-10T09:00:11.105 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:00:11.105 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:00:11.107 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:11.106+0000 7f99ddb69700 1 -- 172.21.15.155:0/1724918651 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f99c002ff10 msgr2=0x7f99c00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:00:11.107 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:11.106+0000 7f99ddb69700 1 --2- 172.21.15.155:0/1724918651 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f99c002ff10 0x7f99c00323d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f99c80096a0 tx=0x7f99c8006b40).stop 2024-02-10T09:00:11.107 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:11.106+0000 7f99ddb69700 1 -- 172.21.15.155:0/1724918651 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99d80f6510 msgr2=0x7f99d81067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:00:11.107 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:11.106+0000 7f99ddb69700 1 --2- 172.21.15.155:0/1724918651 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99d80f6510 0x7f99d81067e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f99cc005010 tx=0x7f99cc00b040).stop 2024-02-10T09:00:11.107 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:11.107+0000 7f99ddb69700 1 -- 172.21.15.155:0/1724918651 shutdown_connections 2024-02-10T09:00:11.107 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:11.107+0000 7f99ddb69700 1 --2- 172.21.15.155:0/1724918651 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f99c002ff10 0x7f99c00323d0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:00:11.107 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:11.107+0000 7f99ddb69700 1 --2- 172.21.15.155:0/1724918651 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99d80f6510 0x7f99d81067e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:00:11.107 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:11.107+0000 7f99ddb69700 1 -- 172.21.15.155:0/1724918651 >> 172.21.15.155:0/1724918651 conn(0x7f99d80f1c70 msgr2=0x7f99d80f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:00:11.107 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:11.107+0000 7f99ddb69700 1 -- 172.21.15.155:0/1724918651 shutdown_connections 2024-02-10T09:00:11.107 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:11.107+0000 7f99ddb69700 1 -- 172.21.15.155:0/1724918651 wait complete. 2024-02-10T09:00:11.108 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:00:11.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:00:11 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:00:10.967224+0000 mgr.smithi077.dxntvw (mgr.14162) 43 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:00:12.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:00:12 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:00:11.105354+0000 mon.smithi077 (mon.0) 130 : audit [DBG] from='client.? 172.21.15.155:0/1724918651' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:00:12.897 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:00:12.897 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:00:14.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:00:13 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:00:12.967520+0000 mgr.smithi077.dxntvw (mgr.14162) 44 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:00:15.083 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:15.081+0000 7f80907a6700 1 -- 172.21.15.155:0/4100785959 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80880f5330 msgr2=0x7f80880f5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:00:15.084 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:15.081+0000 7f80907a6700 1 --2- 172.21.15.155:0/4100785959 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80880f5330 0x7f80880f5750 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f8078009fb0 tx=0x7f8078009b30).stop 2024-02-10T09:00:15.084 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:15.081+0000 7f80907a6700 1 -- 172.21.15.155:0/4100785959 shutdown_connections 2024-02-10T09:00:15.084 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:15.081+0000 7f80907a6700 1 --2- 172.21.15.155:0/4100785959 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80880f5330 0x7f80880f5750 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:00:15.084 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:15.081+0000 7f80907a6700 1 -- 172.21.15.155:0/4100785959 >> 172.21.15.155:0/4100785959 conn(0x7f80880f0a90 msgr2=0x7f80880f2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:00:15.084 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:15.081+0000 7f80907a6700 1 -- 172.21.15.155:0/4100785959 shutdown_connections 2024-02-10T09:00:15.084 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:15.081+0000 7f80907a6700 1 -- 172.21.15.155:0/4100785959 wait complete. 2024-02-10T09:00:15.084 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:15.081+0000 7f80907a6700 1 Processor -- start 2024-02-10T09:00:15.084 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:15.082+0000 7f80907a6700 1 -- start start 2024-02-10T09:00:15.084 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:15.082+0000 7f80907a6700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80880f5330 0x7f8088109780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:00:15.084 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:15.082+0000 7f80907a6700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8088060220 con 0x7f80880f5330 2024-02-10T09:00:15.084 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:15.082+0000 7f808e542700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80880f5330 0x7f8088109780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:00:15.084 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:15.082+0000 7f808e542700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80880f5330 0x7f8088109780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:54476/0 (socket says 172.21.15.155:54476) 2024-02-10T09:00:15.085 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:15.082+0000 7f808e542700 1 -- 172.21.15.155:0/3154618025 learned_addr learned my addr 172.21.15.155:0/3154618025 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:00:15.085 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:15.082+0000 7f808e542700 1 -- 172.21.15.155:0/3154618025 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8078005040 con 0x7f80880f5330 2024-02-10T09:00:15.085 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:15.082+0000 7f808e542700 1 --2- 172.21.15.155:0/3154618025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80880f5330 0x7f8088109780 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f8078005010 tx=0x7f807800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:00:15.085 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:15.083+0000 7f807effd700 1 -- 172.21.15.155:0/3154618025 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8078009320 con 0x7f80880f5330 2024-02-10T09:00:15.085 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:15.083+0000 7f80907a6700 1 -- 172.21.15.155:0/3154618025 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8088060420 con 0x7f80880f5330 2024-02-10T09:00:15.086 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:15.083+0000 7f80907a6700 1 -- 172.21.15.155:0/3154618025 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8088109e30 con 0x7f80880f5330 2024-02-10T09:00:15.086 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:15.083+0000 7f807effd700 1 -- 172.21.15.155:0/3154618025 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8078013380 con 0x7f80880f5330 2024-02-10T09:00:15.086 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:15.083+0000 7f807effd700 1 -- 172.21.15.155:0/3154618025 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8078014df0 con 0x7f80880f5330 2024-02-10T09:00:15.086 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:15.083+0000 7f80907a6700 1 -- 172.21.15.155:0/3154618025 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8088045ad0 con 0x7f80880f5330 2024-02-10T09:00:15.086 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:15.083+0000 7f807effd700 1 -- 172.21.15.155:0/3154618025 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8078014420 con 0x7f80880f5330 2024-02-10T09:00:15.086 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:15.084+0000 7f807effd700 1 --2- 172.21.15.155:0/3154618025 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f807402ff10 0x7f80740323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:00:15.086 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:15.084+0000 7f807effd700 1 -- 172.21.15.155:0/3154618025 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f80780411f0 con 0x7f80880f5330 2024-02-10T09:00:15.086 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:15.084+0000 7f808dd41700 1 --2- 172.21.15.155:0/3154618025 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f807402ff10 0x7f80740323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:00:15.086 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:15.084+0000 7f808dd41700 1 --2- 172.21.15.155:0/3154618025 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f807402ff10 0x7f80740323d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f80840096a0 tx=0x7f8084006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:00:15.089 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:15.088+0000 7f807effd700 1 -- 172.21.15.155:0/3154618025 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8078012b20 con 0x7f80880f5330 2024-02-10T09:00:15.295 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:15.294+0000 7f80907a6700 1 -- 172.21.15.155:0/3154618025 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f808805f270 con 0x7f80880f5330 2024-02-10T09:00:15.296 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:15.295+0000 7f807effd700 1 -- 172.21.15.155:0/3154618025 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8078012b20 con 0x7f80880f5330 2024-02-10T09:00:15.296 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:00:15.296 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:00:15.298 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:15.297+0000 7f80907a6700 1 -- 172.21.15.155:0/3154618025 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f807402ff10 msgr2=0x7f80740323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:00:15.298 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:15.297+0000 7f80907a6700 1 --2- 172.21.15.155:0/3154618025 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f807402ff10 0x7f80740323d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f80840096a0 tx=0x7f8084006b40).stop 2024-02-10T09:00:15.298 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:15.297+0000 7f80907a6700 1 -- 172.21.15.155:0/3154618025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80880f5330 msgr2=0x7f8088109780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:00:15.298 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:15.297+0000 7f80907a6700 1 --2- 172.21.15.155:0/3154618025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80880f5330 0x7f8088109780 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f8078005010 tx=0x7f807800b040).stop 2024-02-10T09:00:15.298 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:15.297+0000 7f80907a6700 1 -- 172.21.15.155:0/3154618025 shutdown_connections 2024-02-10T09:00:15.298 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:15.297+0000 7f80907a6700 1 --2- 172.21.15.155:0/3154618025 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f807402ff10 0x7f80740323d0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:00:15.298 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:15.297+0000 7f80907a6700 1 --2- 172.21.15.155:0/3154618025 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f80880f5330 0x7f8088109780 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:00:15.298 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:15.297+0000 7f80907a6700 1 -- 172.21.15.155:0/3154618025 >> 172.21.15.155:0/3154618025 conn(0x7f80880f0a90 msgr2=0x7f80880f1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:00:15.298 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:15.297+0000 7f80907a6700 1 -- 172.21.15.155:0/3154618025 shutdown_connections 2024-02-10T09:00:15.298 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:15.297+0000 7f80907a6700 1 -- 172.21.15.155:0/3154618025 wait complete. 2024-02-10T09:00:15.299 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:00:15.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:00:15 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:00:14.967782+0000 mgr.smithi077.dxntvw (mgr.14162) 45 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:00:16.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:00:16 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:00:15.296402+0000 mon.smithi077 (mon.0) 131 : audit [DBG] from='client.? 172.21.15.155:0/3154618025' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:00:17.049 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:00:17.049 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:00:17.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:00:17 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:00:16.968119+0000 mgr.smithi077.dxntvw (mgr.14162) 46 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:00:18.994 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:18.991+0000 7f7bf2c81700 1 -- 172.21.15.155:0/927818629 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7bec0f5410 msgr2=0x7f7bec0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:00:18.995 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:18.991+0000 7f7bf2c81700 1 --2- 172.21.15.155:0/927818629 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7bec0f5410 0x7f7bec0f5830 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f7be0009fb0 tx=0x7f7be0009b30).stop 2024-02-10T09:00:18.995 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:18.992+0000 7f7bf2c81700 1 -- 172.21.15.155:0/927818629 shutdown_connections 2024-02-10T09:00:18.995 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:18.992+0000 7f7bf2c81700 1 --2- 172.21.15.155:0/927818629 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7bec0f5410 0x7f7bec0f5830 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:00:18.995 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:18.992+0000 7f7bf2c81700 1 -- 172.21.15.155:0/927818629 >> 172.21.15.155:0/927818629 conn(0x7f7bec0f0b70 msgr2=0x7f7bec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:00:18.995 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:18.992+0000 7f7bf2c81700 1 -- 172.21.15.155:0/927818629 shutdown_connections 2024-02-10T09:00:18.995 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:18.992+0000 7f7bf2c81700 1 -- 172.21.15.155:0/927818629 wait complete. 2024-02-10T09:00:18.995 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:18.992+0000 7f7bf2c81700 1 Processor -- start 2024-02-10T09:00:18.995 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:18.992+0000 7f7bf2c81700 1 -- start start 2024-02-10T09:00:18.995 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:18.992+0000 7f7bf2c81700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7bec0f5410 0x7f7bec1094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:00:18.995 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:18.993+0000 7f7bf2c81700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bec060240 con 0x7f7bec0f5410 2024-02-10T09:00:18.996 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:18.993+0000 7f7bf0a1d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7bec0f5410 0x7f7bec1094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:00:18.996 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:18.993+0000 7f7bf0a1d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7bec0f5410 0x7f7bec1094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:54506/0 (socket says 172.21.15.155:54506) 2024-02-10T09:00:18.996 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:18.993+0000 7f7bf0a1d700 1 -- 172.21.15.155:0/1718776760 learned_addr learned my addr 172.21.15.155:0/1718776760 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:00:18.996 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:18.993+0000 7f7bf0a1d700 1 -- 172.21.15.155:0/1718776760 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7be0005040 con 0x7f7bec0f5410 2024-02-10T09:00:18.996 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:18.993+0000 7f7bf0a1d700 1 --2- 172.21.15.155:0/1718776760 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7bec0f5410 0x7f7bec1094e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f7be0005010 tx=0x7f7be000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:00:18.996 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:18.993+0000 7f7be97fa700 1 -- 172.21.15.155:0/1718776760 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7be0009320 con 0x7f7bec0f5410 2024-02-10T09:00:18.996 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:18.993+0000 7f7bf2c81700 1 -- 172.21.15.155:0/1718776760 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bec060440 con 0x7f7bec0f5410 2024-02-10T09:00:18.996 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:18.993+0000 7f7be97fa700 1 -- 172.21.15.155:0/1718776760 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7be0013380 con 0x7f7bec0f5410 2024-02-10T09:00:18.996 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:18.993+0000 7f7be97fa700 1 -- 172.21.15.155:0/1718776760 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7be0014e50 con 0x7f7bec0f5410 2024-02-10T09:00:18.996 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:18.993+0000 7f7bf2c81700 1 -- 172.21.15.155:0/1718776760 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bec109b90 con 0x7f7bec0f5410 2024-02-10T09:00:18.997 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:18.994+0000 7f7be97fa700 1 -- 172.21.15.155:0/1718776760 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7be0014420 con 0x7f7bec0f5410 2024-02-10T09:00:18.997 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:18.994+0000 7f7bf2c81700 1 -- 172.21.15.155:0/1718776760 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bec045ad0 con 0x7f7bec0f5410 2024-02-10T09:00:18.997 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:18.994+0000 7f7be97fa700 1 --2- 172.21.15.155:0/1718776760 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f7bdc02ff10 0x7f7bdc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:00:18.997 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:18.994+0000 7f7be97fa700 1 -- 172.21.15.155:0/1718776760 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7be0040f40 con 0x7f7bec0f5410 2024-02-10T09:00:18.997 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:18.995+0000 7f7bebfff700 1 --2- 172.21.15.155:0/1718776760 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f7bdc02ff10 0x7f7bdc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:00:18.997 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:18.995+0000 7f7bebfff700 1 --2- 172.21.15.155:0/1718776760 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f7bdc02ff10 0x7f7bdc0323d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f7bd80096a0 tx=0x7f7bd8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:00:19.000 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:18.999+0000 7f7be97fa700 1 -- 172.21.15.155:0/1718776760 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7be00095f0 con 0x7f7bec0f5410 2024-02-10T09:00:19.205 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:19.204+0000 7f7bf2c81700 1 -- 172.21.15.155:0/1718776760 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7bec10a3b0 con 0x7f7bec0f5410 2024-02-10T09:00:19.206 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:19.205+0000 7f7be97fa700 1 -- 172.21.15.155:0/1718776760 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7be001db70 con 0x7f7bec0f5410 2024-02-10T09:00:19.206 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:00:19.207 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:00:19.208 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:19.207+0000 7f7bf2c81700 1 -- 172.21.15.155:0/1718776760 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f7bdc02ff10 msgr2=0x7f7bdc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:00:19.208 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:19.207+0000 7f7bf2c81700 1 --2- 172.21.15.155:0/1718776760 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f7bdc02ff10 0x7f7bdc0323d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f7bd80096a0 tx=0x7f7bd8006b40).stop 2024-02-10T09:00:19.208 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:19.207+0000 7f7bf2c81700 1 -- 172.21.15.155:0/1718776760 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7bec0f5410 msgr2=0x7f7bec1094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:00:19.208 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:19.207+0000 7f7bf2c81700 1 --2- 172.21.15.155:0/1718776760 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7bec0f5410 0x7f7bec1094e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f7be0005010 tx=0x7f7be000b040).stop 2024-02-10T09:00:19.208 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:19.207+0000 7f7bf2c81700 1 -- 172.21.15.155:0/1718776760 shutdown_connections 2024-02-10T09:00:19.208 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:19.207+0000 7f7bf2c81700 1 --2- 172.21.15.155:0/1718776760 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f7bdc02ff10 0x7f7bdc0323d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:00:19.208 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:19.207+0000 7f7bf2c81700 1 --2- 172.21.15.155:0/1718776760 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7bec0f5410 0x7f7bec1094e0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:00:19.209 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:19.207+0000 7f7bf2c81700 1 -- 172.21.15.155:0/1718776760 >> 172.21.15.155:0/1718776760 conn(0x7f7bec0f0b70 msgr2=0x7f7bec0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:00:19.209 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:19.207+0000 7f7bf2c81700 1 -- 172.21.15.155:0/1718776760 shutdown_connections 2024-02-10T09:00:19.209 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:19.207+0000 7f7bf2c81700 1 -- 172.21.15.155:0/1718776760 wait complete. 2024-02-10T09:00:19.209 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:00:19.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:00:19 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:00:18.968446+0000 mgr.smithi077.dxntvw (mgr.14162) 47 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:00:20.398 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:00:20 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:00:19.206670+0000 mon.smithi077 (mon.0) 132 : audit [DBG] from='client.? 172.21.15.155:0/1718776760' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:00:20.988 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:00:20.988 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:00:21.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:00:21 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:00:20.968774+0000 mgr.smithi077.dxntvw (mgr.14162) 48 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:00:23.065 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:23.047+0000 7f502c03e700 1 -- 172.21.15.155:0/3829145991 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50240f5410 msgr2=0x7f50240f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:00:23.065 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:23.047+0000 7f502c03e700 1 --2- 172.21.15.155:0/3829145991 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50240f5410 0x7f50240f5830 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f5014009fb0 tx=0x7f5014009b30).stop 2024-02-10T09:00:23.065 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:23.047+0000 7f502c03e700 1 -- 172.21.15.155:0/3829145991 shutdown_connections 2024-02-10T09:00:23.065 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:23.047+0000 7f502c03e700 1 --2- 172.21.15.155:0/3829145991 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50240f5410 0x7f50240f5830 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:00:23.065 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:23.047+0000 7f502c03e700 1 -- 172.21.15.155:0/3829145991 >> 172.21.15.155:0/3829145991 conn(0x7f50240f0b70 msgr2=0x7f50240f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:00:23.065 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:23.047+0000 7f502c03e700 1 -- 172.21.15.155:0/3829145991 shutdown_connections 2024-02-10T09:00:23.066 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:23.047+0000 7f502c03e700 1 -- 172.21.15.155:0/3829145991 wait complete. 2024-02-10T09:00:23.066 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:23.048+0000 7f502c03e700 1 Processor -- start 2024-02-10T09:00:23.066 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:23.048+0000 7f502c03e700 1 -- start start 2024-02-10T09:00:23.066 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:23.048+0000 7f502c03e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50240f5410 0x7f502410ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:00:23.066 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:23.048+0000 7f502c03e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f502410d340 con 0x7f50240f5410 2024-02-10T09:00:23.066 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:23.048+0000 7f5029dda700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50240f5410 0x7f502410ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:00:23.066 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:23.048+0000 7f5029dda700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50240f5410 0x7f502410ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:45874/0 (socket says 172.21.15.155:45874) 2024-02-10T09:00:23.066 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:23.048+0000 7f5029dda700 1 -- 172.21.15.155:0/4283282497 learned_addr learned my addr 172.21.15.155:0/4283282497 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:00:23.066 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:23.049+0000 7f5029dda700 1 -- 172.21.15.155:0/4283282497 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5014005040 con 0x7f50240f5410 2024-02-10T09:00:23.066 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:23.049+0000 7f5029dda700 1 --2- 172.21.15.155:0/4283282497 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50240f5410 0x7f502410ce50 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f5014005010 tx=0x7f501400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:00:23.066 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:23.049+0000 7f501a7fc700 1 -- 172.21.15.155:0/4283282497 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5014009320 con 0x7f50240f5410 2024-02-10T09:00:23.066 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:23.049+0000 7f502c03e700 1 -- 172.21.15.155:0/4283282497 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f502410d540 con 0x7f50240f5410 2024-02-10T09:00:23.067 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:23.049+0000 7f501a7fc700 1 -- 172.21.15.155:0/4283282497 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5014013380 con 0x7f50240f5410 2024-02-10T09:00:23.067 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:23.049+0000 7f501a7fc700 1 -- 172.21.15.155:0/4283282497 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5014014e50 con 0x7f50240f5410 2024-02-10T09:00:23.067 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:23.049+0000 7f502c03e700 1 -- 172.21.15.155:0/4283282497 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f502410e150 con 0x7f50240f5410 2024-02-10T09:00:23.067 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:23.050+0000 7f501a7fc700 1 -- 172.21.15.155:0/4283282497 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5014014420 con 0x7f50240f5410 2024-02-10T09:00:23.067 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:23.050+0000 7f502c03e700 1 -- 172.21.15.155:0/4283282497 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f502410d9d0 con 0x7f50240f5410 2024-02-10T09:00:23.067 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:23.050+0000 7f501a7fc700 1 --2- 172.21.15.155:0/4283282497 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f501002ff10 0x7f50100323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:00:23.067 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:23.050+0000 7f501a7fc700 1 -- 172.21.15.155:0/4283282497 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5014040f90 con 0x7f50240f5410 2024-02-10T09:00:23.067 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:23.050+0000 7f50295d9700 1 --2- 172.21.15.155:0/4283282497 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f501002ff10 0x7f50100323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:00:23.067 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:23.051+0000 7f50295d9700 1 --2- 172.21.15.155:0/4283282497 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f501002ff10 0x7f50100323d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f50200096a0 tx=0x7f5020006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:00:23.067 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:23.055+0000 7f501a7fc700 1 -- 172.21.15.155:0/4283282497 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f502410d9d0 con 0x7f50240f5410 2024-02-10T09:00:23.264 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:23.262+0000 7f502c03e700 1 -- 172.21.15.155:0/4283282497 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f502410d9d0 con 0x7f50240f5410 2024-02-10T09:00:23.264 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:23.263+0000 7f501a7fc700 1 -- 172.21.15.155:0/4283282497 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f502410d9d0 con 0x7f50240f5410 2024-02-10T09:00:23.264 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:00:23.264 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:00:23.265 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:23.264+0000 7f502c03e700 1 -- 172.21.15.155:0/4283282497 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f501002ff10 msgr2=0x7f50100323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:00:23.265 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:23.264+0000 7f502c03e700 1 --2- 172.21.15.155:0/4283282497 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f501002ff10 0x7f50100323d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f50200096a0 tx=0x7f5020006b40).stop 2024-02-10T09:00:23.265 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:23.264+0000 7f502c03e700 1 -- 172.21.15.155:0/4283282497 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50240f5410 msgr2=0x7f502410ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:00:23.266 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:23.264+0000 7f502c03e700 1 --2- 172.21.15.155:0/4283282497 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50240f5410 0x7f502410ce50 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f5014005010 tx=0x7f501400b040).stop 2024-02-10T09:00:23.266 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:23.264+0000 7f502c03e700 1 -- 172.21.15.155:0/4283282497 shutdown_connections 2024-02-10T09:00:23.266 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:23.264+0000 7f502c03e700 1 --2- 172.21.15.155:0/4283282497 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f501002ff10 0x7f50100323d0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:00:23.266 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:23.264+0000 7f502c03e700 1 --2- 172.21.15.155:0/4283282497 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f50240f5410 0x7f502410ce50 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:00:23.266 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:23.264+0000 7f502c03e700 1 -- 172.21.15.155:0/4283282497 >> 172.21.15.155:0/4283282497 conn(0x7f50240f0b70 msgr2=0x7f502405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:00:23.266 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:23.264+0000 7f502c03e700 1 -- 172.21.15.155:0/4283282497 shutdown_connections 2024-02-10T09:00:23.266 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:23.264+0000 7f502c03e700 1 -- 172.21.15.155:0/4283282497 wait complete. 2024-02-10T09:00:23.266 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:00:24.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:00:23 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:00:22.969142+0000 mgr.smithi077.dxntvw (mgr.14162) 49 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:00:24.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:00:23 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:00:23.264174+0000 mon.smithi077 (mon.0) 133 : audit [DBG] from='client.? 172.21.15.155:0/4283282497' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:00:24.935 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:00:24.936 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:00:25.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:00:25 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:00:24.969447+0000 mgr.smithi077.dxntvw (mgr.14162) 50 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:00:26.901 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:26.896+0000 7f60f5cc8700 1 -- 172.21.15.155:0/3520317225 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60f00f69d0 msgr2=0x7f60f00f6df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:00:26.901 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:26.896+0000 7f60f5cc8700 1 --2- 172.21.15.155:0/3520317225 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60f00f69d0 0x7f60f00f6df0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f60d8009fb0 tx=0x7f60d8009b30).stop 2024-02-10T09:00:26.901 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:26.896+0000 7f60f5cc8700 1 -- 172.21.15.155:0/3520317225 shutdown_connections 2024-02-10T09:00:26.902 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:26.896+0000 7f60f5cc8700 1 --2- 172.21.15.155:0/3520317225 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60f00f69d0 0x7f60f00f6df0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:00:26.902 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:26.896+0000 7f60f5cc8700 1 -- 172.21.15.155:0/3520317225 >> 172.21.15.155:0/3520317225 conn(0x7f60f00f02e0 msgr2=0x7f60f00f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:00:26.902 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:26.896+0000 7f60f5cc8700 1 -- 172.21.15.155:0/3520317225 shutdown_connections 2024-02-10T09:00:26.902 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:26.896+0000 7f60f5cc8700 1 -- 172.21.15.155:0/3520317225 wait complete. 2024-02-10T09:00:26.902 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:26.897+0000 7f60f5cc8700 1 Processor -- start 2024-02-10T09:00:26.902 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:26.897+0000 7f60f5cc8700 1 -- start start 2024-02-10T09:00:26.902 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:26.897+0000 7f60f5cc8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60f00f69d0 0x7f60f01056a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:00:26.902 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:26.897+0000 7f60f5cc8700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60f0105b90 con 0x7f60f00f69d0 2024-02-10T09:00:26.902 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:26.897+0000 7f60ef7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60f00f69d0 0x7f60f01056a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:00:26.902 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:26.897+0000 7f60ef7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60f00f69d0 0x7f60f01056a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:45890/0 (socket says 172.21.15.155:45890) 2024-02-10T09:00:26.902 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:26.897+0000 7f60ef7fe700 1 -- 172.21.15.155:0/4203884813 learned_addr learned my addr 172.21.15.155:0/4203884813 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:00:26.902 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:26.898+0000 7f60ef7fe700 1 -- 172.21.15.155:0/4203884813 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60d8005040 con 0x7f60f00f69d0 2024-02-10T09:00:26.902 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:26.898+0000 7f60ef7fe700 1 --2- 172.21.15.155:0/4203884813 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60f00f69d0 0x7f60f01056a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f60d8005010 tx=0x7f60d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:00:26.902 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:26.898+0000 7f60e7fff700 1 -- 172.21.15.155:0/4203884813 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f60d8009320 con 0x7f60f00f69d0 2024-02-10T09:00:26.903 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:26.898+0000 7f60f5cc8700 1 -- 172.21.15.155:0/4203884813 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60f0105d90 con 0x7f60f00f69d0 2024-02-10T09:00:26.903 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:26.898+0000 7f60e7fff700 1 -- 172.21.15.155:0/4203884813 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f60d8013380 con 0x7f60f00f69d0 2024-02-10T09:00:26.903 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:26.898+0000 7f60e7fff700 1 -- 172.21.15.155:0/4203884813 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f60d8014e50 con 0x7f60f00f69d0 2024-02-10T09:00:26.903 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:26.898+0000 7f60f5cc8700 1 -- 172.21.15.155:0/4203884813 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60f0106220 con 0x7f60f00f69d0 2024-02-10T09:00:26.903 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:26.899+0000 7f60e7fff700 1 -- 172.21.15.155:0/4203884813 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f60d80095f0 con 0x7f60f00f69d0 2024-02-10T09:00:26.903 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:26.899+0000 7f60f5cc8700 1 -- 172.21.15.155:0/4203884813 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60f0045ad0 con 0x7f60f00f69d0 2024-02-10T09:00:26.903 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:26.899+0000 7f60e7fff700 1 --2- 172.21.15.155:0/4203884813 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f60dc02ff10 0x7f60dc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:00:26.903 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:26.899+0000 7f60e7fff700 1 -- 172.21.15.155:0/4203884813 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f60d8040e50 con 0x7f60f00f69d0 2024-02-10T09:00:26.903 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:26.899+0000 7f60eeffd700 1 --2- 172.21.15.155:0/4203884813 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f60dc02ff10 0x7f60dc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:00:26.903 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:26.900+0000 7f60eeffd700 1 --2- 172.21.15.155:0/4203884813 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f60dc02ff10 0x7f60dc0323d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f60e00096a0 tx=0x7f60e0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:00:26.904 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:26.904+0000 7f60e7fff700 1 -- 172.21.15.155:0/4203884813 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f60d801dba0 con 0x7f60f00f69d0 2024-02-10T09:00:27.107 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:27.106+0000 7f60f5cc8700 1 -- 172.21.15.155:0/4203884813 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f60f01070d0 con 0x7f60f00f69d0 2024-02-10T09:00:27.108 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:27.107+0000 7f60e7fff700 1 -- 172.21.15.155:0/4203884813 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f60d801d990 con 0x7f60f00f69d0 2024-02-10T09:00:27.108 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:00:27.109 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:00:27.110 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:27.109+0000 7f60f5cc8700 1 -- 172.21.15.155:0/4203884813 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f60dc02ff10 msgr2=0x7f60dc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:00:27.110 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:27.109+0000 7f60f5cc8700 1 --2- 172.21.15.155:0/4203884813 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f60dc02ff10 0x7f60dc0323d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f60e00096a0 tx=0x7f60e0006b40).stop 2024-02-10T09:00:27.110 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:27.109+0000 7f60f5cc8700 1 -- 172.21.15.155:0/4203884813 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60f00f69d0 msgr2=0x7f60f01056a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:00:27.110 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:27.109+0000 7f60f5cc8700 1 --2- 172.21.15.155:0/4203884813 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60f00f69d0 0x7f60f01056a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f60d8005010 tx=0x7f60d800b040).stop 2024-02-10T09:00:27.110 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:27.109+0000 7f60f5cc8700 1 -- 172.21.15.155:0/4203884813 shutdown_connections 2024-02-10T09:00:27.110 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:27.109+0000 7f60f5cc8700 1 --2- 172.21.15.155:0/4203884813 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f60dc02ff10 0x7f60dc0323d0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:00:27.111 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:27.109+0000 7f60f5cc8700 1 --2- 172.21.15.155:0/4203884813 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60f00f69d0 0x7f60f01056a0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:00:27.111 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:27.109+0000 7f60f5cc8700 1 -- 172.21.15.155:0/4203884813 >> 172.21.15.155:0/4203884813 conn(0x7f60f00f02e0 msgr2=0x7f60f00f0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:00:27.111 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:27.109+0000 7f60f5cc8700 1 -- 172.21.15.155:0/4203884813 shutdown_connections 2024-02-10T09:00:27.111 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:27.109+0000 7f60f5cc8700 1 -- 172.21.15.155:0/4203884813 wait complete. 2024-02-10T09:00:27.111 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:00:27.398 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:00:27 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:00:26.969834+0000 mgr.smithi077.dxntvw (mgr.14162) 51 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:00:28.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:00:28 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:00:27.108542+0000 mon.smithi077 (mon.0) 134 : audit [DBG] from='client.? 172.21.15.155:0/4203884813' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:00:28.841 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:00:28.842 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:00:29.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:00:29 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:00:28.970335+0000 mgr.smithi077.dxntvw (mgr.14162) 52 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:00:30.756 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:30.726+0000 7fda32256700 1 -- 172.21.15.155:0/4027098438 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda2c0f5790 msgr2=0x7fda2c0f5bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:00:30.756 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:30.726+0000 7fda32256700 1 --2- 172.21.15.155:0/4027098438 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda2c0f5790 0x7fda2c0f5bb0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7fda20009fb0 tx=0x7fda20009b30).stop 2024-02-10T09:00:30.756 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:30.726+0000 7fda32256700 1 -- 172.21.15.155:0/4027098438 shutdown_connections 2024-02-10T09:00:30.756 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:30.726+0000 7fda32256700 1 --2- 172.21.15.155:0/4027098438 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda2c0f5790 0x7fda2c0f5bb0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:00:30.756 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:30.726+0000 7fda32256700 1 -- 172.21.15.155:0/4027098438 >> 172.21.15.155:0/4027098438 conn(0x7fda2c0f0120 msgr2=0x7fda2c0f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:00:30.756 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:30.726+0000 7fda32256700 1 -- 172.21.15.155:0/4027098438 shutdown_connections 2024-02-10T09:00:30.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:30.726+0000 7fda32256700 1 -- 172.21.15.155:0/4027098438 wait complete. 2024-02-10T09:00:30.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:30.727+0000 7fda32256700 1 Processor -- start 2024-02-10T09:00:30.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:30.727+0000 7fda32256700 1 -- start start 2024-02-10T09:00:30.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:30.727+0000 7fda32256700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda2c0f5790 0x7fda2c0fd250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:00:30.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:30.727+0000 7fda32256700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda2c0fb6d0 con 0x7fda2c0f5790 2024-02-10T09:00:30.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:30.727+0000 7fda31254700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda2c0f5790 0x7fda2c0fd250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:00:30.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:30.727+0000 7fda31254700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda2c0f5790 0x7fda2c0fd250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:45902/0 (socket says 172.21.15.155:45902) 2024-02-10T09:00:30.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:30.727+0000 7fda31254700 1 -- 172.21.15.155:0/954441584 learned_addr learned my addr 172.21.15.155:0/954441584 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:00:30.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:30.728+0000 7fda31254700 1 -- 172.21.15.155:0/954441584 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda20005040 con 0x7fda2c0f5790 2024-02-10T09:00:30.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:30.728+0000 7fda31254700 1 --2- 172.21.15.155:0/954441584 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda2c0f5790 0x7fda2c0fd250 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fda20009f80 tx=0x7fda2000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:00:30.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:30.728+0000 7fda1dffb700 1 -- 172.21.15.155:0/954441584 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda200095f0 con 0x7fda2c0f5790 2024-02-10T09:00:30.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:30.728+0000 7fda32256700 1 -- 172.21.15.155:0/954441584 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda2c0fb870 con 0x7fda2c0f5790 2024-02-10T09:00:30.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:30.728+0000 7fda1dffb700 1 -- 172.21.15.155:0/954441584 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fda20007e90 con 0x7fda2c0f5790 2024-02-10T09:00:30.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:30.728+0000 7fda1dffb700 1 -- 172.21.15.155:0/954441584 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda20014e60 con 0x7fda2c0f5790 2024-02-10T09:00:30.758 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:30.728+0000 7fda32256700 1 -- 172.21.15.155:0/954441584 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda2c0fbd00 con 0x7fda2c0f5790 2024-02-10T09:00:30.758 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:30.729+0000 7fda1dffb700 1 -- 172.21.15.155:0/954441584 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fda20025070 con 0x7fda2c0f5790 2024-02-10T09:00:30.758 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:30.729+0000 7fda32256700 1 -- 172.21.15.155:0/954441584 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda2c059100 con 0x7fda2c0f5790 2024-02-10T09:00:30.758 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:30.729+0000 7fda1dffb700 1 --2- 172.21.15.155:0/954441584 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fda1802ff10 0x7fda180323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:00:30.758 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:30.729+0000 7fda1dffb700 1 -- 172.21.15.155:0/954441584 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fda20040950 con 0x7fda2c0f5790 2024-02-10T09:00:30.758 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:30.729+0000 7fda30a53700 1 --2- 172.21.15.155:0/954441584 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fda1802ff10 0x7fda180323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:00:30.758 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:30.730+0000 7fda30a53700 1 --2- 172.21.15.155:0/954441584 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fda1802ff10 0x7fda180323d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7fda280096a0 tx=0x7fda28006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:00:30.758 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:30.734+0000 7fda1dffb700 1 -- 172.21.15.155:0/954441584 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fda2000f4e0 con 0x7fda2c0f5790 2024-02-10T09:00:30.939 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:30.938+0000 7fda32256700 1 -- 172.21.15.155:0/954441584 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fda2c045ad0 con 0x7fda2c0f5790 2024-02-10T09:00:30.939 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:30.938+0000 7fda1dffb700 1 -- 172.21.15.155:0/954441584 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fda2000f300 con 0x7fda2c0f5790 2024-02-10T09:00:30.940 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:00:30.940 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:00:30.941 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:30.940+0000 7fda32256700 1 -- 172.21.15.155:0/954441584 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fda1802ff10 msgr2=0x7fda180323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:00:30.941 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:30.940+0000 7fda32256700 1 --2- 172.21.15.155:0/954441584 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fda1802ff10 0x7fda180323d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7fda280096a0 tx=0x7fda28006b40).stop 2024-02-10T09:00:30.941 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:30.940+0000 7fda32256700 1 -- 172.21.15.155:0/954441584 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda2c0f5790 msgr2=0x7fda2c0fd250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:00:30.941 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:30.940+0000 7fda32256700 1 --2- 172.21.15.155:0/954441584 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda2c0f5790 0x7fda2c0fd250 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fda20009f80 tx=0x7fda2000b040).stop 2024-02-10T09:00:30.942 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:30.940+0000 7fda32256700 1 -- 172.21.15.155:0/954441584 shutdown_connections 2024-02-10T09:00:30.942 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:30.940+0000 7fda32256700 1 --2- 172.21.15.155:0/954441584 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fda1802ff10 0x7fda180323d0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:00:30.942 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:30.940+0000 7fda32256700 1 --2- 172.21.15.155:0/954441584 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fda2c0f5790 0x7fda2c0fd250 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:00:30.942 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:30.940+0000 7fda32256700 1 -- 172.21.15.155:0/954441584 >> 172.21.15.155:0/954441584 conn(0x7fda2c0f0120 msgr2=0x7fda2c0f4d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:00:30.942 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:30.940+0000 7fda32256700 1 -- 172.21.15.155:0/954441584 shutdown_connections 2024-02-10T09:00:30.942 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:30.940+0000 7fda32256700 1 -- 172.21.15.155:0/954441584 wait complete. 2024-02-10T09:00:30.942 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:00:31.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:00:30 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:00:30.939860+0000 mon.smithi077 (mon.0) 135 : audit [DBG] from='client.? 172.21.15.155:0/954441584' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:00:31.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:00:30 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:00:30.970629+0000 mgr.smithi077.dxntvw (mgr.14162) 53 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:00:32.684 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:00:32.684 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:00:34.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:00:33 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:00:32.970993+0000 mgr.smithi077.dxntvw (mgr.14162) 54 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:00:34.692 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:34.620+0000 7f7ac2aaf700 1 -- 172.21.15.155:0/2646052703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7abc0f5410 msgr2=0x7f7abc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:00:34.693 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:34.620+0000 7f7ac2aaf700 1 --2- 172.21.15.155:0/2646052703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7abc0f5410 0x7f7abc0f5830 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f7ab0009fb0 tx=0x7f7ab0009b30).stop 2024-02-10T09:00:34.693 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:34.620+0000 7f7ac2aaf700 1 -- 172.21.15.155:0/2646052703 shutdown_connections 2024-02-10T09:00:34.693 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:34.620+0000 7f7ac2aaf700 1 --2- 172.21.15.155:0/2646052703 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7abc0f5410 0x7f7abc0f5830 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:00:34.693 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:34.620+0000 7f7ac2aaf700 1 -- 172.21.15.155:0/2646052703 >> 172.21.15.155:0/2646052703 conn(0x7f7abc0f0b70 msgr2=0x7f7abc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:00:34.693 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:34.620+0000 7f7ac2aaf700 1 -- 172.21.15.155:0/2646052703 shutdown_connections 2024-02-10T09:00:34.693 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:34.620+0000 7f7ac2aaf700 1 -- 172.21.15.155:0/2646052703 wait complete. 2024-02-10T09:00:34.693 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:34.620+0000 7f7ac2aaf700 1 Processor -- start 2024-02-10T09:00:34.693 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:34.621+0000 7f7ac2aaf700 1 -- start start 2024-02-10T09:00:34.693 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:34.621+0000 7f7ac2aaf700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7abc0f5410 0x7f7abc0fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:00:34.693 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:34.621+0000 7f7ac2aaf700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7abc0fb960 con 0x7f7abc0f5410 2024-02-10T09:00:34.693 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:34.621+0000 7f7ac084b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7abc0f5410 0x7f7abc0fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:00:34.693 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:34.621+0000 7f7ac084b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7abc0f5410 0x7f7abc0fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:55052/0 (socket says 172.21.15.155:55052) 2024-02-10T09:00:34.693 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:34.621+0000 7f7ac084b700 1 -- 172.21.15.155:0/4093353605 learned_addr learned my addr 172.21.15.155:0/4093353605 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:00:34.694 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:34.622+0000 7f7ac084b700 1 -- 172.21.15.155:0/4093353605 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ab0005040 con 0x7f7abc0f5410 2024-02-10T09:00:34.694 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:34.622+0000 7f7ac084b700 1 --2- 172.21.15.155:0/4093353605 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7abc0f5410 0x7f7abc0fd4e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f7ab00089f0 tx=0x7f7ab000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:00:34.694 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:34.622+0000 7f7ab97fa700 1 -- 172.21.15.155:0/4093353605 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ab0009320 con 0x7f7abc0f5410 2024-02-10T09:00:34.694 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:34.622+0000 7f7ab97fa700 1 -- 172.21.15.155:0/4093353605 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7ab0013380 con 0x7f7abc0f5410 2024-02-10T09:00:34.694 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:34.622+0000 7f7ab97fa700 1 -- 172.21.15.155:0/4093353605 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ab001ed80 con 0x7f7abc0f5410 2024-02-10T09:00:34.694 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:34.622+0000 7f7ac2aaf700 1 -- 172.21.15.155:0/4093353605 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7abc0fbb00 con 0x7f7abc0f5410 2024-02-10T09:00:34.694 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:34.622+0000 7f7ac2aaf700 1 -- 172.21.15.155:0/4093353605 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7abc0fbf90 con 0x7f7abc0f5410 2024-02-10T09:00:34.694 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:34.623+0000 7f7ab97fa700 1 -- 172.21.15.155:0/4093353605 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7ab001e420 con 0x7f7abc0f5410 2024-02-10T09:00:34.694 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:34.623+0000 7f7ac2aaf700 1 -- 172.21.15.155:0/4093353605 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7abc059100 con 0x7f7abc0f5410 2024-02-10T09:00:34.694 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:34.623+0000 7f7ab97fa700 1 --2- 172.21.15.155:0/4093353605 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f7aac02ff60 0x7f7aac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:00:34.694 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:34.623+0000 7f7ab97fa700 1 -- 172.21.15.155:0/4093353605 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7ab0040fa0 con 0x7f7abc0f5410 2024-02-10T09:00:34.694 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:34.623+0000 7f7abbfff700 1 --2- 172.21.15.155:0/4093353605 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f7aac02ff60 0x7f7aac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:00:34.694 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:34.623+0000 7f7abbfff700 1 --2- 172.21.15.155:0/4093353605 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f7aac02ff60 0x7f7aac032420 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f7aa80096a0 tx=0x7f7aa8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:00:34.694 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:34.628+0000 7f7ab97fa700 1 -- 172.21.15.155:0/4093353605 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7ab00095f0 con 0x7f7abc0f5410 2024-02-10T09:00:34.833 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:34.832+0000 7f7ac2aaf700 1 -- 172.21.15.155:0/4093353605 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7abc0fafa0 con 0x7f7abc0f5410 2024-02-10T09:00:34.834 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:34.833+0000 7f7ab97fa700 1 -- 172.21.15.155:0/4093353605 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7ab001cb10 con 0x7f7abc0f5410 2024-02-10T09:00:34.834 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:00:34.834 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:00:34.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:34.835+0000 7f7ac2aaf700 1 -- 172.21.15.155:0/4093353605 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f7aac02ff60 msgr2=0x7f7aac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:00:34.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:34.835+0000 7f7ac2aaf700 1 --2- 172.21.15.155:0/4093353605 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f7aac02ff60 0x7f7aac032420 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f7aa80096a0 tx=0x7f7aa8006b40).stop 2024-02-10T09:00:34.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:34.835+0000 7f7ac2aaf700 1 -- 172.21.15.155:0/4093353605 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7abc0f5410 msgr2=0x7f7abc0fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:00:34.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:34.835+0000 7f7ac2aaf700 1 --2- 172.21.15.155:0/4093353605 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7abc0f5410 0x7f7abc0fd4e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f7ab00089f0 tx=0x7f7ab000b040).stop 2024-02-10T09:00:34.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:34.835+0000 7f7ac2aaf700 1 -- 172.21.15.155:0/4093353605 shutdown_connections 2024-02-10T09:00:34.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:34.835+0000 7f7ac2aaf700 1 --2- 172.21.15.155:0/4093353605 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f7aac02ff60 0x7f7aac032420 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:00:34.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:34.835+0000 7f7ac2aaf700 1 --2- 172.21.15.155:0/4093353605 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7abc0f5410 0x7f7abc0fd4e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:00:34.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:34.835+0000 7f7ac2aaf700 1 -- 172.21.15.155:0/4093353605 >> 172.21.15.155:0/4093353605 conn(0x7f7abc0f0b70 msgr2=0x7f7abc05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:00:34.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:34.835+0000 7f7ac2aaf700 1 -- 172.21.15.155:0/4093353605 shutdown_connections 2024-02-10T09:00:34.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:34.835+0000 7f7ac2aaf700 1 -- 172.21.15.155:0/4093353605 wait complete. 2024-02-10T09:00:34.837 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:00:35.148 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:00:34 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:00:34.834501+0000 mon.smithi077 (mon.0) 136 : audit [DBG] from='client.? 172.21.15.155:0/4093353605' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:00:36.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:00:35 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:00:34.971293+0000 mgr.smithi077.dxntvw (mgr.14162) 55 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:00:36.580 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:00:36.581 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:00:37.398 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:00:37 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:00:36.971582+0000 mgr.smithi077.dxntvw (mgr.14162) 56 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:00:38.523 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:38.472+0000 7fc4b42cc700 1 -- 172.21.15.155:0/3142755606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4ac0f5410 msgr2=0x7fc4ac0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:00:38.524 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:38.472+0000 7fc4b42cc700 1 --2- 172.21.15.155:0/3142755606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4ac0f5410 0x7fc4ac0f5830 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fc49c009fb0 tx=0x7fc49c009b30).stop 2024-02-10T09:00:38.524 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:38.472+0000 7fc4b42cc700 1 -- 172.21.15.155:0/3142755606 shutdown_connections 2024-02-10T09:00:38.524 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:38.472+0000 7fc4b42cc700 1 --2- 172.21.15.155:0/3142755606 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4ac0f5410 0x7fc4ac0f5830 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:00:38.524 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:38.472+0000 7fc4b42cc700 1 -- 172.21.15.155:0/3142755606 >> 172.21.15.155:0/3142755606 conn(0x7fc4ac0f0b70 msgr2=0x7fc4ac0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:00:38.525 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:38.472+0000 7fc4b42cc700 1 -- 172.21.15.155:0/3142755606 shutdown_connections 2024-02-10T09:00:38.525 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:38.472+0000 7fc4b42cc700 1 -- 172.21.15.155:0/3142755606 wait complete. 2024-02-10T09:00:38.525 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:38.473+0000 7fc4b42cc700 1 Processor -- start 2024-02-10T09:00:38.525 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:38.473+0000 7fc4b42cc700 1 -- start start 2024-02-10T09:00:38.525 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:38.473+0000 7fc4b42cc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4ac0f5410 0x7fc4ac1067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:00:38.525 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:38.473+0000 7fc4b42cc700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ac106ca0 con 0x7fc4ac0f5410 2024-02-10T09:00:38.525 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:38.473+0000 7fc4b2068700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4ac0f5410 0x7fc4ac1067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:00:38.525 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:38.473+0000 7fc4b2068700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4ac0f5410 0x7fc4ac1067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:55070/0 (socket says 172.21.15.155:55070) 2024-02-10T09:00:38.525 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:38.473+0000 7fc4b2068700 1 -- 172.21.15.155:0/1970954208 learned_addr learned my addr 172.21.15.155:0/1970954208 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:00:38.526 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:38.473+0000 7fc4b2068700 1 -- 172.21.15.155:0/1970954208 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc49c005040 con 0x7fc4ac0f5410 2024-02-10T09:00:38.526 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:38.474+0000 7fc4b2068700 1 --2- 172.21.15.155:0/1970954208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4ac0f5410 0x7fc4ac1067b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7fc49c005010 tx=0x7fc49c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:00:38.526 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:38.474+0000 7fc4a2ffd700 1 -- 172.21.15.155:0/1970954208 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc49c009320 con 0x7fc4ac0f5410 2024-02-10T09:00:38.526 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:38.474+0000 7fc4b42cc700 1 -- 172.21.15.155:0/1970954208 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4ac106ea0 con 0x7fc4ac0f5410 2024-02-10T09:00:38.526 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:38.474+0000 7fc4a2ffd700 1 -- 172.21.15.155:0/1970954208 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc49c013380 con 0x7fc4ac0f5410 2024-02-10T09:00:38.526 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:38.474+0000 7fc4a2ffd700 1 -- 172.21.15.155:0/1970954208 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc49c014e50 con 0x7fc4ac0f5410 2024-02-10T09:00:38.526 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:38.474+0000 7fc4b42cc700 1 -- 172.21.15.155:0/1970954208 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4ac107330 con 0x7fc4ac0f5410 2024-02-10T09:00:38.526 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:38.475+0000 7fc4b42cc700 1 -- 172.21.15.155:0/1970954208 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4ac045ad0 con 0x7fc4ac0f5410 2024-02-10T09:00:38.526 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:38.475+0000 7fc4a2ffd700 1 -- 172.21.15.155:0/1970954208 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc49c0095f0 con 0x7fc4ac0f5410 2024-02-10T09:00:38.526 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:38.475+0000 7fc4a2ffd700 1 --2- 172.21.15.155:0/1970954208 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fc49802ff10 0x7fc4980323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:00:38.526 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:38.475+0000 7fc4a2ffd700 1 -- 172.21.15.155:0/1970954208 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc49c040e90 con 0x7fc4ac0f5410 2024-02-10T09:00:38.526 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:38.475+0000 7fc4b1867700 1 --2- 172.21.15.155:0/1970954208 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fc49802ff10 0x7fc4980323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:00:38.526 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:38.476+0000 7fc4b1867700 1 --2- 172.21.15.155:0/1970954208 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fc49802ff10 0x7fc4980323d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7fc4a80096a0 tx=0x7fc4a8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:00:38.527 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:38.480+0000 7fc4a2ffd700 1 -- 172.21.15.155:0/1970954208 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc49c014420 con 0x7fc4ac0f5410 2024-02-10T09:00:38.690 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:38.689+0000 7fc4b42cc700 1 -- 172.21.15.155:0/1970954208 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc4ac0fafa0 con 0x7fc4ac0f5410 2024-02-10T09:00:38.691 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:38.689+0000 7fc4a2ffd700 1 -- 172.21.15.155:0/1970954208 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc49c010230 con 0x7fc4ac0f5410 2024-02-10T09:00:38.692 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:00:38.692 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:00:38.693 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:38.691+0000 7fc4b42cc700 1 -- 172.21.15.155:0/1970954208 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fc49802ff10 msgr2=0x7fc4980323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:00:38.693 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:38.691+0000 7fc4b42cc700 1 --2- 172.21.15.155:0/1970954208 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fc49802ff10 0x7fc4980323d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7fc4a80096a0 tx=0x7fc4a8006b40).stop 2024-02-10T09:00:38.694 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:38.691+0000 7fc4b42cc700 1 -- 172.21.15.155:0/1970954208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4ac0f5410 msgr2=0x7fc4ac1067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:00:38.694 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:38.691+0000 7fc4b42cc700 1 --2- 172.21.15.155:0/1970954208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4ac0f5410 0x7fc4ac1067b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7fc49c005010 tx=0x7fc49c00b040).stop 2024-02-10T09:00:38.694 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:38.691+0000 7fc4b42cc700 1 -- 172.21.15.155:0/1970954208 shutdown_connections 2024-02-10T09:00:38.694 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:38.691+0000 7fc4b42cc700 1 --2- 172.21.15.155:0/1970954208 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fc49802ff10 0x7fc4980323d0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:00:38.694 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:38.691+0000 7fc4b42cc700 1 --2- 172.21.15.155:0/1970954208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4ac0f5410 0x7fc4ac1067b0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:00:38.695 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:38.691+0000 7fc4b42cc700 1 -- 172.21.15.155:0/1970954208 >> 172.21.15.155:0/1970954208 conn(0x7fc4ac0f0b70 msgr2=0x7fc4ac05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:00:38.695 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:38.691+0000 7fc4b42cc700 1 -- 172.21.15.155:0/1970954208 shutdown_connections 2024-02-10T09:00:38.695 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:38.691+0000 7fc4b42cc700 1 -- 172.21.15.155:0/1970954208 wait complete. 2024-02-10T09:00:38.695 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:00:39.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:00:38 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:00:38.690756+0000 mon.smithi077 (mon.0) 137 : audit [DBG] from='client.? 172.21.15.155:0/1970954208' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:00:40.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:00:39 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:00:38.971940+0000 mgr.smithi077.dxntvw (mgr.14162) 57 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:00:40.423 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:00:40.424 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:00:41.400 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:00:41 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:00:40.972315+0000 mgr.smithi077.dxntvw (mgr.14162) 58 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:00:42.224 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:42.223+0000 7f0cd03c8700 1 -- 172.21.15.155:0/2019993141 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0cc80f6990 msgr2=0x7f0cc80f6db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:00:42.225 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:42.223+0000 7f0cd03c8700 1 --2- 172.21.15.155:0/2019993141 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0cc80f6990 0x7f0cc80f6db0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f0cb8009fb0 tx=0x7f0cb8009b30).stop 2024-02-10T09:00:42.225 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:42.223+0000 7f0cd03c8700 1 -- 172.21.15.155:0/2019993141 shutdown_connections 2024-02-10T09:00:42.225 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:42.223+0000 7f0cd03c8700 1 --2- 172.21.15.155:0/2019993141 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0cc80f6990 0x7f0cc80f6db0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:00:42.225 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:42.223+0000 7f0cd03c8700 1 -- 172.21.15.155:0/2019993141 >> 172.21.15.155:0/2019993141 conn(0x7f0cc80f0260 msgr2=0x7f0cc80f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:00:42.225 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:42.223+0000 7f0cd03c8700 1 -- 172.21.15.155:0/2019993141 shutdown_connections 2024-02-10T09:00:42.225 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:42.223+0000 7f0cd03c8700 1 -- 172.21.15.155:0/2019993141 wait complete. 2024-02-10T09:00:42.225 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:42.223+0000 7f0cd03c8700 1 Processor -- start 2024-02-10T09:00:42.225 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:42.223+0000 7f0cd03c8700 1 -- start start 2024-02-10T09:00:42.225 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:42.223+0000 7f0cd03c8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0cc80f6990 0x7f0cc81094b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:00:42.225 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:42.223+0000 7f0cd03c8700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc80f5310 con 0x7f0cc80f6990 2024-02-10T09:00:42.227 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:42.224+0000 7f0cce164700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0cc80f6990 0x7f0cc81094b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:00:42.227 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:42.224+0000 7f0cce164700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0cc80f6990 0x7f0cc81094b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:55076/0 (socket says 172.21.15.155:55076) 2024-02-10T09:00:42.227 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:42.224+0000 7f0cce164700 1 -- 172.21.15.155:0/1247707253 learned_addr learned my addr 172.21.15.155:0/1247707253 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:00:42.227 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:42.224+0000 7f0cce164700 1 -- 172.21.15.155:0/1247707253 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cb8005040 con 0x7f0cc80f6990 2024-02-10T09:00:42.227 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:42.225+0000 7f0cce164700 1 --2- 172.21.15.155:0/1247707253 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0cc80f6990 0x7f0cc81094b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f0cb80089f0 tx=0x7f0cb800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:00:42.227 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:42.225+0000 7f0cbeffd700 1 -- 172.21.15.155:0/1247707253 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0cb8009320 con 0x7f0cc80f6990 2024-02-10T09:00:42.227 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:42.225+0000 7f0cd03c8700 1 -- 172.21.15.155:0/1247707253 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cc80f5510 con 0x7f0cc80f6990 2024-02-10T09:00:42.228 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:42.225+0000 7f0cbeffd700 1 -- 172.21.15.155:0/1247707253 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0cb8013380 con 0x7f0cc80f6990 2024-02-10T09:00:42.228 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:42.225+0000 7f0cbeffd700 1 -- 172.21.15.155:0/1247707253 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0cb801ed80 con 0x7f0cc80f6990 2024-02-10T09:00:42.228 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:42.225+0000 7f0cd03c8700 1 -- 172.21.15.155:0/1247707253 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cc8109be0 con 0x7f0cc80f6990 2024-02-10T09:00:42.228 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:42.226+0000 7f0cbeffd700 1 -- 172.21.15.155:0/1247707253 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0cb801e420 con 0x7f0cc80f6990 2024-02-10T09:00:42.228 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:42.226+0000 7f0cd03c8700 1 -- 172.21.15.155:0/1247707253 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cc8045ad0 con 0x7f0cc80f6990 2024-02-10T09:00:42.228 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:42.226+0000 7f0cbeffd700 1 --2- 172.21.15.155:0/1247707253 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f0cb402ff60 0x7f0cb4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:00:42.228 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:42.226+0000 7f0cbeffd700 1 -- 172.21.15.155:0/1247707253 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0cb8041080 con 0x7f0cc80f6990 2024-02-10T09:00:42.228 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:42.226+0000 7f0ccd963700 1 --2- 172.21.15.155:0/1247707253 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f0cb402ff60 0x7f0cb4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:00:42.228 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:42.227+0000 7f0ccd963700 1 --2- 172.21.15.155:0/1247707253 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f0cb402ff60 0x7f0cb4032420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f0cc40096a0 tx=0x7f0cc4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:00:42.231 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:42.231+0000 7f0cbeffd700 1 -- 172.21.15.155:0/1247707253 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0cb801cb10 con 0x7f0cc80f6990 2024-02-10T09:00:42.438 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:42.437+0000 7f0cd03c8700 1 -- 172.21.15.155:0/1247707253 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0cc80faf90 con 0x7f0cc80f6990 2024-02-10T09:00:42.438 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:42.438+0000 7f0cbeffd700 1 -- 172.21.15.155:0/1247707253 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0cb8004490 con 0x7f0cc80f6990 2024-02-10T09:00:42.439 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:00:42.439 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:00:42.440 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:42.439+0000 7f0cd03c8700 1 -- 172.21.15.155:0/1247707253 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f0cb402ff60 msgr2=0x7f0cb4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:00:42.440 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:42.439+0000 7f0cd03c8700 1 --2- 172.21.15.155:0/1247707253 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f0cb402ff60 0x7f0cb4032420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f0cc40096a0 tx=0x7f0cc4006b40).stop 2024-02-10T09:00:42.440 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:42.439+0000 7f0cd03c8700 1 -- 172.21.15.155:0/1247707253 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0cc80f6990 msgr2=0x7f0cc81094b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:00:42.440 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:42.439+0000 7f0cd03c8700 1 --2- 172.21.15.155:0/1247707253 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0cc80f6990 0x7f0cc81094b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f0cb80089f0 tx=0x7f0cb800b040).stop 2024-02-10T09:00:42.441 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:42.439+0000 7f0cd03c8700 1 -- 172.21.15.155:0/1247707253 shutdown_connections 2024-02-10T09:00:42.441 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:42.439+0000 7f0cd03c8700 1 --2- 172.21.15.155:0/1247707253 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f0cb402ff60 0x7f0cb4032420 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:00:42.441 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:42.439+0000 7f0cd03c8700 1 --2- 172.21.15.155:0/1247707253 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0cc80f6990 0x7f0cc81094b0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:00:42.441 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:42.439+0000 7f0cd03c8700 1 -- 172.21.15.155:0/1247707253 >> 172.21.15.155:0/1247707253 conn(0x7f0cc80f0260 msgr2=0x7f0cc80f0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:00:42.441 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:42.439+0000 7f0cd03c8700 1 -- 172.21.15.155:0/1247707253 shutdown_connections 2024-02-10T09:00:42.441 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:42.439+0000 7f0cd03c8700 1 -- 172.21.15.155:0/1247707253 wait complete. 2024-02-10T09:00:42.441 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:00:42.899 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:00:42 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:00:42.439014+0000 mon.smithi077 (mon.0) 138 : audit [DBG] from='client.? 172.21.15.155:0/1247707253' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:00:44.121 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:00:44.121 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:00:44.148 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:00:43 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:00:42.972574+0000 mgr.smithi077.dxntvw (mgr.14162) 59 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:00:45.398 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:00:45 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:00:44.972922+0000 mgr.smithi077.dxntvw (mgr.14162) 60 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:00:46.230 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:46.213+0000 7fd3b9327700 1 -- 172.21.15.155:0/2624143789 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3b40f51a0 msgr2=0x7fd3b40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:00:46.230 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:46.213+0000 7fd3b9327700 1 --2- 172.21.15.155:0/2624143789 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3b40f51a0 0x7fd3b40f55c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fd3a8009fb0 tx=0x7fd3a8009b30).stop 2024-02-10T09:00:46.230 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:46.213+0000 7fd3b9327700 1 -- 172.21.15.155:0/2624143789 shutdown_connections 2024-02-10T09:00:46.230 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:46.213+0000 7fd3b9327700 1 --2- 172.21.15.155:0/2624143789 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3b40f51a0 0x7fd3b40f55c0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:00:46.230 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:46.213+0000 7fd3b9327700 1 -- 172.21.15.155:0/2624143789 >> 172.21.15.155:0/2624143789 conn(0x7fd3b40f09a0 msgr2=0x7fd3b40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:00:46.230 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:46.213+0000 7fd3b9327700 1 -- 172.21.15.155:0/2624143789 shutdown_connections 2024-02-10T09:00:46.230 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:46.213+0000 7fd3b9327700 1 -- 172.21.15.155:0/2624143789 wait complete. 2024-02-10T09:00:46.231 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:46.213+0000 7fd3b9327700 1 Processor -- start 2024-02-10T09:00:46.231 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:46.214+0000 7fd3b9327700 1 -- start start 2024-02-10T09:00:46.231 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:46.214+0000 7fd3b9327700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3b40f51a0 0x7fd3b4109290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:00:46.231 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:46.214+0000 7fd3b9327700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b4060280 con 0x7fd3b40f51a0 2024-02-10T09:00:46.231 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:46.214+0000 7fd3b3fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3b40f51a0 0x7fd3b4109290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:00:46.231 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:46.214+0000 7fd3b3fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3b40f51a0 0x7fd3b4109290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:60250/0 (socket says 172.21.15.155:60250) 2024-02-10T09:00:46.231 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:46.214+0000 7fd3b3fff700 1 -- 172.21.15.155:0/4046295214 learned_addr learned my addr 172.21.15.155:0/4046295214 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:00:46.231 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:46.215+0000 7fd3b3fff700 1 -- 172.21.15.155:0/4046295214 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3a8005040 con 0x7fd3b40f51a0 2024-02-10T09:00:46.231 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:46.215+0000 7fd3b3fff700 1 --2- 172.21.15.155:0/4046295214 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3b40f51a0 0x7fd3b4109290 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fd3a80089f0 tx=0x7fd3a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:00:46.231 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:46.215+0000 7fd3b0ff9700 1 -- 172.21.15.155:0/4046295214 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd3a8009320 con 0x7fd3b40f51a0 2024-02-10T09:00:46.231 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:46.215+0000 7fd3b9327700 1 -- 172.21.15.155:0/4046295214 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3b4060480 con 0x7fd3b40f51a0 2024-02-10T09:00:46.231 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:46.215+0000 7fd3b0ff9700 1 -- 172.21.15.155:0/4046295214 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd3a8013380 con 0x7fd3b40f51a0 2024-02-10T09:00:46.231 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:46.215+0000 7fd3b0ff9700 1 -- 172.21.15.155:0/4046295214 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd3a8014e20 con 0x7fd3b40f51a0 2024-02-10T09:00:46.232 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:46.215+0000 7fd3b9327700 1 -- 172.21.15.155:0/4046295214 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3b41099c0 con 0x7fd3b40f51a0 2024-02-10T09:00:46.232 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:46.216+0000 7fd3b9327700 1 -- 172.21.15.155:0/4046295214 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3b4045ad0 con 0x7fd3b40f51a0 2024-02-10T09:00:46.232 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:46.216+0000 7fd3b0ff9700 1 -- 172.21.15.155:0/4046295214 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd3a80095f0 con 0x7fd3b40f51a0 2024-02-10T09:00:46.232 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:46.216+0000 7fd3b0ff9700 1 --2- 172.21.15.155:0/4046295214 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fd39c02ff10 0x7fd39c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:00:46.232 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:46.216+0000 7fd3b0ff9700 1 -- 172.21.15.155:0/4046295214 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd3a8040ef0 con 0x7fd3b40f51a0 2024-02-10T09:00:46.232 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:46.216+0000 7fd3b37fe700 1 --2- 172.21.15.155:0/4046295214 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fd39c02ff10 0x7fd39c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:00:46.232 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:46.217+0000 7fd3b37fe700 1 --2- 172.21.15.155:0/4046295214 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fd39c02ff10 0x7fd39c0323d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7fd3a40096a0 tx=0x7fd3a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:00:46.232 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:46.221+0000 7fd3b0ff9700 1 -- 172.21.15.155:0/4046295214 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd3a8014420 con 0x7fd3b40f51a0 2024-02-10T09:00:46.431 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:46.431+0000 7fd3b9327700 1 -- 172.21.15.155:0/4046295214 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd3b40fad30 con 0x7fd3b40f51a0 2024-02-10T09:00:46.431 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:46.431+0000 7fd3b0ff9700 1 -- 172.21.15.155:0/4046295214 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd3a8010230 con 0x7fd3b40f51a0 2024-02-10T09:00:46.432 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:00:46.432 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:00:46.433 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:46.433+0000 7fd3b9327700 1 -- 172.21.15.155:0/4046295214 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fd39c02ff10 msgr2=0x7fd39c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:00:46.433 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:46.433+0000 7fd3b9327700 1 --2- 172.21.15.155:0/4046295214 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fd39c02ff10 0x7fd39c0323d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7fd3a40096a0 tx=0x7fd3a4006b40).stop 2024-02-10T09:00:46.433 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:46.433+0000 7fd3b9327700 1 -- 172.21.15.155:0/4046295214 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3b40f51a0 msgr2=0x7fd3b4109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:00:46.433 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:46.433+0000 7fd3b9327700 1 --2- 172.21.15.155:0/4046295214 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3b40f51a0 0x7fd3b4109290 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fd3a80089f0 tx=0x7fd3a800b040).stop 2024-02-10T09:00:46.434 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:46.433+0000 7fd3b9327700 1 -- 172.21.15.155:0/4046295214 shutdown_connections 2024-02-10T09:00:46.434 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:46.433+0000 7fd3b9327700 1 --2- 172.21.15.155:0/4046295214 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fd39c02ff10 0x7fd39c0323d0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:00:46.434 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:46.433+0000 7fd3b9327700 1 --2- 172.21.15.155:0/4046295214 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3b40f51a0 0x7fd3b4109290 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:00:46.434 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:46.433+0000 7fd3b9327700 1 -- 172.21.15.155:0/4046295214 >> 172.21.15.155:0/4046295214 conn(0x7fd3b40f09a0 msgr2=0x7fd3b40f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:00:46.434 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:46.433+0000 7fd3b9327700 1 -- 172.21.15.155:0/4046295214 shutdown_connections 2024-02-10T09:00:46.434 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:46.433+0000 7fd3b9327700 1 -- 172.21.15.155:0/4046295214 wait complete. 2024-02-10T09:00:46.434 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:00:46.899 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:00:46 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:00:46.431930+0000 mon.smithi077 (mon.0) 139 : audit [DBG] from='client.? 172.21.15.155:0/4046295214' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:00:47.899 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:00:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:00:46.973201+0000 mgr.smithi077.dxntvw (mgr.14162) 61 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:00:47.899 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:00:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:00:47.088471+0000 mon.smithi077 (mon.0) 140 : audit [INF] from='mgr.14162 172.21.15.77:0/2029106299' entity='mgr.smithi077.dxntvw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi077.dxntvw/mirror_snapshot_schedule"}]: dispatch 2024-02-10T09:00:47.899 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:00:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:00:47.096405+0000 mon.smithi077 (mon.0) 141 : audit [INF] from='mgr.14162 172.21.15.77:0/2029106299' entity='mgr.smithi077.dxntvw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi077.dxntvw/trash_purge_schedule"}]: dispatch 2024-02-10T09:00:48.106 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:00:48.106 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:00:49.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:00:49 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:00:48.973476+0000 mgr.smithi077.dxntvw (mgr.14162) 62 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:00:50.092 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:50.076+0000 7f3bec38f700 1 -- 172.21.15.155:0/3171887743 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3be40f4740 msgr2=0x7f3be40f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:00:50.093 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:50.076+0000 7f3bec38f700 1 --2- 172.21.15.155:0/3171887743 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3be40f4740 0x7f3be40f4b60 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f3bd4009fb0 tx=0x7f3bd4009b30).stop 2024-02-10T09:00:50.093 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:50.076+0000 7f3bec38f700 1 -- 172.21.15.155:0/3171887743 shutdown_connections 2024-02-10T09:00:50.093 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:50.076+0000 7f3bec38f700 1 --2- 172.21.15.155:0/3171887743 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3be40f4740 0x7f3be40f4b60 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:00:50.093 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:50.076+0000 7f3bec38f700 1 -- 172.21.15.155:0/3171887743 >> 172.21.15.155:0/3171887743 conn(0x7f3be40f0260 msgr2=0x7f3be40f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:00:50.093 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:50.076+0000 7f3bec38f700 1 -- 172.21.15.155:0/3171887743 shutdown_connections 2024-02-10T09:00:50.093 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:50.076+0000 7f3bec38f700 1 -- 172.21.15.155:0/3171887743 wait complete. 2024-02-10T09:00:50.093 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:50.076+0000 7f3bec38f700 1 Processor -- start 2024-02-10T09:00:50.093 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:50.076+0000 7f3bec38f700 1 -- start start 2024-02-10T09:00:50.093 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:50.077+0000 7f3bec38f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3be40f4740 0x7f3be41067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:00:50.093 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:50.077+0000 7f3bec38f700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3be4106ca0 con 0x7f3be40f4740 2024-02-10T09:00:50.093 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:50.077+0000 7f3bea12b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3be40f4740 0x7f3be41067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:00:50.094 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:50.077+0000 7f3bea12b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3be40f4740 0x7f3be41067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:60282/0 (socket says 172.21.15.155:60282) 2024-02-10T09:00:50.094 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:50.077+0000 7f3bea12b700 1 -- 172.21.15.155:0/3087044768 learned_addr learned my addr 172.21.15.155:0/3087044768 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:00:50.094 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:50.077+0000 7f3bea12b700 1 -- 172.21.15.155:0/3087044768 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bd4005040 con 0x7f3be40f4740 2024-02-10T09:00:50.094 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:50.077+0000 7f3bea12b700 1 --2- 172.21.15.155:0/3087044768 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3be40f4740 0x7f3be41067b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f3bd40089f0 tx=0x7f3bd400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:00:50.094 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:50.077+0000 7f3bdaffd700 1 -- 172.21.15.155:0/3087044768 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3bd4009320 con 0x7f3be40f4740 2024-02-10T09:00:50.094 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:50.077+0000 7f3bec38f700 1 -- 172.21.15.155:0/3087044768 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3be4106ea0 con 0x7f3be40f4740 2024-02-10T09:00:50.094 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:50.077+0000 7f3bdaffd700 1 -- 172.21.15.155:0/3087044768 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3bd4013380 con 0x7f3be40f4740 2024-02-10T09:00:50.094 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:50.077+0000 7f3bdaffd700 1 -- 172.21.15.155:0/3087044768 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3bd401ed80 con 0x7f3be40f4740 2024-02-10T09:00:50.094 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:50.077+0000 7f3bec38f700 1 -- 172.21.15.155:0/3087044768 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3be4107330 con 0x7f3be40f4740 2024-02-10T09:00:50.094 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:50.078+0000 7f3bec38f700 1 -- 172.21.15.155:0/3087044768 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3be4045ad0 con 0x7f3be40f4740 2024-02-10T09:00:50.094 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:50.078+0000 7f3bdaffd700 1 -- 172.21.15.155:0/3087044768 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3bd401e420 con 0x7f3be40f4740 2024-02-10T09:00:50.094 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:50.078+0000 7f3bdaffd700 1 --2- 172.21.15.155:0/3087044768 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f3bd002ff60 0x7f3bd0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:00:50.094 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:50.078+0000 7f3bdaffd700 1 -- 172.21.15.155:0/3087044768 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3bd4041150 con 0x7f3be40f4740 2024-02-10T09:00:50.094 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:50.079+0000 7f3be992a700 1 --2- 172.21.15.155:0/3087044768 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f3bd002ff60 0x7f3bd0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:00:50.095 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:50.079+0000 7f3be992a700 1 --2- 172.21.15.155:0/3087044768 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f3bd002ff60 0x7f3bd0032420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f3be00096a0 tx=0x7f3be0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:00:50.095 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:50.083+0000 7f3bdaffd700 1 -- 172.21.15.155:0/3087044768 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3bd40095f0 con 0x7f3be40f4740 2024-02-10T09:00:50.288 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:50.288+0000 7f3bec38f700 1 -- 172.21.15.155:0/3087044768 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3be41081e0 con 0x7f3be40f4740 2024-02-10T09:00:50.289 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:50.289+0000 7f3bdaffd700 1 -- 172.21.15.155:0/3087044768 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3bd4004490 con 0x7f3be40f4740 2024-02-10T09:00:50.289 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:00:50.289 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:00:50.290 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:50.290+0000 7f3bec38f700 1 -- 172.21.15.155:0/3087044768 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f3bd002ff60 msgr2=0x7f3bd0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:00:50.290 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:50.290+0000 7f3bec38f700 1 --2- 172.21.15.155:0/3087044768 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f3bd002ff60 0x7f3bd0032420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f3be00096a0 tx=0x7f3be0006b40).stop 2024-02-10T09:00:50.291 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:50.290+0000 7f3bec38f700 1 -- 172.21.15.155:0/3087044768 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3be40f4740 msgr2=0x7f3be41067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:00:50.291 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:50.290+0000 7f3bec38f700 1 --2- 172.21.15.155:0/3087044768 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3be40f4740 0x7f3be41067b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f3bd40089f0 tx=0x7f3bd400b040).stop 2024-02-10T09:00:50.291 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:50.290+0000 7f3bec38f700 1 -- 172.21.15.155:0/3087044768 shutdown_connections 2024-02-10T09:00:50.291 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:50.290+0000 7f3bec38f700 1 --2- 172.21.15.155:0/3087044768 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f3bd002ff60 0x7f3bd0032420 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:00:50.291 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:50.290+0000 7f3bec38f700 1 --2- 172.21.15.155:0/3087044768 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3be40f4740 0x7f3be41067b0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:00:50.291 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:50.290+0000 7f3bec38f700 1 -- 172.21.15.155:0/3087044768 >> 172.21.15.155:0/3087044768 conn(0x7f3be40f0260 msgr2=0x7f3be40f0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:00:50.291 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:50.290+0000 7f3bec38f700 1 -- 172.21.15.155:0/3087044768 shutdown_connections 2024-02-10T09:00:50.291 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:50.290+0000 7f3bec38f700 1 -- 172.21.15.155:0/3087044768 wait complete. 2024-02-10T09:00:50.291 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:00:50.649 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:00:50 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:00:50.289221+0000 mon.smithi077 (mon.0) 142 : audit [DBG] from='client.? 172.21.15.155:0/3087044768' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:00:51.649 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:00:51 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:00:50.973718+0000 mgr.smithi077.dxntvw (mgr.14162) 63 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:00:52.055 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:00:52.055 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:00:53.944 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:53.918+0000 7fdb56632700 1 -- 172.21.15.155:0/4053833309 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb500f51a0 msgr2=0x7fdb500f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:00:53.945 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:53.918+0000 7fdb56632700 1 --2- 172.21.15.155:0/4053833309 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb500f51a0 0x7fdb500f55c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7fdb4c009fb0 tx=0x7fdb4c009b30).stop 2024-02-10T09:00:53.945 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:53.918+0000 7fdb56632700 1 -- 172.21.15.155:0/4053833309 shutdown_connections 2024-02-10T09:00:53.945 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:53.918+0000 7fdb56632700 1 --2- 172.21.15.155:0/4053833309 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb500f51a0 0x7fdb500f55c0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:00:53.945 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:53.918+0000 7fdb56632700 1 -- 172.21.15.155:0/4053833309 >> 172.21.15.155:0/4053833309 conn(0x7fdb500f09a0 msgr2=0x7fdb500f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:00:53.945 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:53.918+0000 7fdb56632700 1 -- 172.21.15.155:0/4053833309 shutdown_connections 2024-02-10T09:00:53.945 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:53.918+0000 7fdb56632700 1 -- 172.21.15.155:0/4053833309 wait complete. 2024-02-10T09:00:53.945 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:53.918+0000 7fdb56632700 1 Processor -- start 2024-02-10T09:00:53.945 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:53.918+0000 7fdb56632700 1 -- start start 2024-02-10T09:00:53.945 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:53.918+0000 7fdb56632700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb500f51a0 0x7fdb50109290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:00:53.945 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:53.918+0000 7fdb56632700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb50060280 con 0x7fdb500f51a0 2024-02-10T09:00:53.945 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:53.919+0000 7fdb55630700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb500f51a0 0x7fdb50109290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:00:53.946 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:53.919+0000 7fdb55630700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb500f51a0 0x7fdb50109290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:38282/0 (socket says 172.21.15.155:38282) 2024-02-10T09:00:53.946 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:53.919+0000 7fdb55630700 1 -- 172.21.15.155:0/1083703638 learned_addr learned my addr 172.21.15.155:0/1083703638 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:00:53.946 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:53.919+0000 7fdb55630700 1 -- 172.21.15.155:0/1083703638 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb4c005040 con 0x7fdb500f51a0 2024-02-10T09:00:53.946 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:53.919+0000 7fdb55630700 1 --2- 172.21.15.155:0/1083703638 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb500f51a0 0x7fdb50109290 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fdb4c0089f0 tx=0x7fdb4c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:00:53.946 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:53.919+0000 7fdb45ffb700 1 -- 172.21.15.155:0/1083703638 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdb4c009320 con 0x7fdb500f51a0 2024-02-10T09:00:53.946 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:53.919+0000 7fdb56632700 1 -- 172.21.15.155:0/1083703638 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb50060480 con 0x7fdb500f51a0 2024-02-10T09:00:53.946 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:53.919+0000 7fdb45ffb700 1 -- 172.21.15.155:0/1083703638 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdb4c013380 con 0x7fdb500f51a0 2024-02-10T09:00:53.946 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:53.919+0000 7fdb45ffb700 1 -- 172.21.15.155:0/1083703638 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdb4c014e20 con 0x7fdb500f51a0 2024-02-10T09:00:53.946 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:53.919+0000 7fdb56632700 1 -- 172.21.15.155:0/1083703638 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb501099c0 con 0x7fdb500f51a0 2024-02-10T09:00:53.946 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:53.920+0000 7fdb45ffb700 1 -- 172.21.15.155:0/1083703638 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdb4c0095f0 con 0x7fdb500f51a0 2024-02-10T09:00:53.946 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:53.920+0000 7fdb56632700 1 -- 172.21.15.155:0/1083703638 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb50045ad0 con 0x7fdb500f51a0 2024-02-10T09:00:53.946 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:53.921+0000 7fdb45ffb700 1 --2- 172.21.15.155:0/1083703638 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fdb3c02ff10 0x7fdb3c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:00:53.946 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:53.921+0000 7fdb45ffb700 1 -- 172.21.15.155:0/1083703638 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdb4c040e90 con 0x7fdb500f51a0 2024-02-10T09:00:53.946 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:53.921+0000 7fdb54e2f700 1 --2- 172.21.15.155:0/1083703638 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fdb3c02ff10 0x7fdb3c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:00:53.946 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:53.921+0000 7fdb54e2f700 1 --2- 172.21.15.155:0/1083703638 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fdb3c02ff10 0x7fdb3c0323d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fdb400096a0 tx=0x7fdb40006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:00:53.947 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:53.925+0000 7fdb45ffb700 1 -- 172.21.15.155:0/1083703638 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdb4c014420 con 0x7fdb500f51a0 2024-02-10T09:00:54.132 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:54.132+0000 7fdb56632700 1 -- 172.21.15.155:0/1083703638 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdb500fad30 con 0x7fdb500f51a0 2024-02-10T09:00:54.134 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:54.133+0000 7fdb45ffb700 1 -- 172.21.15.155:0/1083703638 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdb4c01d980 con 0x7fdb500f51a0 2024-02-10T09:00:54.134 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:00:54.134 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:00:54.135 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:54.135+0000 7fdb56632700 1 -- 172.21.15.155:0/1083703638 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fdb3c02ff10 msgr2=0x7fdb3c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:00:54.135 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:54.135+0000 7fdb56632700 1 --2- 172.21.15.155:0/1083703638 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fdb3c02ff10 0x7fdb3c0323d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fdb400096a0 tx=0x7fdb40006b40).stop 2024-02-10T09:00:54.136 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:54.135+0000 7fdb56632700 1 -- 172.21.15.155:0/1083703638 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb500f51a0 msgr2=0x7fdb50109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:00:54.136 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:54.135+0000 7fdb56632700 1 --2- 172.21.15.155:0/1083703638 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb500f51a0 0x7fdb50109290 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fdb4c0089f0 tx=0x7fdb4c00b040).stop 2024-02-10T09:00:54.136 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:54.135+0000 7fdb56632700 1 -- 172.21.15.155:0/1083703638 shutdown_connections 2024-02-10T09:00:54.136 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:54.135+0000 7fdb56632700 1 --2- 172.21.15.155:0/1083703638 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fdb3c02ff10 0x7fdb3c0323d0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:00:54.136 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:54.135+0000 7fdb56632700 1 --2- 172.21.15.155:0/1083703638 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdb500f51a0 0x7fdb50109290 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:00:54.136 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:54.135+0000 7fdb56632700 1 -- 172.21.15.155:0/1083703638 >> 172.21.15.155:0/1083703638 conn(0x7fdb500f09a0 msgr2=0x7fdb500f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:00:54.136 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:54.135+0000 7fdb56632700 1 -- 172.21.15.155:0/1083703638 shutdown_connections 2024-02-10T09:00:54.136 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:54.135+0000 7fdb56632700 1 -- 172.21.15.155:0/1083703638 wait complete. 2024-02-10T09:00:54.136 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:00:54.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:00:53 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:00:52.974032+0000 mgr.smithi077.dxntvw (mgr.14162) 64 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:00:55.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:00:54 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:00:54.133330+0000 mon.smithi077 (mon.0) 143 : audit [DBG] from='client.? 172.21.15.155:0/1083703638' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:00:55.684 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:00:55.684 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:00:56.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:00:55 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:00:54.974370+0000 mgr.smithi077.dxntvw (mgr.14162) 65 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:00:57.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:00:57 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:00:56.974694+0000 mgr.smithi077.dxntvw (mgr.14162) 66 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:00:57.645 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:57.643+0000 7f1a04e6a700 1 -- 172.21.15.155:0/331421436 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a000f62a0 msgr2=0x7f1a000f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:00:57.645 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:57.643+0000 7f1a04e6a700 1 --2- 172.21.15.155:0/331421436 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a000f62a0 0x7f1a000f66c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f19f0009fb0 tx=0x7f19f0009b30).stop 2024-02-10T09:00:57.646 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:57.644+0000 7f1a04e6a700 1 -- 172.21.15.155:0/331421436 shutdown_connections 2024-02-10T09:00:57.646 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:57.644+0000 7f1a04e6a700 1 --2- 172.21.15.155:0/331421436 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a000f62a0 0x7f1a000f66c0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:00:57.646 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:57.644+0000 7f1a04e6a700 1 -- 172.21.15.155:0/331421436 >> 172.21.15.155:0/331421436 conn(0x7f1a000f1aa0 msgr2=0x7f1a000f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:00:57.646 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:57.644+0000 7f1a04e6a700 1 -- 172.21.15.155:0/331421436 shutdown_connections 2024-02-10T09:00:57.646 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:57.644+0000 7f1a04e6a700 1 -- 172.21.15.155:0/331421436 wait complete. 2024-02-10T09:00:57.646 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:57.644+0000 7f1a04e6a700 1 Processor -- start 2024-02-10T09:00:57.646 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:57.644+0000 7f1a04e6a700 1 -- start start 2024-02-10T09:00:57.646 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:57.644+0000 7f1a04e6a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a000f62a0 0x7f1a001065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:00:57.646 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:57.644+0000 7f1a04e6a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a00106ac0 con 0x7f1a000f62a0 2024-02-10T09:00:57.646 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:57.645+0000 7f19ff7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a000f62a0 0x7f1a001065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:00:57.646 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:57.645+0000 7f19ff7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a000f62a0 0x7f1a001065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:38290/0 (socket says 172.21.15.155:38290) 2024-02-10T09:00:57.646 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:57.645+0000 7f19ff7fe700 1 -- 172.21.15.155:0/3001084040 learned_addr learned my addr 172.21.15.155:0/3001084040 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:00:57.646 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:57.645+0000 7f19ff7fe700 1 -- 172.21.15.155:0/3001084040 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19f0005040 con 0x7f1a000f62a0 2024-02-10T09:00:57.646 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:57.645+0000 7f19ff7fe700 1 --2- 172.21.15.155:0/3001084040 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a000f62a0 0x7f1a001065d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f19f00089f0 tx=0x7f19f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:00:57.647 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:57.645+0000 7f19e7fff700 1 -- 172.21.15.155:0/3001084040 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f19f0009320 con 0x7f1a000f62a0 2024-02-10T09:00:57.647 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:57.645+0000 7f1a04e6a700 1 -- 172.21.15.155:0/3001084040 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a00106cc0 con 0x7f1a000f62a0 2024-02-10T09:00:57.647 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:57.645+0000 7f19e7fff700 1 -- 172.21.15.155:0/3001084040 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f19f0013380 con 0x7f1a000f62a0 2024-02-10T09:00:57.647 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:57.645+0000 7f19e7fff700 1 -- 172.21.15.155:0/3001084040 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f19f0014e20 con 0x7f1a000f62a0 2024-02-10T09:00:57.647 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:57.645+0000 7f1a04e6a700 1 -- 172.21.15.155:0/3001084040 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a00107150 con 0x7f1a000f62a0 2024-02-10T09:00:57.648 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:57.646+0000 7f19e7fff700 1 -- 172.21.15.155:0/3001084040 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f19f00095f0 con 0x7f1a000f62a0 2024-02-10T09:00:57.648 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:57.646+0000 7f1a04e6a700 1 -- 172.21.15.155:0/3001084040 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a00045ad0 con 0x7f1a000f62a0 2024-02-10T09:00:57.648 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:57.646+0000 7f19e7fff700 1 --2- 172.21.15.155:0/3001084040 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f19e802ff10 0x7f19e80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:00:57.648 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:57.646+0000 7f19e7fff700 1 -- 172.21.15.155:0/3001084040 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f19f0040f50 con 0x7f1a000f62a0 2024-02-10T09:00:57.648 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:57.646+0000 7f19feffd700 1 --2- 172.21.15.155:0/3001084040 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f19e802ff10 0x7f19e80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:00:57.648 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:57.647+0000 7f19feffd700 1 --2- 172.21.15.155:0/3001084040 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f19e802ff10 0x7f19e80323d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f19f40096a0 tx=0x7f19f4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:00:57.651 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:57.651+0000 7f19e7fff700 1 -- 172.21.15.155:0/3001084040 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f19f0014420 con 0x7f1a000f62a0 2024-02-10T09:00:57.855 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:57.855+0000 7f1a04e6a700 1 -- 172.21.15.155:0/3001084040 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1a00108000 con 0x7f1a000f62a0 2024-02-10T09:00:57.855 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:57.856+0000 7f19e7fff700 1 -- 172.21.15.155:0/3001084040 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f19f001d980 con 0x7f1a000f62a0 2024-02-10T09:00:57.856 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:00:57.856 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:00:57.857 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:57.857+0000 7f1a04e6a700 1 -- 172.21.15.155:0/3001084040 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f19e802ff10 msgr2=0x7f19e80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:00:57.857 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:57.857+0000 7f1a04e6a700 1 --2- 172.21.15.155:0/3001084040 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f19e802ff10 0x7f19e80323d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f19f40096a0 tx=0x7f19f4006b40).stop 2024-02-10T09:00:57.857 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:57.857+0000 7f1a04e6a700 1 -- 172.21.15.155:0/3001084040 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a000f62a0 msgr2=0x7f1a001065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:00:57.857 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:57.857+0000 7f1a04e6a700 1 --2- 172.21.15.155:0/3001084040 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a000f62a0 0x7f1a001065d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f19f00089f0 tx=0x7f19f000b040).stop 2024-02-10T09:00:57.858 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:57.857+0000 7f1a04e6a700 1 -- 172.21.15.155:0/3001084040 shutdown_connections 2024-02-10T09:00:57.858 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:57.857+0000 7f1a04e6a700 1 --2- 172.21.15.155:0/3001084040 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f19e802ff10 0x7f19e80323d0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:00:57.858 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:57.857+0000 7f1a04e6a700 1 --2- 172.21.15.155:0/3001084040 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a000f62a0 0x7f1a001065d0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:00:57.858 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:57.857+0000 7f1a04e6a700 1 -- 172.21.15.155:0/3001084040 >> 172.21.15.155:0/3001084040 conn(0x7f1a000f1aa0 msgr2=0x7f1a000f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:00:57.858 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:57.857+0000 7f1a04e6a700 1 -- 172.21.15.155:0/3001084040 shutdown_connections 2024-02-10T09:00:57.858 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:00:57.857+0000 7f1a04e6a700 1 -- 172.21.15.155:0/3001084040 wait complete. 2024-02-10T09:00:57.858 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:00:59.148 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:00:58 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:00:57.856149+0000 mon.smithi077 (mon.0) 144 : audit [DBG] from='client.? 172.21.15.155:0/3001084040' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:00:59.628 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:00:59.629 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:01:00.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:00:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:00:58.974981+0000 mgr.smithi077.dxntvw (mgr.14162) 67 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:01:01.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:01:01 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:01:00.975286+0000 mgr.smithi077.dxntvw (mgr.14162) 68 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:01:01.611 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:01.609+0000 7fb56f8fc700 1 -- 172.21.15.155:0/2469988610 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5680f5410 msgr2=0x7fb5680f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:01:01.611 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:01.609+0000 7fb56f8fc700 1 --2- 172.21.15.155:0/2469988610 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5680f5410 0x7fb5680f5830 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fb558009fb0 tx=0x7fb558009b30).stop 2024-02-10T09:01:01.611 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:01.609+0000 7fb56f8fc700 1 -- 172.21.15.155:0/2469988610 shutdown_connections 2024-02-10T09:01:01.611 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:01.609+0000 7fb56f8fc700 1 --2- 172.21.15.155:0/2469988610 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5680f5410 0x7fb5680f5830 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:01:01.611 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:01.609+0000 7fb56f8fc700 1 -- 172.21.15.155:0/2469988610 >> 172.21.15.155:0/2469988610 conn(0x7fb5680f0b70 msgr2=0x7fb5680f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:01:01.611 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:01.609+0000 7fb56f8fc700 1 -- 172.21.15.155:0/2469988610 shutdown_connections 2024-02-10T09:01:01.611 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:01.609+0000 7fb56f8fc700 1 -- 172.21.15.155:0/2469988610 wait complete. 2024-02-10T09:01:01.611 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:01.610+0000 7fb56f8fc700 1 Processor -- start 2024-02-10T09:01:01.611 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:01.610+0000 7fb56f8fc700 1 -- start start 2024-02-10T09:01:01.612 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:01.610+0000 7fb56f8fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5680f5410 0x7fb56810b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:01:01.612 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:01.610+0000 7fb56f8fc700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5681072f0 con 0x7fb5680f5410 2024-02-10T09:01:01.612 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:01.610+0000 7fb56d698700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5680f5410 0x7fb56810b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:01:01.612 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:01.610+0000 7fb56d698700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5680f5410 0x7fb56810b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:38318/0 (socket says 172.21.15.155:38318) 2024-02-10T09:01:01.612 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:01.610+0000 7fb56d698700 1 -- 172.21.15.155:0/403458526 learned_addr learned my addr 172.21.15.155:0/403458526 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:01:01.612 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:01.611+0000 7fb56d698700 1 -- 172.21.15.155:0/403458526 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb558005040 con 0x7fb5680f5410 2024-02-10T09:01:01.612 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:01.611+0000 7fb56d698700 1 --2- 172.21.15.155:0/403458526 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5680f5410 0x7fb56810b900 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fb558005010 tx=0x7fb55800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:01:01.612 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:01.611+0000 7fb55dffb700 1 -- 172.21.15.155:0/403458526 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb558009320 con 0x7fb5680f5410 2024-02-10T09:01:01.612 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:01.611+0000 7fb55dffb700 1 -- 172.21.15.155:0/403458526 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb558013380 con 0x7fb5680f5410 2024-02-10T09:01:01.612 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:01.611+0000 7fb55dffb700 1 -- 172.21.15.155:0/403458526 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb558014e50 con 0x7fb5680f5410 2024-02-10T09:01:01.612 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:01.611+0000 7fb56f8fc700 1 -- 172.21.15.155:0/403458526 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb568060220 con 0x7fb5680f5410 2024-02-10T09:01:01.613 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:01.611+0000 7fb56f8fc700 1 -- 172.21.15.155:0/403458526 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb56810bdf0 con 0x7fb5680f5410 2024-02-10T09:01:01.613 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:01.612+0000 7fb56f8fc700 1 -- 172.21.15.155:0/403458526 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb568045ad0 con 0x7fb5680f5410 2024-02-10T09:01:01.613 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:01.612+0000 7fb55dffb700 1 -- 172.21.15.155:0/403458526 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb5580095f0 con 0x7fb5680f5410 2024-02-10T09:01:01.613 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:01.612+0000 7fb55dffb700 1 --2- 172.21.15.155:0/403458526 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb55402ff10 0x7fb5540323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:01:01.613 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:01.612+0000 7fb55dffb700 1 -- 172.21.15.155:0/403458526 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb558040f70 con 0x7fb5680f5410 2024-02-10T09:01:01.614 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:01.613+0000 7fb56ce97700 1 --2- 172.21.15.155:0/403458526 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb55402ff10 0x7fb5540323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:01:01.614 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:01.613+0000 7fb56ce97700 1 --2- 172.21.15.155:0/403458526 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb55402ff10 0x7fb5540323d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fb5640096a0 tx=0x7fb564006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:01:01.617 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:01.617+0000 7fb55dffb700 1 -- 172.21.15.155:0/403458526 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb558014420 con 0x7fb5680f5410 2024-02-10T09:01:01.823 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:01.823+0000 7fb56f8fc700 1 -- 172.21.15.155:0/403458526 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb56810c5b0 con 0x7fb5680f5410 2024-02-10T09:01:01.824 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:01.824+0000 7fb55dffb700 1 -- 172.21.15.155:0/403458526 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb55801d980 con 0x7fb5680f5410 2024-02-10T09:01:01.824 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:01:01.824 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:01:01.826 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:01.825+0000 7fb56f8fc700 1 -- 172.21.15.155:0/403458526 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb55402ff10 msgr2=0x7fb5540323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:01:01.826 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:01.825+0000 7fb56f8fc700 1 --2- 172.21.15.155:0/403458526 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb55402ff10 0x7fb5540323d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fb5640096a0 tx=0x7fb564006b40).stop 2024-02-10T09:01:01.826 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:01.825+0000 7fb56f8fc700 1 -- 172.21.15.155:0/403458526 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5680f5410 msgr2=0x7fb56810b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:01:01.826 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:01.825+0000 7fb56f8fc700 1 --2- 172.21.15.155:0/403458526 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5680f5410 0x7fb56810b900 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fb558005010 tx=0x7fb55800b040).stop 2024-02-10T09:01:01.826 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:01.826+0000 7fb56f8fc700 1 -- 172.21.15.155:0/403458526 shutdown_connections 2024-02-10T09:01:01.826 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:01.826+0000 7fb56f8fc700 1 --2- 172.21.15.155:0/403458526 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb55402ff10 0x7fb5540323d0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:01:01.826 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:01.826+0000 7fb56f8fc700 1 --2- 172.21.15.155:0/403458526 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5680f5410 0x7fb56810b900 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:01:01.826 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:01.826+0000 7fb56f8fc700 1 -- 172.21.15.155:0/403458526 >> 172.21.15.155:0/403458526 conn(0x7fb5680f0b70 msgr2=0x7fb5680f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:01:01.826 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:01.826+0000 7fb56f8fc700 1 -- 172.21.15.155:0/403458526 shutdown_connections 2024-02-10T09:01:01.827 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:01.826+0000 7fb56f8fc700 1 -- 172.21.15.155:0/403458526 wait complete. 2024-02-10T09:01:01.827 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:01:02.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:01:02 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:01:01.824243+0000 mon.smithi077 (mon.0) 145 : audit [DBG] from='client.? 172.21.15.155:0/403458526' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:01:03.597 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:01:03.597 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:01:04.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:01:03 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:01:02.975589+0000 mgr.smithi077.dxntvw (mgr.14162) 69 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:01:05.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:01:05 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:01:04.975891+0000 mgr.smithi077.dxntvw (mgr.14162) 70 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:01:05.620 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:05.570+0000 7fec95b82700 1 -- 172.21.15.155:0/4006560229 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec900f7ae0 msgr2=0x7fec900f7f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:01:05.620 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:05.570+0000 7fec95b82700 1 --2- 172.21.15.155:0/4006560229 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec900f7ae0 0x7fec900f7f00 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7fec80009fb0 tx=0x7fec80009b30).stop 2024-02-10T09:01:05.621 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:05.570+0000 7fec95b82700 1 -- 172.21.15.155:0/4006560229 shutdown_connections 2024-02-10T09:01:05.621 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:05.570+0000 7fec95b82700 1 --2- 172.21.15.155:0/4006560229 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec900f7ae0 0x7fec900f7f00 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:01:05.621 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:05.570+0000 7fec95b82700 1 -- 172.21.15.155:0/4006560229 >> 172.21.15.155:0/4006560229 conn(0x7fec900f0260 msgr2=0x7fec900f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:01:05.621 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:05.570+0000 7fec95b82700 1 -- 172.21.15.155:0/4006560229 shutdown_connections 2024-02-10T09:01:05.621 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:05.570+0000 7fec95b82700 1 -- 172.21.15.155:0/4006560229 wait complete. 2024-02-10T09:01:05.621 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:05.570+0000 7fec95b82700 1 Processor -- start 2024-02-10T09:01:05.621 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:05.571+0000 7fec95b82700 1 -- start start 2024-02-10T09:01:05.621 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:05.571+0000 7fec95b82700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec900f7ae0 0x7fec900f73b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:01:05.621 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:05.571+0000 7fec95b82700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec900f5830 con 0x7fec900f7ae0 2024-02-10T09:01:05.621 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:05.571+0000 7fec8f7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec900f7ae0 0x7fec900f73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:01:05.621 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:05.571+0000 7fec8f7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec900f7ae0 0x7fec900f73b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:53192/0 (socket says 172.21.15.155:53192) 2024-02-10T09:01:05.621 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:05.571+0000 7fec8f7fe700 1 -- 172.21.15.155:0/1866933109 learned_addr learned my addr 172.21.15.155:0/1866933109 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:01:05.621 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:05.571+0000 7fec8f7fe700 1 -- 172.21.15.155:0/1866933109 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec80005040 con 0x7fec900f7ae0 2024-02-10T09:01:05.621 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:05.572+0000 7fec8f7fe700 1 --2- 172.21.15.155:0/1866933109 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec900f7ae0 0x7fec900f73b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fec80005010 tx=0x7fec8000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:01:05.622 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:05.572+0000 7fec87fff700 1 -- 172.21.15.155:0/1866933109 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec80009320 con 0x7fec900f7ae0 2024-02-10T09:01:05.622 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:05.572+0000 7fec95b82700 1 -- 172.21.15.155:0/1866933109 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec900f59d0 con 0x7fec900f7ae0 2024-02-10T09:01:05.622 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:05.572+0000 7fec87fff700 1 -- 172.21.15.155:0/1866933109 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fec80013380 con 0x7fec900f7ae0 2024-02-10T09:01:05.622 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:05.572+0000 7fec87fff700 1 -- 172.21.15.155:0/1866933109 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec80014e50 con 0x7fec900f7ae0 2024-02-10T09:01:05.622 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:05.572+0000 7fec95b82700 1 -- 172.21.15.155:0/1866933109 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec900f5e60 con 0x7fec900f7ae0 2024-02-10T09:01:05.622 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:05.573+0000 7fec87fff700 1 -- 172.21.15.155:0/1866933109 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fec800263a0 con 0x7fec900f7ae0 2024-02-10T09:01:05.622 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:05.573+0000 7fec87fff700 1 --2- 172.21.15.155:0/1866933109 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fec7c02ff10 0x7fec7c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:01:05.622 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:05.573+0000 7fec87fff700 1 -- 172.21.15.155:0/1866933109 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fec80040fd0 con 0x7fec900f7ae0 2024-02-10T09:01:05.622 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:05.573+0000 7fec95b82700 1 -- 172.21.15.155:0/1866933109 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec90059100 con 0x7fec900f7ae0 2024-02-10T09:01:05.622 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:05.573+0000 7fec8effd700 1 --2- 172.21.15.155:0/1866933109 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fec7c02ff10 0x7fec7c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:01:05.622 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:05.573+0000 7fec8effd700 1 --2- 172.21.15.155:0/1866933109 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fec7c02ff10 0x7fec7c0323d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fec780096a0 tx=0x7fec78006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:01:05.622 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:05.578+0000 7fec87fff700 1 -- 172.21.15.155:0/1866933109 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fec80012920 con 0x7fec900f7ae0 2024-02-10T09:01:05.783 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:05.782+0000 7fec95b82700 1 -- 172.21.15.155:0/1866933109 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fec900f6560 con 0x7fec900f7ae0 2024-02-10T09:01:05.784 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:05.783+0000 7fec87fff700 1 -- 172.21.15.155:0/1866933109 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fec80012330 con 0x7fec900f7ae0 2024-02-10T09:01:05.784 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:01:05.784 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:01:05.786 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:05.785+0000 7fec95b82700 1 -- 172.21.15.155:0/1866933109 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fec7c02ff10 msgr2=0x7fec7c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:01:05.786 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:05.785+0000 7fec95b82700 1 --2- 172.21.15.155:0/1866933109 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fec7c02ff10 0x7fec7c0323d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fec780096a0 tx=0x7fec78006b40).stop 2024-02-10T09:01:05.786 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:05.785+0000 7fec95b82700 1 -- 172.21.15.155:0/1866933109 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec900f7ae0 msgr2=0x7fec900f73b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:01:05.786 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:05.785+0000 7fec95b82700 1 --2- 172.21.15.155:0/1866933109 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec900f7ae0 0x7fec900f73b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7fec80005010 tx=0x7fec8000b040).stop 2024-02-10T09:01:05.786 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:05.785+0000 7fec95b82700 1 -- 172.21.15.155:0/1866933109 shutdown_connections 2024-02-10T09:01:05.786 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:05.785+0000 7fec95b82700 1 --2- 172.21.15.155:0/1866933109 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fec7c02ff10 0x7fec7c0323d0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:01:05.786 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:05.785+0000 7fec95b82700 1 --2- 172.21.15.155:0/1866933109 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fec900f7ae0 0x7fec900f73b0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:01:05.786 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:05.785+0000 7fec95b82700 1 -- 172.21.15.155:0/1866933109 >> 172.21.15.155:0/1866933109 conn(0x7fec900f0260 msgr2=0x7fec900f0d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:01:05.786 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:05.785+0000 7fec95b82700 1 -- 172.21.15.155:0/1866933109 shutdown_connections 2024-02-10T09:01:05.786 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:05.785+0000 7fec95b82700 1 -- 172.21.15.155:0/1866933109 wait complete. 2024-02-10T09:01:05.787 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:01:06.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:01:06 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:01:05.784548+0000 mon.smithi077 (mon.0) 146 : audit [DBG] from='client.? 172.21.15.155:0/1866933109' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:01:07.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:01:07 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:01:06.976127+0000 mgr.smithi077.dxntvw (mgr.14162) 71 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:01:07.516 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:01:07.516 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:01:09.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:01:09 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:01:08.976420+0000 mgr.smithi077.dxntvw (mgr.14162) 72 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:01:09.499 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:09.457+0000 7fb95e138700 1 -- 172.21.15.155:0/791698054 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9580f6510 msgr2=0x7fb9580f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:01:09.499 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:09.457+0000 7fb95e138700 1 --2- 172.21.15.155:0/791698054 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9580f6510 0x7fb9580f6930 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7fb94c009fb0 tx=0x7fb94c009b30).stop 2024-02-10T09:01:09.499 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:09.457+0000 7fb95e138700 1 -- 172.21.15.155:0/791698054 shutdown_connections 2024-02-10T09:01:09.499 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:09.457+0000 7fb95e138700 1 --2- 172.21.15.155:0/791698054 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9580f6510 0x7fb9580f6930 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:01:09.499 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:09.457+0000 7fb95e138700 1 -- 172.21.15.155:0/791698054 >> 172.21.15.155:0/791698054 conn(0x7fb9580f1c70 msgr2=0x7fb9580f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:01:09.499 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:09.457+0000 7fb95e138700 1 -- 172.21.15.155:0/791698054 shutdown_connections 2024-02-10T09:01:09.500 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:09.457+0000 7fb95e138700 1 -- 172.21.15.155:0/791698054 wait complete. 2024-02-10T09:01:09.500 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:09.458+0000 7fb95e138700 1 Processor -- start 2024-02-10T09:01:09.500 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:09.458+0000 7fb95e138700 1 -- start start 2024-02-10T09:01:09.500 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:09.458+0000 7fb95e138700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9580f6510 0x7fb9581067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:01:09.500 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:09.458+0000 7fb95e138700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb958106cd0 con 0x7fb9580f6510 2024-02-10T09:01:09.500 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:09.458+0000 7fb9577fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9580f6510 0x7fb9581067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:01:09.500 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:09.458+0000 7fb9577fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9580f6510 0x7fb9581067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:53210/0 (socket says 172.21.15.155:53210) 2024-02-10T09:01:09.500 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:09.458+0000 7fb9577fe700 1 -- 172.21.15.155:0/1278560883 learned_addr learned my addr 172.21.15.155:0/1278560883 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:01:09.500 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:09.459+0000 7fb9577fe700 1 -- 172.21.15.155:0/1278560883 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb94c005040 con 0x7fb9580f6510 2024-02-10T09:01:09.500 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:09.459+0000 7fb9577fe700 1 --2- 172.21.15.155:0/1278560883 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9580f6510 0x7fb9581067e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fb94c005010 tx=0x7fb94c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:01:09.500 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:09.459+0000 7fb93ffff700 1 -- 172.21.15.155:0/1278560883 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb94c009320 con 0x7fb9580f6510 2024-02-10T09:01:09.500 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:09.459+0000 7fb95e138700 1 -- 172.21.15.155:0/1278560883 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb958106ed0 con 0x7fb9580f6510 2024-02-10T09:01:09.500 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:09.459+0000 7fb93ffff700 1 -- 172.21.15.155:0/1278560883 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb94c013380 con 0x7fb9580f6510 2024-02-10T09:01:09.501 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:09.459+0000 7fb93ffff700 1 -- 172.21.15.155:0/1278560883 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb94c014e50 con 0x7fb9580f6510 2024-02-10T09:01:09.501 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:09.459+0000 7fb95e138700 1 -- 172.21.15.155:0/1278560883 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb958107360 con 0x7fb9580f6510 2024-02-10T09:01:09.501 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:09.460+0000 7fb93ffff700 1 -- 172.21.15.155:0/1278560883 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb94c0263a0 con 0x7fb9580f6510 2024-02-10T09:01:09.501 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:09.460+0000 7fb93ffff700 1 --2- 172.21.15.155:0/1278560883 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb94002ff10 0x7fb9400323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:01:09.501 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:09.460+0000 7fb93ffff700 1 -- 172.21.15.155:0/1278560883 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb94c040f10 con 0x7fb9580f6510 2024-02-10T09:01:09.501 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:09.460+0000 7fb95e138700 1 -- 172.21.15.155:0/1278560883 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb958045ad0 con 0x7fb9580f6510 2024-02-10T09:01:09.501 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:09.460+0000 7fb956ffd700 1 --2- 172.21.15.155:0/1278560883 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb94002ff10 0x7fb9400323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:01:09.501 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:09.461+0000 7fb956ffd700 1 --2- 172.21.15.155:0/1278560883 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb94002ff10 0x7fb9400323d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fb9480096a0 tx=0x7fb948006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:01:09.501 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:09.465+0000 7fb93ffff700 1 -- 172.21.15.155:0/1278560883 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb94c012920 con 0x7fb9580f6510 2024-02-10T09:01:09.673 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:09.671+0000 7fb95e138700 1 -- 172.21.15.155:0/1278560883 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb9581080b0 con 0x7fb9580f6510 2024-02-10T09:01:09.674 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:09.671+0000 7fb93ffff700 1 -- 172.21.15.155:0/1278560883 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb94c02b020 con 0x7fb9580f6510 2024-02-10T09:01:09.674 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:01:09.674 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:01:09.676 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:09.673+0000 7fb95e138700 1 -- 172.21.15.155:0/1278560883 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb94002ff10 msgr2=0x7fb9400323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:01:09.676 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:09.673+0000 7fb95e138700 1 --2- 172.21.15.155:0/1278560883 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb94002ff10 0x7fb9400323d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fb9480096a0 tx=0x7fb948006b40).stop 2024-02-10T09:01:09.676 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:09.673+0000 7fb95e138700 1 -- 172.21.15.155:0/1278560883 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9580f6510 msgr2=0x7fb9581067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:01:09.676 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:09.673+0000 7fb95e138700 1 --2- 172.21.15.155:0/1278560883 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9580f6510 0x7fb9581067e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fb94c005010 tx=0x7fb94c00b040).stop 2024-02-10T09:01:09.676 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:09.673+0000 7fb95e138700 1 -- 172.21.15.155:0/1278560883 shutdown_connections 2024-02-10T09:01:09.676 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:09.673+0000 7fb95e138700 1 --2- 172.21.15.155:0/1278560883 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb94002ff10 0x7fb9400323d0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:01:09.676 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:09.673+0000 7fb95e138700 1 --2- 172.21.15.155:0/1278560883 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9580f6510 0x7fb9581067e0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:01:09.676 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:09.673+0000 7fb95e138700 1 -- 172.21.15.155:0/1278560883 >> 172.21.15.155:0/1278560883 conn(0x7fb9580f1c70 msgr2=0x7fb9580f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:01:09.676 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:09.673+0000 7fb95e138700 1 -- 172.21.15.155:0/1278560883 shutdown_connections 2024-02-10T09:01:09.676 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:09.673+0000 7fb95e138700 1 -- 172.21.15.155:0/1278560883 wait complete. 2024-02-10T09:01:09.676 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:01:10.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:01:10 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:01:09.672839+0000 mon.smithi077 (mon.0) 147 : audit [DBG] from='client.? 172.21.15.155:0/1278560883' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:01:11.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:01:11 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:01:10.976644+0000 mgr.smithi077.dxntvw (mgr.14162) 73 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:01:11.408 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:01:11.408 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:01:13.305 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:13.304+0000 7f66d3e3a700 1 -- 172.21.15.155:0/2569575832 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f66cc0f6510 msgr2=0x7f66cc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:01:13.306 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:13.304+0000 7f66d3e3a700 1 --2- 172.21.15.155:0/2569575832 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f66cc0f6510 0x7f66cc0f6930 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f66bc009fb0 tx=0x7f66bc009b30).stop 2024-02-10T09:01:13.306 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:13.304+0000 7f66d3e3a700 1 -- 172.21.15.155:0/2569575832 shutdown_connections 2024-02-10T09:01:13.306 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:13.304+0000 7f66d3e3a700 1 --2- 172.21.15.155:0/2569575832 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f66cc0f6510 0x7f66cc0f6930 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:01:13.306 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:13.304+0000 7f66d3e3a700 1 -- 172.21.15.155:0/2569575832 >> 172.21.15.155:0/2569575832 conn(0x7f66cc0f1c70 msgr2=0x7f66cc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:01:13.306 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:13.304+0000 7f66d3e3a700 1 -- 172.21.15.155:0/2569575832 shutdown_connections 2024-02-10T09:01:13.306 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:13.304+0000 7f66d3e3a700 1 -- 172.21.15.155:0/2569575832 wait complete. 2024-02-10T09:01:13.308 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:13.305+0000 7f66d3e3a700 1 Processor -- start 2024-02-10T09:01:13.308 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:13.305+0000 7f66d3e3a700 1 -- start start 2024-02-10T09:01:13.308 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:13.305+0000 7f66d3e3a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f66cc0f6510 0x7f66cc1089e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:01:13.308 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:13.305+0000 7f66d3e3a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66cc108ed0 con 0x7f66cc0f6510 2024-02-10T09:01:13.308 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:13.305+0000 7f66d1bd6700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f66cc0f6510 0x7f66cc1089e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:01:13.308 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:13.305+0000 7f66d1bd6700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f66cc0f6510 0x7f66cc1089e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:33572/0 (socket says 172.21.15.155:33572) 2024-02-10T09:01:13.308 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:13.305+0000 7f66d1bd6700 1 -- 172.21.15.155:0/4052038259 learned_addr learned my addr 172.21.15.155:0/4052038259 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:01:13.308 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:13.306+0000 7f66d1bd6700 1 -- 172.21.15.155:0/4052038259 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66bc005040 con 0x7f66cc0f6510 2024-02-10T09:01:13.308 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:13.306+0000 7f66d1bd6700 1 --2- 172.21.15.155:0/4052038259 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f66cc0f6510 0x7f66cc1089e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f66bc0089f0 tx=0x7f66bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:01:13.308 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:13.306+0000 7f66c27fc700 1 -- 172.21.15.155:0/4052038259 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66bc009320 con 0x7f66cc0f6510 2024-02-10T09:01:13.309 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:13.306+0000 7f66d3e3a700 1 -- 172.21.15.155:0/4052038259 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66cc1090d0 con 0x7f66cc0f6510 2024-02-10T09:01:13.309 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:13.306+0000 7f66c27fc700 1 -- 172.21.15.155:0/4052038259 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f66bc013380 con 0x7f66cc0f6510 2024-02-10T09:01:13.309 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:13.306+0000 7f66c27fc700 1 -- 172.21.15.155:0/4052038259 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66bc01ed80 con 0x7f66cc0f6510 2024-02-10T09:01:13.309 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:13.306+0000 7f66d3e3a700 1 -- 172.21.15.155:0/4052038259 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66cc109560 con 0x7f66cc0f6510 2024-02-10T09:01:13.309 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:13.307+0000 7f66c27fc700 1 -- 172.21.15.155:0/4052038259 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f66bc01e420 con 0x7f66cc0f6510 2024-02-10T09:01:13.309 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:13.307+0000 7f66d3e3a700 1 -- 172.21.15.155:0/4052038259 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66cc045ad0 con 0x7f66cc0f6510 2024-02-10T09:01:13.309 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:13.307+0000 7f66c27fc700 1 --2- 172.21.15.155:0/4052038259 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f66b802ff60 0x7f66b8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:01:13.309 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:13.307+0000 7f66c27fc700 1 -- 172.21.15.155:0/4052038259 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f66bc040f30 con 0x7f66cc0f6510 2024-02-10T09:01:13.310 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:13.307+0000 7f66d13d5700 1 --2- 172.21.15.155:0/4052038259 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f66b802ff60 0x7f66b8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:01:13.310 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:13.308+0000 7f66d13d5700 1 --2- 172.21.15.155:0/4052038259 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f66b802ff60 0x7f66b8032420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f66c80096a0 tx=0x7f66c8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:01:13.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:13.312+0000 7f66c27fc700 1 -- 172.21.15.155:0/4052038259 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f66bc00e2a0 con 0x7f66cc0f6510 2024-02-10T09:01:13.520 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:13.519+0000 7f66d3e3a700 1 -- 172.21.15.155:0/4052038259 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f66cc10a410 con 0x7f66cc0f6510 2024-02-10T09:01:13.520 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:13.519+0000 7f66c27fc700 1 -- 172.21.15.155:0/4052038259 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f66bc00e2a0 con 0x7f66cc0f6510 2024-02-10T09:01:13.520 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:01:13.520 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:01:13.522 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:13.521+0000 7f66d3e3a700 1 -- 172.21.15.155:0/4052038259 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f66b802ff60 msgr2=0x7f66b8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:01:13.522 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:13.521+0000 7f66d3e3a700 1 --2- 172.21.15.155:0/4052038259 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f66b802ff60 0x7f66b8032420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f66c80096a0 tx=0x7f66c8006b40).stop 2024-02-10T09:01:13.522 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:13.521+0000 7f66d3e3a700 1 -- 172.21.15.155:0/4052038259 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f66cc0f6510 msgr2=0x7f66cc1089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:01:13.522 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:13.521+0000 7f66d3e3a700 1 --2- 172.21.15.155:0/4052038259 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f66cc0f6510 0x7f66cc1089e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f66bc0089f0 tx=0x7f66bc00b040).stop 2024-02-10T09:01:13.522 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:13.521+0000 7f66d3e3a700 1 -- 172.21.15.155:0/4052038259 shutdown_connections 2024-02-10T09:01:13.522 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:13.521+0000 7f66d3e3a700 1 --2- 172.21.15.155:0/4052038259 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f66b802ff60 0x7f66b8032420 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:01:13.522 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:13.521+0000 7f66d3e3a700 1 --2- 172.21.15.155:0/4052038259 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f66cc0f6510 0x7f66cc1089e0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:01:13.522 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:13.521+0000 7f66d3e3a700 1 -- 172.21.15.155:0/4052038259 >> 172.21.15.155:0/4052038259 conn(0x7f66cc0f1c70 msgr2=0x7f66cc0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:01:13.522 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:13.521+0000 7f66d3e3a700 1 -- 172.21.15.155:0/4052038259 shutdown_connections 2024-02-10T09:01:13.522 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:13.521+0000 7f66d3e3a700 1 -- 172.21.15.155:0/4052038259 wait complete. 2024-02-10T09:01:13.523 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:01:14.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:01:13 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:01:12.976958+0000 mgr.smithi077.dxntvw (mgr.14162) 74 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:01:14.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:01:13 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:01:13.520738+0000 mon.smithi077 (mon.0) 148 : audit [DBG] from='client.? 172.21.15.155:0/4052038259' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:01:15.135 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:01:15.135 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:01:15.398 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:01:15 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:01:14.977257+0000 mgr.smithi077.dxntvw (mgr.14162) 75 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:01:17.294 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:17.291+0000 7ffaaf776700 1 -- 172.21.15.155:0/1114714498 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffaa80f69d0 msgr2=0x7ffaa80f6df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:01:17.294 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:17.291+0000 7ffaaf776700 1 --2- 172.21.15.155:0/1114714498 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffaa80f69d0 0x7ffaa80f6df0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7ffa9c009fb0 tx=0x7ffa9c009b30).stop 2024-02-10T09:01:17.295 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:17.291+0000 7ffaaf776700 1 -- 172.21.15.155:0/1114714498 shutdown_connections 2024-02-10T09:01:17.295 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:17.291+0000 7ffaaf776700 1 --2- 172.21.15.155:0/1114714498 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffaa80f69d0 0x7ffaa80f6df0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:01:17.295 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:17.291+0000 7ffaaf776700 1 -- 172.21.15.155:0/1114714498 >> 172.21.15.155:0/1114714498 conn(0x7ffaa80f02e0 msgr2=0x7ffaa80f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:01:17.295 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:17.291+0000 7ffaaf776700 1 -- 172.21.15.155:0/1114714498 shutdown_connections 2024-02-10T09:01:17.295 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:17.292+0000 7ffaaf776700 1 -- 172.21.15.155:0/1114714498 wait complete. 2024-02-10T09:01:17.295 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:17.292+0000 7ffaaf776700 1 Processor -- start 2024-02-10T09:01:17.295 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:17.292+0000 7ffaaf776700 1 -- start start 2024-02-10T09:01:17.295 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:17.292+0000 7ffaaf776700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffaa80f69d0 0x7ffaa81094d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:01:17.295 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:17.292+0000 7ffaaf776700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa80f5250 con 0x7ffaa80f69d0 2024-02-10T09:01:17.295 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:17.292+0000 7ffaad512700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffaa80f69d0 0x7ffaa81094d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:01:17.295 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:17.292+0000 7ffaad512700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffaa80f69d0 0x7ffaa81094d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:33594/0 (socket says 172.21.15.155:33594) 2024-02-10T09:01:17.295 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:17.292+0000 7ffaad512700 1 -- 172.21.15.155:0/1800214523 learned_addr learned my addr 172.21.15.155:0/1800214523 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:01:17.296 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:17.293+0000 7ffaad512700 1 -- 172.21.15.155:0/1800214523 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa9c005040 con 0x7ffaa80f69d0 2024-02-10T09:01:17.296 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:17.293+0000 7ffaad512700 1 --2- 172.21.15.155:0/1800214523 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffaa80f69d0 0x7ffaa81094d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7ffa9c005010 tx=0x7ffa9c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:01:17.296 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:17.293+0000 7ffa99ffb700 1 -- 172.21.15.155:0/1800214523 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa9c009320 con 0x7ffaa80f69d0 2024-02-10T09:01:17.296 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:17.293+0000 7ffa99ffb700 1 -- 172.21.15.155:0/1800214523 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffa9c013380 con 0x7ffaa80f69d0 2024-02-10T09:01:17.296 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:17.293+0000 7ffaaf776700 1 -- 172.21.15.155:0/1800214523 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaa80f5450 con 0x7ffaa80f69d0 2024-02-10T09:01:17.296 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:17.293+0000 7ffa99ffb700 1 -- 172.21.15.155:0/1800214523 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa9c014e50 con 0x7ffaa80f69d0 2024-02-10T09:01:17.296 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:17.293+0000 7ffaaf776700 1 -- 172.21.15.155:0/1800214523 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaa8109b80 con 0x7ffaa80f69d0 2024-02-10T09:01:17.297 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:17.294+0000 7ffa99ffb700 1 -- 172.21.15.155:0/1800214523 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ffa9c0095f0 con 0x7ffaa80f69d0 2024-02-10T09:01:17.297 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:17.294+0000 7ffaaf776700 1 -- 172.21.15.155:0/1800214523 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaa8059100 con 0x7ffaa80f69d0 2024-02-10T09:01:17.297 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:17.294+0000 7ffa99ffb700 1 --2- 172.21.15.155:0/1800214523 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7ffa9402ff10 0x7ffa940323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:01:17.297 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:17.294+0000 7ffa99ffb700 1 -- 172.21.15.155:0/1800214523 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ffa9c040e70 con 0x7ffaa80f69d0 2024-02-10T09:01:17.297 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:17.294+0000 7ffaacd11700 1 --2- 172.21.15.155:0/1800214523 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7ffa9402ff10 0x7ffa940323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:01:17.297 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:17.295+0000 7ffaacd11700 1 --2- 172.21.15.155:0/1800214523 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7ffa9402ff10 0x7ffa940323d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7ffaa40096a0 tx=0x7ffaa4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:01:17.299 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:17.299+0000 7ffa99ffb700 1 -- 172.21.15.155:0/1800214523 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ffa9c014420 con 0x7ffaa80f69d0 2024-02-10T09:01:17.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:01:17 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:01:16.977478+0000 mgr.smithi077.dxntvw (mgr.14162) 76 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:01:17.508 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:17.507+0000 7ffaaf776700 1 -- 172.21.15.155:0/1800214523 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ffaa80faf90 con 0x7ffaa80f69d0 2024-02-10T09:01:17.509 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:17.508+0000 7ffa99ffb700 1 -- 172.21.15.155:0/1800214523 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ffa9c010230 con 0x7ffaa80f69d0 2024-02-10T09:01:17.509 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:01:17.509 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:01:17.511 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:17.509+0000 7ffaaf776700 1 -- 172.21.15.155:0/1800214523 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7ffa9402ff10 msgr2=0x7ffa940323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:01:17.511 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:17.509+0000 7ffaaf776700 1 --2- 172.21.15.155:0/1800214523 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7ffa9402ff10 0x7ffa940323d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7ffaa40096a0 tx=0x7ffaa4006b40).stop 2024-02-10T09:01:17.511 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:17.509+0000 7ffaaf776700 1 -- 172.21.15.155:0/1800214523 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffaa80f69d0 msgr2=0x7ffaa81094d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:01:17.511 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:17.509+0000 7ffaaf776700 1 --2- 172.21.15.155:0/1800214523 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffaa80f69d0 0x7ffaa81094d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7ffa9c005010 tx=0x7ffa9c00b040).stop 2024-02-10T09:01:17.511 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:17.510+0000 7ffaaf776700 1 -- 172.21.15.155:0/1800214523 shutdown_connections 2024-02-10T09:01:17.511 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:17.510+0000 7ffaaf776700 1 --2- 172.21.15.155:0/1800214523 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7ffa9402ff10 0x7ffa940323d0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:01:17.511 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:17.510+0000 7ffaaf776700 1 --2- 172.21.15.155:0/1800214523 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffaa80f69d0 0x7ffaa81094d0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:01:17.511 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:17.510+0000 7ffaaf776700 1 -- 172.21.15.155:0/1800214523 >> 172.21.15.155:0/1800214523 conn(0x7ffaa80f02e0 msgr2=0x7ffaa80f0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:01:17.511 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:17.510+0000 7ffaaf776700 1 -- 172.21.15.155:0/1800214523 shutdown_connections 2024-02-10T09:01:17.511 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:17.510+0000 7ffaaf776700 1 -- 172.21.15.155:0/1800214523 wait complete. 2024-02-10T09:01:17.512 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:01:18.398 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:01:18 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:01:17.509366+0000 mon.smithi077 (mon.0) 149 : audit [DBG] from='client.? 172.21.15.155:0/1800214523' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:01:19.140 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:01:19.140 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:01:19.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:01:19 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:01:18.977837+0000 mgr.smithi077.dxntvw (mgr.14162) 77 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:01:21.049 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:21.046+0000 7fe51e88b700 1 -- 172.21.15.155:0/1785048327 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5180f6510 msgr2=0x7fe5180f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:01:21.049 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:21.046+0000 7fe51e88b700 1 --2- 172.21.15.155:0/1785048327 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5180f6510 0x7fe5180f6930 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fe50c009fb0 tx=0x7fe50c009b30).stop 2024-02-10T09:01:21.050 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:21.047+0000 7fe51e88b700 1 -- 172.21.15.155:0/1785048327 shutdown_connections 2024-02-10T09:01:21.050 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:21.047+0000 7fe51e88b700 1 --2- 172.21.15.155:0/1785048327 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5180f6510 0x7fe5180f6930 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:01:21.050 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:21.047+0000 7fe51e88b700 1 -- 172.21.15.155:0/1785048327 >> 172.21.15.155:0/1785048327 conn(0x7fe5180f1c70 msgr2=0x7fe5180f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:01:21.050 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:21.047+0000 7fe51e88b700 1 -- 172.21.15.155:0/1785048327 shutdown_connections 2024-02-10T09:01:21.050 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:21.047+0000 7fe51e88b700 1 -- 172.21.15.155:0/1785048327 wait complete. 2024-02-10T09:01:21.050 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:21.047+0000 7fe51e88b700 1 Processor -- start 2024-02-10T09:01:21.050 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:21.047+0000 7fe51e88b700 1 -- start start 2024-02-10T09:01:21.050 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:21.047+0000 7fe51e88b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5180f6510 0x7fe518106820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:01:21.050 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:21.047+0000 7fe51e88b700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe518106d10 con 0x7fe5180f6510 2024-02-10T09:01:21.050 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:21.048+0000 7fe517fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5180f6510 0x7fe518106820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:01:21.050 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:21.048+0000 7fe517fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5180f6510 0x7fe518106820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:33602/0 (socket says 172.21.15.155:33602) 2024-02-10T09:01:21.051 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:21.048+0000 7fe517fff700 1 -- 172.21.15.155:0/2125416256 learned_addr learned my addr 172.21.15.155:0/2125416256 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:01:21.051 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:21.048+0000 7fe517fff700 1 -- 172.21.15.155:0/2125416256 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe50c005040 con 0x7fe5180f6510 2024-02-10T09:01:21.051 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:21.048+0000 7fe517fff700 1 --2- 172.21.15.155:0/2125416256 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5180f6510 0x7fe518106820 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fe50c000c00 tx=0x7fe50c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:01:21.051 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:21.048+0000 7fe514ff9700 1 -- 172.21.15.155:0/2125416256 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe50c009320 con 0x7fe5180f6510 2024-02-10T09:01:21.051 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:21.048+0000 7fe51e88b700 1 -- 172.21.15.155:0/2125416256 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe518106f10 con 0x7fe5180f6510 2024-02-10T09:01:21.052 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:21.048+0000 7fe51e88b700 1 -- 172.21.15.155:0/2125416256 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5181073a0 con 0x7fe5180f6510 2024-02-10T09:01:21.052 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:21.048+0000 7fe514ff9700 1 -- 172.21.15.155:0/2125416256 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe50c013380 con 0x7fe5180f6510 2024-02-10T09:01:21.052 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:21.048+0000 7fe514ff9700 1 -- 172.21.15.155:0/2125416256 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe50c01ec50 con 0x7fe5180f6510 2024-02-10T09:01:21.052 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:21.049+0000 7fe514ff9700 1 -- 172.21.15.155:0/2125416256 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe50c01e420 con 0x7fe5180f6510 2024-02-10T09:01:21.052 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:21.049+0000 7fe51e88b700 1 -- 172.21.15.155:0/2125416256 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe518045ad0 con 0x7fe5180f6510 2024-02-10T09:01:21.052 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:21.049+0000 7fe514ff9700 1 --2- 172.21.15.155:0/2125416256 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fe50002ffb0 0x7fe500032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:01:21.052 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:21.049+0000 7fe514ff9700 1 -- 172.21.15.155:0/2125416256 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe50c040e30 con 0x7fe5180f6510 2024-02-10T09:01:21.052 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:21.050+0000 7fe5177fe700 1 --2- 172.21.15.155:0/2125416256 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fe50002ffb0 0x7fe500032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:01:21.053 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:21.050+0000 7fe5177fe700 1 --2- 172.21.15.155:0/2125416256 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fe50002ffb0 0x7fe500032470 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fe5080096a0 tx=0x7fe508006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:01:21.055 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:21.054+0000 7fe514ff9700 1 -- 172.21.15.155:0/2125416256 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe50c01e790 con 0x7fe5180f6510 2024-02-10T09:01:21.260 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:21.259+0000 7fe51e88b700 1 -- 172.21.15.155:0/2125416256 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe5180fc0a0 con 0x7fe5180f6510 2024-02-10T09:01:21.260 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:21.259+0000 7fe514ff9700 1 -- 172.21.15.155:0/2125416256 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe50c01d980 con 0x7fe5180f6510 2024-02-10T09:01:21.261 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:01:21.261 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:01:21.262 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:21.261+0000 7fe51e88b700 1 -- 172.21.15.155:0/2125416256 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fe50002ffb0 msgr2=0x7fe500032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:01:21.262 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:21.261+0000 7fe51e88b700 1 --2- 172.21.15.155:0/2125416256 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fe50002ffb0 0x7fe500032470 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fe5080096a0 tx=0x7fe508006b40).stop 2024-02-10T09:01:21.262 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:21.261+0000 7fe51e88b700 1 -- 172.21.15.155:0/2125416256 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5180f6510 msgr2=0x7fe518106820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:01:21.263 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:21.261+0000 7fe51e88b700 1 --2- 172.21.15.155:0/2125416256 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5180f6510 0x7fe518106820 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fe50c000c00 tx=0x7fe50c00b040).stop 2024-02-10T09:01:21.263 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:21.261+0000 7fe51e88b700 1 -- 172.21.15.155:0/2125416256 shutdown_connections 2024-02-10T09:01:21.263 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:21.261+0000 7fe51e88b700 1 --2- 172.21.15.155:0/2125416256 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fe50002ffb0 0x7fe500032470 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:01:21.263 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:21.261+0000 7fe51e88b700 1 --2- 172.21.15.155:0/2125416256 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5180f6510 0x7fe518106820 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:01:21.263 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:21.261+0000 7fe51e88b700 1 -- 172.21.15.155:0/2125416256 >> 172.21.15.155:0/2125416256 conn(0x7fe5180f1c70 msgr2=0x7fe5180f27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:01:21.263 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:21.261+0000 7fe51e88b700 1 -- 172.21.15.155:0/2125416256 shutdown_connections 2024-02-10T09:01:21.263 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:21.261+0000 7fe51e88b700 1 -- 172.21.15.155:0/2125416256 wait complete. 2024-02-10T09:01:21.263 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:01:21.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:01:21 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:01:20.978185+0000 mgr.smithi077.dxntvw (mgr.14162) 78 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:01:22.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:01:22 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:01:21.260861+0000 mon.smithi077 (mon.0) 150 : audit [DBG] from='client.? 172.21.15.155:0/2125416256' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:01:23.042 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:01:23.042 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:01:24.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:01:23 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:01:22.978466+0000 mgr.smithi077.dxntvw (mgr.14162) 79 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:01:25.109 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:25.106+0000 7fb9166f3700 1 -- 172.21.15.155:0/616245975 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9100f5410 msgr2=0x7fb9100f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:01:25.110 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:25.106+0000 7fb9166f3700 1 --2- 172.21.15.155:0/616245975 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9100f5410 0x7fb9100f5830 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fb900009fb0 tx=0x7fb900009b30).stop 2024-02-10T09:01:25.110 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:25.106+0000 7fb9166f3700 1 -- 172.21.15.155:0/616245975 shutdown_connections 2024-02-10T09:01:25.110 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:25.106+0000 7fb9166f3700 1 --2- 172.21.15.155:0/616245975 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9100f5410 0x7fb9100f5830 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:01:25.110 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:25.106+0000 7fb9166f3700 1 -- 172.21.15.155:0/616245975 >> 172.21.15.155:0/616245975 conn(0x7fb9100f0b70 msgr2=0x7fb9100f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:01:25.110 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:25.106+0000 7fb9166f3700 1 -- 172.21.15.155:0/616245975 shutdown_connections 2024-02-10T09:01:25.110 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:25.106+0000 7fb9166f3700 1 -- 172.21.15.155:0/616245975 wait complete. 2024-02-10T09:01:25.110 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:25.107+0000 7fb9166f3700 1 Processor -- start 2024-02-10T09:01:25.110 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:25.107+0000 7fb9166f3700 1 -- start start 2024-02-10T09:01:25.110 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:25.107+0000 7fb9166f3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9100f5410 0x7fb9101094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:01:25.110 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:25.107+0000 7fb9166f3700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb910060240 con 0x7fb9100f5410 2024-02-10T09:01:25.110 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:25.107+0000 7fb90ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9100f5410 0x7fb9101094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:01:25.110 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:25.107+0000 7fb90ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9100f5410 0x7fb9101094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:58318/0 (socket says 172.21.15.155:58318) 2024-02-10T09:01:25.110 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:25.107+0000 7fb90ffff700 1 -- 172.21.15.155:0/337557720 learned_addr learned my addr 172.21.15.155:0/337557720 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:01:25.111 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:25.108+0000 7fb90ffff700 1 -- 172.21.15.155:0/337557720 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb900005040 con 0x7fb9100f5410 2024-02-10T09:01:25.111 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:25.108+0000 7fb90ffff700 1 --2- 172.21.15.155:0/337557720 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9100f5410 0x7fb9101094e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fb9000089f0 tx=0x7fb90000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:01:25.111 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:25.108+0000 7fb90cff9700 1 -- 172.21.15.155:0/337557720 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb900009320 con 0x7fb9100f5410 2024-02-10T09:01:25.111 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:25.108+0000 7fb90cff9700 1 -- 172.21.15.155:0/337557720 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb900013380 con 0x7fb9100f5410 2024-02-10T09:01:25.111 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:25.108+0000 7fb9166f3700 1 -- 172.21.15.155:0/337557720 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb910060440 con 0x7fb9100f5410 2024-02-10T09:01:25.111 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:25.108+0000 7fb90cff9700 1 -- 172.21.15.155:0/337557720 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb90001ed80 con 0x7fb9100f5410 2024-02-10T09:01:25.111 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:25.108+0000 7fb9166f3700 1 -- 172.21.15.155:0/337557720 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb910109b90 con 0x7fb9100f5410 2024-02-10T09:01:25.111 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:25.109+0000 7fb90cff9700 1 -- 172.21.15.155:0/337557720 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb900014420 con 0x7fb9100f5410 2024-02-10T09:01:25.112 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:25.109+0000 7fb90cff9700 1 --2- 172.21.15.155:0/337557720 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb8fc02ffb0 0x7fb8fc032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:01:25.112 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:25.109+0000 7fb90cff9700 1 -- 172.21.15.155:0/337557720 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb900040dc0 con 0x7fb9100f5410 2024-02-10T09:01:25.112 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:25.109+0000 7fb9166f3700 1 -- 172.21.15.155:0/337557720 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb910045ad0 con 0x7fb9100f5410 2024-02-10T09:01:25.112 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:25.109+0000 7fb90f7fe700 1 --2- 172.21.15.155:0/337557720 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb8fc02ffb0 0x7fb8fc032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:01:25.112 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:25.110+0000 7fb90f7fe700 1 --2- 172.21.15.155:0/337557720 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb8fc02ffb0 0x7fb8fc032470 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fb8f80096a0 tx=0x7fb8f8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:01:25.114 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:25.114+0000 7fb90cff9700 1 -- 172.21.15.155:0/337557720 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb90001c910 con 0x7fb9100f5410 2024-02-10T09:01:25.323 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:25.322+0000 7fb9166f3700 1 -- 172.21.15.155:0/337557720 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb91010a2a0 con 0x7fb9100f5410 2024-02-10T09:01:25.324 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:25.323+0000 7fb90cff9700 1 -- 172.21.15.155:0/337557720 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb90001a230 con 0x7fb9100f5410 2024-02-10T09:01:25.324 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:01:25.324 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:01:25.325 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:25.324+0000 7fb9166f3700 1 -- 172.21.15.155:0/337557720 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb8fc02ffb0 msgr2=0x7fb8fc032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:01:25.326 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:25.324+0000 7fb9166f3700 1 --2- 172.21.15.155:0/337557720 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb8fc02ffb0 0x7fb8fc032470 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fb8f80096a0 tx=0x7fb8f8006b40).stop 2024-02-10T09:01:25.326 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:25.324+0000 7fb9166f3700 1 -- 172.21.15.155:0/337557720 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9100f5410 msgr2=0x7fb9101094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:01:25.326 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:25.324+0000 7fb9166f3700 1 --2- 172.21.15.155:0/337557720 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9100f5410 0x7fb9101094e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fb9000089f0 tx=0x7fb90000b040).stop 2024-02-10T09:01:25.326 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:25.324+0000 7fb9166f3700 1 -- 172.21.15.155:0/337557720 shutdown_connections 2024-02-10T09:01:25.326 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:25.324+0000 7fb9166f3700 1 --2- 172.21.15.155:0/337557720 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb8fc02ffb0 0x7fb8fc032470 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:01:25.326 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:25.324+0000 7fb9166f3700 1 --2- 172.21.15.155:0/337557720 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9100f5410 0x7fb9101094e0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:01:25.326 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:25.324+0000 7fb9166f3700 1 -- 172.21.15.155:0/337557720 >> 172.21.15.155:0/337557720 conn(0x7fb9100f0b70 msgr2=0x7fb9100f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:01:25.326 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:25.324+0000 7fb9166f3700 1 -- 172.21.15.155:0/337557720 shutdown_connections 2024-02-10T09:01:25.326 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:25.325+0000 7fb9166f3700 1 -- 172.21.15.155:0/337557720 wait complete. 2024-02-10T09:01:25.327 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:01:25.398 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:01:25 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:01:24.978800+0000 mgr.smithi077.dxntvw (mgr.14162) 80 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:01:26.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:01:26 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:01:25.324171+0000 mon.smithi077 (mon.0) 151 : audit [DBG] from='client.? 172.21.15.155:0/337557720' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:01:27.104 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:01:27.104 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:01:27.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:01:27 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:01:26.979177+0000 mgr.smithi077.dxntvw (mgr.14162) 81 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:01:29.182 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:29.132+0000 7f60329e1700 1 -- 172.21.15.155:0/2890165004 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f602c0f5410 msgr2=0x7f602c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:01:29.182 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:29.132+0000 7f60329e1700 1 --2- 172.21.15.155:0/2890165004 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f602c0f5410 0x7f602c0f5830 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f601c009fb0 tx=0x7f601c009b30).stop 2024-02-10T09:01:29.182 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:29.132+0000 7f60329e1700 1 -- 172.21.15.155:0/2890165004 shutdown_connections 2024-02-10T09:01:29.182 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:29.132+0000 7f60329e1700 1 --2- 172.21.15.155:0/2890165004 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f602c0f5410 0x7f602c0f5830 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:01:29.182 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:29.132+0000 7f60329e1700 1 -- 172.21.15.155:0/2890165004 >> 172.21.15.155:0/2890165004 conn(0x7f602c0f0b70 msgr2=0x7f602c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:01:29.182 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:29.132+0000 7f60329e1700 1 -- 172.21.15.155:0/2890165004 shutdown_connections 2024-02-10T09:01:29.182 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:29.132+0000 7f60329e1700 1 -- 172.21.15.155:0/2890165004 wait complete. 2024-02-10T09:01:29.182 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:29.133+0000 7f60329e1700 1 Processor -- start 2024-02-10T09:01:29.182 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:29.133+0000 7f60329e1700 1 -- start start 2024-02-10T09:01:29.182 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:29.133+0000 7f60329e1700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f602c0f5410 0x7f602c1094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:01:29.182 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:29.133+0000 7f60329e1700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f602c060240 con 0x7f602c0f5410 2024-02-10T09:01:29.183 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:29.133+0000 7f602bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f602c0f5410 0x7f602c1094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:01:29.183 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:29.133+0000 7f602bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f602c0f5410 0x7f602c1094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:58326/0 (socket says 172.21.15.155:58326) 2024-02-10T09:01:29.183 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:29.133+0000 7f602bfff700 1 -- 172.21.15.155:0/3983372681 learned_addr learned my addr 172.21.15.155:0/3983372681 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:01:29.183 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:29.134+0000 7f602bfff700 1 -- 172.21.15.155:0/3983372681 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f601c005040 con 0x7f602c0f5410 2024-02-10T09:01:29.183 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:29.134+0000 7f602bfff700 1 --2- 172.21.15.155:0/3983372681 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f602c0f5410 0x7f602c1094e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f601c0089f0 tx=0x7f601c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:01:29.183 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:29.134+0000 7f6028ff9700 1 -- 172.21.15.155:0/3983372681 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f601c009320 con 0x7f602c0f5410 2024-02-10T09:01:29.183 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:29.134+0000 7f60329e1700 1 -- 172.21.15.155:0/3983372681 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f602c060440 con 0x7f602c0f5410 2024-02-10T09:01:29.183 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:29.134+0000 7f6028ff9700 1 -- 172.21.15.155:0/3983372681 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f601c013380 con 0x7f602c0f5410 2024-02-10T09:01:29.183 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:29.134+0000 7f6028ff9700 1 -- 172.21.15.155:0/3983372681 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f601c01ed80 con 0x7f602c0f5410 2024-02-10T09:01:29.183 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:29.134+0000 7f60329e1700 1 -- 172.21.15.155:0/3983372681 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f602c109b90 con 0x7f602c0f5410 2024-02-10T09:01:29.183 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:29.135+0000 7f6028ff9700 1 -- 172.21.15.155:0/3983372681 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f601c01e420 con 0x7f602c0f5410 2024-02-10T09:01:29.183 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:29.135+0000 7f60329e1700 1 -- 172.21.15.155:0/3983372681 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f602c045ad0 con 0x7f602c0f5410 2024-02-10T09:01:29.183 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:29.135+0000 7f6028ff9700 1 --2- 172.21.15.155:0/3983372681 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f601402ff60 0x7f6014032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:01:29.184 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:29.135+0000 7f6028ff9700 1 -- 172.21.15.155:0/3983372681 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f601c040fc0 con 0x7f602c0f5410 2024-02-10T09:01:29.184 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:29.135+0000 7f602b7fe700 1 --2- 172.21.15.155:0/3983372681 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f601402ff60 0x7f6014032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:01:29.184 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:29.136+0000 7f602b7fe700 1 --2- 172.21.15.155:0/3983372681 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f601402ff60 0x7f6014032420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f60200096a0 tx=0x7f6020006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:01:29.184 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:29.140+0000 7f6028ff9700 1 -- 172.21.15.155:0/3983372681 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f601c014cc0 con 0x7f602c0f5410 2024-02-10T09:01:29.352 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:29.351+0000 7f60329e1700 1 -- 172.21.15.155:0/3983372681 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f602c10a410 con 0x7f602c0f5410 2024-02-10T09:01:29.352 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:29.352+0000 7f6028ff9700 1 -- 172.21.15.155:0/3983372681 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f601c01cb10 con 0x7f602c0f5410 2024-02-10T09:01:29.353 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:01:29.353 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:01:29.354 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:29.353+0000 7f60329e1700 1 -- 172.21.15.155:0/3983372681 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f601402ff60 msgr2=0x7f6014032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:01:29.354 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:29.353+0000 7f60329e1700 1 --2- 172.21.15.155:0/3983372681 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f601402ff60 0x7f6014032420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f60200096a0 tx=0x7f6020006b40).stop 2024-02-10T09:01:29.354 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:29.353+0000 7f60329e1700 1 -- 172.21.15.155:0/3983372681 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f602c0f5410 msgr2=0x7f602c1094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:01:29.354 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:29.353+0000 7f60329e1700 1 --2- 172.21.15.155:0/3983372681 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f602c0f5410 0x7f602c1094e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f601c0089f0 tx=0x7f601c00b040).stop 2024-02-10T09:01:29.355 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:29.353+0000 7f60329e1700 1 -- 172.21.15.155:0/3983372681 shutdown_connections 2024-02-10T09:01:29.355 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:29.353+0000 7f60329e1700 1 --2- 172.21.15.155:0/3983372681 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f601402ff60 0x7f6014032420 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:01:29.355 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:29.353+0000 7f60329e1700 1 --2- 172.21.15.155:0/3983372681 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f602c0f5410 0x7f602c1094e0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:01:29.355 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:29.353+0000 7f60329e1700 1 -- 172.21.15.155:0/3983372681 >> 172.21.15.155:0/3983372681 conn(0x7f602c0f0b70 msgr2=0x7f602c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:01:29.355 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:29.353+0000 7f60329e1700 1 -- 172.21.15.155:0/3983372681 shutdown_connections 2024-02-10T09:01:29.355 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:29.353+0000 7f60329e1700 1 -- 172.21.15.155:0/3983372681 wait complete. 2024-02-10T09:01:29.355 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:01:29.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:01:29 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:01:28.979494+0000 mgr.smithi077.dxntvw (mgr.14162) 82 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:01:30.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:01:30 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:01:29.352911+0000 mon.smithi077 (mon.0) 152 : audit [DBG] from='client.? 172.21.15.155:0/3983372681' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:01:31.065 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:01:31.066 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:01:31.398 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:01:31 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:01:30.979853+0000 mgr.smithi077.dxntvw (mgr.14162) 83 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:01:33.148 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:33.147+0000 7f0c082e8700 1 -- 172.21.15.155:0/4113041539 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c000f40e0 msgr2=0x7f0c000f4500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:01:33.149 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:33.147+0000 7f0c082e8700 1 --2- 172.21.15.155:0/4113041539 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c000f40e0 0x7f0c000f4500 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f0bf0009fb0 tx=0x7f0bf0009b30).stop 2024-02-10T09:01:33.149 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:33.147+0000 7f0c082e8700 1 -- 172.21.15.155:0/4113041539 shutdown_connections 2024-02-10T09:01:33.149 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:33.147+0000 7f0c082e8700 1 --2- 172.21.15.155:0/4113041539 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c000f40e0 0x7f0c000f4500 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:01:33.149 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:33.147+0000 7f0c082e8700 1 -- 172.21.15.155:0/4113041539 >> 172.21.15.155:0/4113041539 conn(0x7f0c000efe40 msgr2=0x7f0c000f2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:01:33.149 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:33.147+0000 7f0c082e8700 1 -- 172.21.15.155:0/4113041539 shutdown_connections 2024-02-10T09:01:33.149 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:33.147+0000 7f0c082e8700 1 -- 172.21.15.155:0/4113041539 wait complete. 2024-02-10T09:01:33.149 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:33.147+0000 7f0c082e8700 1 Processor -- start 2024-02-10T09:01:33.150 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:33.147+0000 7f0c082e8700 1 -- start start 2024-02-10T09:01:33.150 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:33.147+0000 7f0c082e8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c000f40e0 0x7f0c001067a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:01:33.150 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:33.147+0000 7f0c082e8700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c00106c90 con 0x7f0c000f40e0 2024-02-10T09:01:33.150 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:33.148+0000 7f0c06084700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c000f40e0 0x7f0c001067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:01:33.150 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:33.148+0000 7f0c06084700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c000f40e0 0x7f0c001067a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:50112/0 (socket says 172.21.15.155:50112) 2024-02-10T09:01:33.150 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:33.148+0000 7f0c06084700 1 -- 172.21.15.155:0/23266609 learned_addr learned my addr 172.21.15.155:0/23266609 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:01:33.150 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:33.148+0000 7f0c06084700 1 -- 172.21.15.155:0/23266609 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bf0005040 con 0x7f0c000f40e0 2024-02-10T09:01:33.150 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:33.148+0000 7f0c06084700 1 --2- 172.21.15.155:0/23266609 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c000f40e0 0x7f0c001067a0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f0bf0005010 tx=0x7f0bf000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:01:33.150 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:33.148+0000 7f0bf6ffd700 1 -- 172.21.15.155:0/23266609 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0bf0009320 con 0x7f0c000f40e0 2024-02-10T09:01:33.150 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:33.148+0000 7f0c082e8700 1 -- 172.21.15.155:0/23266609 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c00106e90 con 0x7f0c000f40e0 2024-02-10T09:01:33.150 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:33.148+0000 7f0bf6ffd700 1 -- 172.21.15.155:0/23266609 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0bf0013380 con 0x7f0c000f40e0 2024-02-10T09:01:33.150 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:33.148+0000 7f0bf6ffd700 1 -- 172.21.15.155:0/23266609 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0bf0014e50 con 0x7f0c000f40e0 2024-02-10T09:01:33.150 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:33.148+0000 7f0c082e8700 1 -- 172.21.15.155:0/23266609 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c00107320 con 0x7f0c000f40e0 2024-02-10T09:01:33.151 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:33.149+0000 7f0c082e8700 1 -- 172.21.15.155:0/23266609 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c00045ad0 con 0x7f0c000f40e0 2024-02-10T09:01:33.151 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:33.149+0000 7f0bf6ffd700 1 -- 172.21.15.155:0/23266609 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0bf00095f0 con 0x7f0c000f40e0 2024-02-10T09:01:33.151 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:33.149+0000 7f0bf6ffd700 1 --2- 172.21.15.155:0/23266609 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f0bec02ff10 0x7f0bec0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:01:33.152 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:33.149+0000 7f0bf6ffd700 1 -- 172.21.15.155:0/23266609 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0bf0040e60 con 0x7f0c000f40e0 2024-02-10T09:01:33.152 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:33.150+0000 7f0c05883700 1 --2- 172.21.15.155:0/23266609 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f0bec02ff10 0x7f0bec0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:01:33.152 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:33.150+0000 7f0c05883700 1 --2- 172.21.15.155:0/23266609 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f0bec02ff10 0x7f0bec0323d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f0bfc0096a0 tx=0x7f0bfc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:01:33.154 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:33.154+0000 7f0bf6ffd700 1 -- 172.21.15.155:0/23266609 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0bf0014420 con 0x7f0c000f40e0 2024-02-10T09:01:33.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:33.361+0000 7f0c082e8700 1 -- 172.21.15.155:0/23266609 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0c00108170 con 0x7f0c000f40e0 2024-02-10T09:01:33.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:33.362+0000 7f0bf6ffd700 1 -- 172.21.15.155:0/23266609 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0bf0010230 con 0x7f0c000f40e0 2024-02-10T09:01:33.362 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:01:33.362 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:01:33.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:33.364+0000 7f0c082e8700 1 -- 172.21.15.155:0/23266609 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f0bec02ff10 msgr2=0x7f0bec0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:01:33.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:33.364+0000 7f0c082e8700 1 --2- 172.21.15.155:0/23266609 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f0bec02ff10 0x7f0bec0323d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f0bfc0096a0 tx=0x7f0bfc006b40).stop 2024-02-10T09:01:33.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:33.364+0000 7f0c082e8700 1 -- 172.21.15.155:0/23266609 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c000f40e0 msgr2=0x7f0c001067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:01:33.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:33.364+0000 7f0c082e8700 1 --2- 172.21.15.155:0/23266609 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c000f40e0 0x7f0c001067a0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f0bf0005010 tx=0x7f0bf000b040).stop 2024-02-10T09:01:33.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:33.364+0000 7f0c082e8700 1 -- 172.21.15.155:0/23266609 shutdown_connections 2024-02-10T09:01:33.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:33.364+0000 7f0c082e8700 1 --2- 172.21.15.155:0/23266609 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f0bec02ff10 0x7f0bec0323d0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:01:33.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:33.364+0000 7f0c082e8700 1 --2- 172.21.15.155:0/23266609 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c000f40e0 0x7f0c001067a0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:01:33.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:33.364+0000 7f0c082e8700 1 -- 172.21.15.155:0/23266609 >> 172.21.15.155:0/23266609 conn(0x7f0c000efe40 msgr2=0x7f0c000f0890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:01:33.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:33.364+0000 7f0c082e8700 1 -- 172.21.15.155:0/23266609 shutdown_connections 2024-02-10T09:01:33.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:33.364+0000 7f0c082e8700 1 -- 172.21.15.155:0/23266609 wait complete. 2024-02-10T09:01:33.365 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:01:34.148 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:01:33 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:01:32.980192+0000 mgr.smithi077.dxntvw (mgr.14162) 84 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:01:34.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:01:33 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:01:33.362487+0000 mon.smithi077 (mon.0) 153 : audit [DBG] from='client.? 172.21.15.155:0/23266609' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:01:35.132 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:01:35.134 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:01:35.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:01:35 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:01:34.980421+0000 mgr.smithi077.dxntvw (mgr.14162) 85 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:01:37.206 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:37.206+0000 7f4cb127e700 1 -- 172.21.15.155:0/3166710792 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cac0f5410 msgr2=0x7f4cac0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:01:37.207 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:37.206+0000 7f4cb127e700 1 --2- 172.21.15.155:0/3166710792 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cac0f5410 0x7f4cac0f5830 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f4ca0009fb0 tx=0x7f4ca0009b30).stop 2024-02-10T09:01:37.207 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:37.206+0000 7f4cb127e700 1 -- 172.21.15.155:0/3166710792 shutdown_connections 2024-02-10T09:01:37.207 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:37.206+0000 7f4cb127e700 1 --2- 172.21.15.155:0/3166710792 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cac0f5410 0x7f4cac0f5830 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:01:37.207 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:37.206+0000 7f4cb127e700 1 -- 172.21.15.155:0/3166710792 >> 172.21.15.155:0/3166710792 conn(0x7f4cac0f0b70 msgr2=0x7f4cac0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:01:37.207 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:37.206+0000 7f4cb127e700 1 -- 172.21.15.155:0/3166710792 shutdown_connections 2024-02-10T09:01:37.207 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:37.206+0000 7f4cb127e700 1 -- 172.21.15.155:0/3166710792 wait complete. 2024-02-10T09:01:37.209 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:37.207+0000 7f4cb127e700 1 Processor -- start 2024-02-10T09:01:37.209 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:37.207+0000 7f4cb127e700 1 -- start start 2024-02-10T09:01:37.209 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:37.207+0000 7f4cb127e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cac0f5410 0x7f4cac1067f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:01:37.209 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:37.207+0000 7f4cb127e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cac106ce0 con 0x7f4cac0f5410 2024-02-10T09:01:37.209 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:37.207+0000 7f4caaffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cac0f5410 0x7f4cac1067f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:01:37.209 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:37.207+0000 7f4caaffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cac0f5410 0x7f4cac1067f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:50122/0 (socket says 172.21.15.155:50122) 2024-02-10T09:01:37.209 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:37.207+0000 7f4caaffd700 1 -- 172.21.15.155:0/1586648351 learned_addr learned my addr 172.21.15.155:0/1586648351 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:01:37.209 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:37.208+0000 7f4caaffd700 1 -- 172.21.15.155:0/1586648351 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ca0005040 con 0x7f4cac0f5410 2024-02-10T09:01:37.209 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:37.208+0000 7f4caaffd700 1 --2- 172.21.15.155:0/1586648351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cac0f5410 0x7f4cac1067f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f4ca00089f0 tx=0x7f4ca000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:01:37.210 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:37.208+0000 7f4c937fe700 1 -- 172.21.15.155:0/1586648351 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ca0009320 con 0x7f4cac0f5410 2024-02-10T09:01:37.210 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:37.208+0000 7f4c937fe700 1 -- 172.21.15.155:0/1586648351 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4ca0013380 con 0x7f4cac0f5410 2024-02-10T09:01:37.210 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:37.208+0000 7f4cb127e700 1 -- 172.21.15.155:0/1586648351 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cac106ee0 con 0x7f4cac0f5410 2024-02-10T09:01:37.210 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:37.208+0000 7f4c937fe700 1 -- 172.21.15.155:0/1586648351 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ca001ed80 con 0x7f4cac0f5410 2024-02-10T09:01:37.210 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:37.208+0000 7f4cb127e700 1 -- 172.21.15.155:0/1586648351 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cac107370 con 0x7f4cac0f5410 2024-02-10T09:01:37.210 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:37.209+0000 7f4c937fe700 1 -- 172.21.15.155:0/1586648351 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4ca001e420 con 0x7f4cac0f5410 2024-02-10T09:01:37.210 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:37.209+0000 7f4cb127e700 1 -- 172.21.15.155:0/1586648351 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cac045ad0 con 0x7f4cac0f5410 2024-02-10T09:01:37.210 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:37.209+0000 7f4c937fe700 1 --2- 172.21.15.155:0/1586648351 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f4c9402ffb0 0x7f4c94032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:01:37.210 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:37.209+0000 7f4c937fe700 1 -- 172.21.15.155:0/1586648351 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4ca0041180 con 0x7f4cac0f5410 2024-02-10T09:01:37.210 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:37.209+0000 7f4caa7fc700 1 --2- 172.21.15.155:0/1586648351 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f4c9402ffb0 0x7f4c94032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:01:37.211 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:37.210+0000 7f4caa7fc700 1 --2- 172.21.15.155:0/1586648351 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f4c9402ffb0 0x7f4c94032470 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f4c9c0096a0 tx=0x7f4c9c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:01:37.213 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:37.214+0000 7f4c937fe700 1 -- 172.21.15.155:0/1586648351 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4ca001cb10 con 0x7f4cac0f5410 2024-02-10T09:01:37.398 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:01:37 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:01:36.980672+0000 mgr.smithi077.dxntvw (mgr.14162) 86 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:01:37.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:37.424+0000 7f4cb127e700 1 -- 172.21.15.155:0/1586648351 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4cac108220 con 0x7f4cac0f5410 2024-02-10T09:01:37.424 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:37.424+0000 7f4c937fe700 1 -- 172.21.15.155:0/1586648351 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4ca001cb10 con 0x7f4cac0f5410 2024-02-10T09:01:37.424 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:01:37.424 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:01:37.426 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:37.426+0000 7f4cb127e700 1 -- 172.21.15.155:0/1586648351 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f4c9402ffb0 msgr2=0x7f4c94032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:01:37.426 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:37.426+0000 7f4cb127e700 1 --2- 172.21.15.155:0/1586648351 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f4c9402ffb0 0x7f4c94032470 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f4c9c0096a0 tx=0x7f4c9c006b40).stop 2024-02-10T09:01:37.426 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:37.426+0000 7f4cb127e700 1 -- 172.21.15.155:0/1586648351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cac0f5410 msgr2=0x7f4cac1067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:01:37.426 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:37.426+0000 7f4cb127e700 1 --2- 172.21.15.155:0/1586648351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cac0f5410 0x7f4cac1067f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f4ca00089f0 tx=0x7f4ca000b040).stop 2024-02-10T09:01:37.426 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:37.426+0000 7f4cb127e700 1 -- 172.21.15.155:0/1586648351 shutdown_connections 2024-02-10T09:01:37.426 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:37.426+0000 7f4cb127e700 1 --2- 172.21.15.155:0/1586648351 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f4c9402ffb0 0x7f4c94032470 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:01:37.427 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:37.426+0000 7f4cb127e700 1 --2- 172.21.15.155:0/1586648351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cac0f5410 0x7f4cac1067f0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:01:37.427 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:37.426+0000 7f4cb127e700 1 -- 172.21.15.155:0/1586648351 >> 172.21.15.155:0/1586648351 conn(0x7f4cac0f0b70 msgr2=0x7f4cac05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:01:37.427 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:37.426+0000 7f4cb127e700 1 -- 172.21.15.155:0/1586648351 shutdown_connections 2024-02-10T09:01:37.427 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:37.426+0000 7f4cb127e700 1 -- 172.21.15.155:0/1586648351 wait complete. 2024-02-10T09:01:37.427 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:01:38.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:01:38 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:01:37.424689+0000 mon.smithi077 (mon.0) 154 : audit [DBG] from='client.? 172.21.15.155:0/1586648351' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:01:39.082 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:01:39.082 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:01:39.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:01:39 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:01:38.981023+0000 mgr.smithi077.dxntvw (mgr.14162) 87 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:01:41.131 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:41.127+0000 7f6962afe700 1 -- 172.21.15.155:0/2968490472 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f695c0f5410 msgr2=0x7f695c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:01:41.131 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:41.127+0000 7f6962afe700 1 --2- 172.21.15.155:0/2968490472 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f695c0f5410 0x7f695c0f5830 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f6950009fb0 tx=0x7f6950009b30).stop 2024-02-10T09:01:41.131 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:41.127+0000 7f6962afe700 1 -- 172.21.15.155:0/2968490472 shutdown_connections 2024-02-10T09:01:41.131 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:41.127+0000 7f6962afe700 1 --2- 172.21.15.155:0/2968490472 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f695c0f5410 0x7f695c0f5830 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:01:41.131 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:41.127+0000 7f6962afe700 1 -- 172.21.15.155:0/2968490472 >> 172.21.15.155:0/2968490472 conn(0x7f695c0f0b70 msgr2=0x7f695c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:01:41.132 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:41.127+0000 7f6962afe700 1 -- 172.21.15.155:0/2968490472 shutdown_connections 2024-02-10T09:01:41.132 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:41.127+0000 7f6962afe700 1 -- 172.21.15.155:0/2968490472 wait complete. 2024-02-10T09:01:41.132 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:41.128+0000 7f6962afe700 1 Processor -- start 2024-02-10T09:01:41.132 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:41.128+0000 7f6962afe700 1 -- start start 2024-02-10T09:01:41.132 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:41.128+0000 7f6962afe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f695c0f5410 0x7f695c1067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:01:41.132 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:41.128+0000 7f6962afe700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f695c106ca0 con 0x7f695c0f5410 2024-02-10T09:01:41.132 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:41.128+0000 7f696089a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f695c0f5410 0x7f695c1067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:01:41.132 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:41.129+0000 7f696089a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f695c0f5410 0x7f695c1067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:50138/0 (socket says 172.21.15.155:50138) 2024-02-10T09:01:41.132 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:41.129+0000 7f696089a700 1 -- 172.21.15.155:0/1422249356 learned_addr learned my addr 172.21.15.155:0/1422249356 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:01:41.134 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:41.129+0000 7f696089a700 1 -- 172.21.15.155:0/1422249356 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6950005040 con 0x7f695c0f5410 2024-02-10T09:01:41.134 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:41.129+0000 7f696089a700 1 --2- 172.21.15.155:0/1422249356 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f695c0f5410 0x7f695c1067b0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f6950005010 tx=0x7f695000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:01:41.134 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:41.129+0000 7f69597fa700 1 -- 172.21.15.155:0/1422249356 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6950009320 con 0x7f695c0f5410 2024-02-10T09:01:41.134 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:41.129+0000 7f69597fa700 1 -- 172.21.15.155:0/1422249356 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6950013380 con 0x7f695c0f5410 2024-02-10T09:01:41.134 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:41.129+0000 7f6962afe700 1 -- 172.21.15.155:0/1422249356 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f695c106ea0 con 0x7f695c0f5410 2024-02-10T09:01:41.134 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:41.129+0000 7f69597fa700 1 -- 172.21.15.155:0/1422249356 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6950014e50 con 0x7f695c0f5410 2024-02-10T09:01:41.134 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:41.129+0000 7f6962afe700 1 -- 172.21.15.155:0/1422249356 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f695c107330 con 0x7f695c0f5410 2024-02-10T09:01:41.134 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:41.130+0000 7f69597fa700 1 -- 172.21.15.155:0/1422249356 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f69500095f0 con 0x7f695c0f5410 2024-02-10T09:01:41.134 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:41.130+0000 7f6962afe700 1 -- 172.21.15.155:0/1422249356 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f695c045ad0 con 0x7f695c0f5410 2024-02-10T09:01:41.134 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:41.130+0000 7f69597fa700 1 --2- 172.21.15.155:0/1422249356 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f694c02ff10 0x7f694c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:01:41.134 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:41.130+0000 7f69597fa700 1 -- 172.21.15.155:0/1422249356 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6950040e90 con 0x7f695c0f5410 2024-02-10T09:01:41.134 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:41.131+0000 7f695bfff700 1 --2- 172.21.15.155:0/1422249356 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f694c02ff10 0x7f694c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:01:41.134 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:41.131+0000 7f695bfff700 1 --2- 172.21.15.155:0/1422249356 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f694c02ff10 0x7f694c0323d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f69480096a0 tx=0x7f6948006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:01:41.135 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:41.135+0000 7f69597fa700 1 -- 172.21.15.155:0/1422249356 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6950014420 con 0x7f695c0f5410 2024-02-10T09:01:41.344 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:41.344+0000 7f6962afe700 1 -- 172.21.15.155:0/1422249356 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f695c0fafa0 con 0x7f695c0f5410 2024-02-10T09:01:41.344 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:41.344+0000 7f69597fa700 1 -- 172.21.15.155:0/1422249356 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6950010230 con 0x7f695c0f5410 2024-02-10T09:01:41.345 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:01:41.345 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:01:41.346 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:41.346+0000 7f6962afe700 1 -- 172.21.15.155:0/1422249356 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f694c02ff10 msgr2=0x7f694c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:01:41.346 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:41.346+0000 7f6962afe700 1 --2- 172.21.15.155:0/1422249356 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f694c02ff10 0x7f694c0323d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f69480096a0 tx=0x7f6948006b40).stop 2024-02-10T09:01:41.346 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:41.346+0000 7f6962afe700 1 -- 172.21.15.155:0/1422249356 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f695c0f5410 msgr2=0x7f695c1067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:01:41.346 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:41.346+0000 7f6962afe700 1 --2- 172.21.15.155:0/1422249356 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f695c0f5410 0x7f695c1067b0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f6950005010 tx=0x7f695000b040).stop 2024-02-10T09:01:41.347 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:41.346+0000 7f6962afe700 1 -- 172.21.15.155:0/1422249356 shutdown_connections 2024-02-10T09:01:41.347 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:41.346+0000 7f6962afe700 1 --2- 172.21.15.155:0/1422249356 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f694c02ff10 0x7f694c0323d0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:01:41.347 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:41.346+0000 7f6962afe700 1 --2- 172.21.15.155:0/1422249356 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f695c0f5410 0x7f695c1067b0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:01:41.347 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:41.346+0000 7f6962afe700 1 -- 172.21.15.155:0/1422249356 >> 172.21.15.155:0/1422249356 conn(0x7f695c0f0b70 msgr2=0x7f695c05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:01:41.347 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:41.346+0000 7f6962afe700 1 -- 172.21.15.155:0/1422249356 shutdown_connections 2024-02-10T09:01:41.347 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:41.346+0000 7f6962afe700 1 -- 172.21.15.155:0/1422249356 wait complete. 2024-02-10T09:01:41.347 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:01:41.398 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:01:41 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:01:40.981332+0000 mgr.smithi077.dxntvw (mgr.14162) 88 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:01:42.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:01:42 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:01:41.344723+0000 mon.smithi077 (mon.0) 155 : audit [DBG] from='client.? 172.21.15.155:0/1422249356' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:01:43.083 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:01:43.083 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:01:44.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:01:43 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:01:42.981643+0000 mgr.smithi077.dxntvw (mgr.14162) 89 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:01:45.257 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:45.256+0000 7f0de206c700 1 -- 172.21.15.155:0/2363092581 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ddc0f5410 msgr2=0x7f0ddc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:01:45.257 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:45.256+0000 7f0de206c700 1 --2- 172.21.15.155:0/2363092581 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ddc0f5410 0x7f0ddc0f5830 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f0dd0009fb0 tx=0x7f0dd0009b30).stop 2024-02-10T09:01:45.257 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:45.256+0000 7f0de206c700 1 -- 172.21.15.155:0/2363092581 shutdown_connections 2024-02-10T09:01:45.257 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:45.256+0000 7f0de206c700 1 --2- 172.21.15.155:0/2363092581 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ddc0f5410 0x7f0ddc0f5830 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:01:45.257 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:45.256+0000 7f0de206c700 1 -- 172.21.15.155:0/2363092581 >> 172.21.15.155:0/2363092581 conn(0x7f0ddc0f0b70 msgr2=0x7f0ddc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:01:45.257 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:45.256+0000 7f0de206c700 1 -- 172.21.15.155:0/2363092581 shutdown_connections 2024-02-10T09:01:45.257 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:45.256+0000 7f0de206c700 1 -- 172.21.15.155:0/2363092581 wait complete. 2024-02-10T09:01:45.257 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:45.257+0000 7f0de206c700 1 Processor -- start 2024-02-10T09:01:45.257 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:45.257+0000 7f0de206c700 1 -- start start 2024-02-10T09:01:45.258 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:45.257+0000 7f0de206c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ddc0f5410 0x7f0ddc1067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:01:45.258 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:45.257+0000 7f0de206c700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ddc106cc0 con 0x7f0ddc0f5410 2024-02-10T09:01:45.259 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:45.257+0000 7f0ddb7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ddc0f5410 0x7f0ddc1067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:01:45.259 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:45.257+0000 7f0ddb7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ddc0f5410 0x7f0ddc1067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:57896/0 (socket says 172.21.15.155:57896) 2024-02-10T09:01:45.260 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:45.257+0000 7f0ddb7fe700 1 -- 172.21.15.155:0/3193618356 learned_addr learned my addr 172.21.15.155:0/3193618356 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:01:45.260 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:45.257+0000 7f0ddb7fe700 1 -- 172.21.15.155:0/3193618356 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dd0005040 con 0x7f0ddc0f5410 2024-02-10T09:01:45.260 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:45.258+0000 7f0ddb7fe700 1 --2- 172.21.15.155:0/3193618356 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ddc0f5410 0x7f0ddc1067d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f0dd0005010 tx=0x7f0dd000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:01:45.260 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:45.258+0000 7f0dc3fff700 1 -- 172.21.15.155:0/3193618356 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0dd0009320 con 0x7f0ddc0f5410 2024-02-10T09:01:45.260 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:45.258+0000 7f0de206c700 1 -- 172.21.15.155:0/3193618356 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ddc106ec0 con 0x7f0ddc0f5410 2024-02-10T09:01:45.260 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:45.258+0000 7f0dc3fff700 1 -- 172.21.15.155:0/3193618356 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0dd0013380 con 0x7f0ddc0f5410 2024-02-10T09:01:45.260 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:45.258+0000 7f0dc3fff700 1 -- 172.21.15.155:0/3193618356 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0dd0014e50 con 0x7f0ddc0f5410 2024-02-10T09:01:45.260 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:45.258+0000 7f0de206c700 1 -- 172.21.15.155:0/3193618356 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ddc107350 con 0x7f0ddc0f5410 2024-02-10T09:01:45.260 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:45.259+0000 7f0dc3fff700 1 -- 172.21.15.155:0/3193618356 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0dd00263a0 con 0x7f0ddc0f5410 2024-02-10T09:01:45.260 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:45.259+0000 7f0dc3fff700 1 --2- 172.21.15.155:0/3193618356 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f0dc402ff10 0x7f0dc40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:01:45.260 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:45.259+0000 7f0dc3fff700 1 -- 172.21.15.155:0/3193618356 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0dd0040eb0 con 0x7f0ddc0f5410 2024-02-10T09:01:45.260 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:45.259+0000 7f0de206c700 1 -- 172.21.15.155:0/3193618356 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ddc045ad0 con 0x7f0ddc0f5410 2024-02-10T09:01:45.260 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:45.259+0000 7f0ddaffd700 1 --2- 172.21.15.155:0/3193618356 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f0dc402ff10 0x7f0dc40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:01:45.261 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:45.259+0000 7f0ddaffd700 1 --2- 172.21.15.155:0/3193618356 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f0dc402ff10 0x7f0dc40323d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f0dcc0096a0 tx=0x7f0dcc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:01:45.264 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:45.264+0000 7f0dc3fff700 1 -- 172.21.15.155:0/3193618356 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0dd0012920 con 0x7f0ddc0f5410 2024-02-10T09:01:45.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:01:45 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:01:44.981914+0000 mgr.smithi077.dxntvw (mgr.14162) 90 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:01:45.471 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:45.471+0000 7f0de206c700 1 -- 172.21.15.155:0/3193618356 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0ddc0fb0f0 con 0x7f0ddc0f5410 2024-02-10T09:01:45.472 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:45.472+0000 7f0dc3fff700 1 -- 172.21.15.155:0/3193618356 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0dd00126f0 con 0x7f0ddc0f5410 2024-02-10T09:01:45.472 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:01:45.472 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:01:45.474 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:45.473+0000 7f0de206c700 1 -- 172.21.15.155:0/3193618356 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f0dc402ff10 msgr2=0x7f0dc40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:01:45.474 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:45.473+0000 7f0de206c700 1 --2- 172.21.15.155:0/3193618356 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f0dc402ff10 0x7f0dc40323d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f0dcc0096a0 tx=0x7f0dcc006b40).stop 2024-02-10T09:01:45.474 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:45.473+0000 7f0de206c700 1 -- 172.21.15.155:0/3193618356 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ddc0f5410 msgr2=0x7f0ddc1067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:01:45.474 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:45.473+0000 7f0de206c700 1 --2- 172.21.15.155:0/3193618356 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ddc0f5410 0x7f0ddc1067d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f0dd0005010 tx=0x7f0dd000b040).stop 2024-02-10T09:01:45.474 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:45.474+0000 7f0de206c700 1 -- 172.21.15.155:0/3193618356 shutdown_connections 2024-02-10T09:01:45.474 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:45.474+0000 7f0de206c700 1 --2- 172.21.15.155:0/3193618356 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f0dc402ff10 0x7f0dc40323d0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:01:45.474 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:45.474+0000 7f0de206c700 1 --2- 172.21.15.155:0/3193618356 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ddc0f5410 0x7f0ddc1067d0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:01:45.474 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:45.474+0000 7f0de206c700 1 -- 172.21.15.155:0/3193618356 >> 172.21.15.155:0/3193618356 conn(0x7f0ddc0f0b70 msgr2=0x7f0ddc05fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:01:45.474 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:45.474+0000 7f0de206c700 1 -- 172.21.15.155:0/3193618356 shutdown_connections 2024-02-10T09:01:45.474 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:45.474+0000 7f0de206c700 1 -- 172.21.15.155:0/3193618356 wait complete. 2024-02-10T09:01:45.475 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:01:46.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:01:46 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:01:45.472055+0000 mon.smithi077 (mon.0) 156 : audit [DBG] from='client.? 172.21.15.155:0/3193618356' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:01:47.171 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:01:47.171 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:01:47.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:01:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:01:46.982226+0000 mgr.smithi077.dxntvw (mgr.14162) 91 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:01:48.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:01:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:01:47.090051+0000 mon.smithi077 (mon.0) 157 : audit [INF] from='mgr.14162 172.21.15.77:0/2029106299' entity='mgr.smithi077.dxntvw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi077.dxntvw/mirror_snapshot_schedule"}]: dispatch 2024-02-10T09:01:48.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:01:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:01:47.097374+0000 mon.smithi077 (mon.0) 158 : audit [INF] from='mgr.14162 172.21.15.77:0/2029106299' entity='mgr.smithi077.dxntvw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi077.dxntvw/trash_purge_schedule"}]: dispatch 2024-02-10T09:01:49.081 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:49.057+0000 7f12b8d93700 1 -- 172.21.15.155:0/3285816960 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12b40f5410 msgr2=0x7f12b40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:01:49.081 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:49.057+0000 7f12b8d93700 1 --2- 172.21.15.155:0/3285816960 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12b40f5410 0x7f12b40f5830 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f12a8009fb0 tx=0x7f12a8009b30).stop 2024-02-10T09:01:49.081 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:49.057+0000 7f12b8d93700 1 -- 172.21.15.155:0/3285816960 shutdown_connections 2024-02-10T09:01:49.081 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:49.057+0000 7f12b8d93700 1 --2- 172.21.15.155:0/3285816960 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12b40f5410 0x7f12b40f5830 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:01:49.081 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:49.057+0000 7f12b8d93700 1 -- 172.21.15.155:0/3285816960 >> 172.21.15.155:0/3285816960 conn(0x7f12b40f0b70 msgr2=0x7f12b40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:01:49.081 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:49.057+0000 7f12b8d93700 1 -- 172.21.15.155:0/3285816960 shutdown_connections 2024-02-10T09:01:49.082 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:49.057+0000 7f12b8d93700 1 -- 172.21.15.155:0/3285816960 wait complete. 2024-02-10T09:01:49.082 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:49.057+0000 7f12b8d93700 1 Processor -- start 2024-02-10T09:01:49.082 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:49.057+0000 7f12b8d93700 1 -- start start 2024-02-10T09:01:49.082 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:49.058+0000 7f12b8d93700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12b40f5410 0x7f12b40fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:01:49.082 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:49.058+0000 7f12b8d93700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b40fb960 con 0x7f12b40f5410 2024-02-10T09:01:49.082 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:49.058+0000 7f12b259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12b40f5410 0x7f12b40fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:01:49.082 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:49.058+0000 7f12b259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12b40f5410 0x7f12b40fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:57908/0 (socket says 172.21.15.155:57908) 2024-02-10T09:01:49.082 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:49.058+0000 7f12b259c700 1 -- 172.21.15.155:0/3677100426 learned_addr learned my addr 172.21.15.155:0/3677100426 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:01:49.082 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:49.058+0000 7f12b259c700 1 -- 172.21.15.155:0/3677100426 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12a8005040 con 0x7f12b40f5410 2024-02-10T09:01:49.082 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:49.058+0000 7f12b259c700 1 --2- 172.21.15.155:0/3677100426 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12b40f5410 0x7f12b40fd4e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f12a8005010 tx=0x7f12a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:01:49.082 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:49.061+0000 7f12a2ffd700 1 -- 172.21.15.155:0/3677100426 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12a8009320 con 0x7f12b40f5410 2024-02-10T09:01:49.083 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:49.061+0000 7f12a2ffd700 1 -- 172.21.15.155:0/3677100426 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f12a8013380 con 0x7f12b40f5410 2024-02-10T09:01:49.083 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:49.061+0000 7f12a2ffd700 1 -- 172.21.15.155:0/3677100426 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12a8014e50 con 0x7f12b40f5410 2024-02-10T09:01:49.083 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:49.061+0000 7f12b8d93700 1 -- 172.21.15.155:0/3677100426 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12b40fbb00 con 0x7f12b40f5410 2024-02-10T09:01:49.083 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:49.061+0000 7f12b8d93700 1 -- 172.21.15.155:0/3677100426 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12b40fbf90 con 0x7f12b40f5410 2024-02-10T09:01:49.083 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:49.062+0000 7f12b8d93700 1 -- 172.21.15.155:0/3677100426 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12b4059100 con 0x7f12b40f5410 2024-02-10T09:01:49.083 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:49.062+0000 7f12a2ffd700 1 -- 172.21.15.155:0/3677100426 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f12a80095f0 con 0x7f12b40f5410 2024-02-10T09:01:49.083 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:49.062+0000 7f12a2ffd700 1 --2- 172.21.15.155:0/3677100426 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f129c02ff10 0x7f129c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:01:49.083 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:49.062+0000 7f12a2ffd700 1 -- 172.21.15.155:0/3677100426 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f12a8040f70 con 0x7f12b40f5410 2024-02-10T09:01:49.083 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:49.062+0000 7f12b1d9b700 1 --2- 172.21.15.155:0/3677100426 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f129c02ff10 0x7f129c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:01:49.083 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:49.062+0000 7f12b1d9b700 1 --2- 172.21.15.155:0/3677100426 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f129c02ff10 0x7f129c0323d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f12a40096a0 tx=0x7f12a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:01:49.083 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:49.067+0000 7f12a2ffd700 1 -- 172.21.15.155:0/3677100426 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f12a8014420 con 0x7f12b40f5410 2024-02-10T09:01:49.271 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:49.271+0000 7f12b8d93700 1 -- 172.21.15.155:0/3677100426 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f12b40fafa0 con 0x7f12b40f5410 2024-02-10T09:01:49.271 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:49.271+0000 7f12a2ffd700 1 -- 172.21.15.155:0/3677100426 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f12a801d980 con 0x7f12b40f5410 2024-02-10T09:01:49.271 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:01:49.272 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:01:49.273 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:49.273+0000 7f12b8d93700 1 -- 172.21.15.155:0/3677100426 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f129c02ff10 msgr2=0x7f129c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:01:49.273 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:49.273+0000 7f12b8d93700 1 --2- 172.21.15.155:0/3677100426 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f129c02ff10 0x7f129c0323d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f12a40096a0 tx=0x7f12a4006b40).stop 2024-02-10T09:01:49.273 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:49.273+0000 7f12b8d93700 1 -- 172.21.15.155:0/3677100426 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12b40f5410 msgr2=0x7f12b40fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:01:49.273 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:49.273+0000 7f12b8d93700 1 --2- 172.21.15.155:0/3677100426 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12b40f5410 0x7f12b40fd4e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f12a8005010 tx=0x7f12a800b040).stop 2024-02-10T09:01:49.273 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:49.273+0000 7f12b8d93700 1 -- 172.21.15.155:0/3677100426 shutdown_connections 2024-02-10T09:01:49.273 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:49.273+0000 7f12b8d93700 1 --2- 172.21.15.155:0/3677100426 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f129c02ff10 0x7f129c0323d0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:01:49.273 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:49.273+0000 7f12b8d93700 1 --2- 172.21.15.155:0/3677100426 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12b40f5410 0x7f12b40fd4e0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:01:49.273 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:49.273+0000 7f12b8d93700 1 -- 172.21.15.155:0/3677100426 >> 172.21.15.155:0/3677100426 conn(0x7f12b40f0b70 msgr2=0x7f12b405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:01:49.274 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:49.273+0000 7f12b8d93700 1 -- 172.21.15.155:0/3677100426 shutdown_connections 2024-02-10T09:01:49.274 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:49.273+0000 7f12b8d93700 1 -- 172.21.15.155:0/3677100426 wait complete. 2024-02-10T09:01:49.274 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:01:49.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:01:49 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:01:48.982596+0000 mgr.smithi077.dxntvw (mgr.14162) 92 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:01:50.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:01:50 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:01:49.271794+0000 mon.smithi077 (mon.0) 159 : audit [DBG] from='client.? 172.21.15.155:0/3677100426' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:01:51.003 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:01:51.004 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:01:51.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:01:51 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:01:50.982952+0000 mgr.smithi077.dxntvw (mgr.14162) 93 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:01:53.014 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:53.013+0000 7fa89dd6e700 1 -- 172.21.15.155:0/2945216058 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8980f6510 msgr2=0x7fa8980f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:01:53.015 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:53.013+0000 7fa89dd6e700 1 --2- 172.21.15.155:0/2945216058 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8980f6510 0x7fa8980f6930 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7fa88c009fb0 tx=0x7fa88c009b30).stop 2024-02-10T09:01:53.015 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:53.013+0000 7fa89dd6e700 1 -- 172.21.15.155:0/2945216058 shutdown_connections 2024-02-10T09:01:53.015 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:53.013+0000 7fa89dd6e700 1 --2- 172.21.15.155:0/2945216058 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8980f6510 0x7fa8980f6930 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:01:53.015 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:53.013+0000 7fa89dd6e700 1 -- 172.21.15.155:0/2945216058 >> 172.21.15.155:0/2945216058 conn(0x7fa8980f1c70 msgr2=0x7fa8980f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:01:53.015 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:53.013+0000 7fa89dd6e700 1 -- 172.21.15.155:0/2945216058 shutdown_connections 2024-02-10T09:01:53.015 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:53.013+0000 7fa89dd6e700 1 -- 172.21.15.155:0/2945216058 wait complete. 2024-02-10T09:01:53.018 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:53.014+0000 7fa89dd6e700 1 Processor -- start 2024-02-10T09:01:53.018 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:53.014+0000 7fa89dd6e700 1 -- start start 2024-02-10T09:01:53.018 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:53.014+0000 7fa89dd6e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8980f6510 0x7fa8981067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:01:53.018 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:53.014+0000 7fa89dd6e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa898106cd0 con 0x7fa8980f6510 2024-02-10T09:01:53.018 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:53.015+0000 7fa8977fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8980f6510 0x7fa8981067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:01:53.018 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:53.015+0000 7fa8977fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8980f6510 0x7fa8981067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:44124/0 (socket says 172.21.15.155:44124) 2024-02-10T09:01:53.018 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:53.015+0000 7fa8977fe700 1 -- 172.21.15.155:0/369924898 learned_addr learned my addr 172.21.15.155:0/369924898 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:01:53.018 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:53.015+0000 7fa8977fe700 1 -- 172.21.15.155:0/369924898 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa88c005040 con 0x7fa8980f6510 2024-02-10T09:01:53.018 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:53.015+0000 7fa8977fe700 1 --2- 172.21.15.155:0/369924898 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8980f6510 0x7fa8981067e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fa88c005010 tx=0x7fa88c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:01:53.018 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:53.015+0000 7fa87ffff700 1 -- 172.21.15.155:0/369924898 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa88c009320 con 0x7fa8980f6510 2024-02-10T09:01:53.018 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:53.015+0000 7fa89dd6e700 1 -- 172.21.15.155:0/369924898 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa898106ed0 con 0x7fa8980f6510 2024-02-10T09:01:53.018 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:53.015+0000 7fa87ffff700 1 -- 172.21.15.155:0/369924898 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa88c013380 con 0x7fa8980f6510 2024-02-10T09:01:53.019 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:53.015+0000 7fa87ffff700 1 -- 172.21.15.155:0/369924898 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa88c014e50 con 0x7fa8980f6510 2024-02-10T09:01:53.019 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:53.015+0000 7fa89dd6e700 1 -- 172.21.15.155:0/369924898 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa898107360 con 0x7fa8980f6510 2024-02-10T09:01:53.019 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:53.016+0000 7fa87ffff700 1 -- 172.21.15.155:0/369924898 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa88c014420 con 0x7fa8980f6510 2024-02-10T09:01:53.019 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:53.016+0000 7fa87ffff700 1 --2- 172.21.15.155:0/369924898 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fa88002ff10 0x7fa8800323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:01:53.019 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:53.016+0000 7fa87ffff700 1 -- 172.21.15.155:0/369924898 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa88c040f40 con 0x7fa8980f6510 2024-02-10T09:01:53.019 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:53.016+0000 7fa89dd6e700 1 -- 172.21.15.155:0/369924898 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa898045ad0 con 0x7fa8980f6510 2024-02-10T09:01:53.019 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:53.016+0000 7fa896ffd700 1 --2- 172.21.15.155:0/369924898 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fa88002ff10 0x7fa8800323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:01:53.019 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:53.017+0000 7fa896ffd700 1 --2- 172.21.15.155:0/369924898 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fa88002ff10 0x7fa8800323d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fa8880096a0 tx=0x7fa888006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:01:53.022 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:53.021+0000 7fa87ffff700 1 -- 172.21.15.155:0/369924898 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa88c0095f0 con 0x7fa8980f6510 2024-02-10T09:01:53.228 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:53.227+0000 7fa89dd6e700 1 -- 172.21.15.155:0/369924898 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa898108210 con 0x7fa8980f6510 2024-02-10T09:01:53.228 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:53.228+0000 7fa87ffff700 1 -- 172.21.15.155:0/369924898 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa88c02b020 con 0x7fa8980f6510 2024-02-10T09:01:53.229 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:01:53.229 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:01:53.230 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:53.229+0000 7fa89dd6e700 1 -- 172.21.15.155:0/369924898 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fa88002ff10 msgr2=0x7fa8800323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:01:53.230 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:53.229+0000 7fa89dd6e700 1 --2- 172.21.15.155:0/369924898 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fa88002ff10 0x7fa8800323d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fa8880096a0 tx=0x7fa888006b40).stop 2024-02-10T09:01:53.230 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:53.229+0000 7fa89dd6e700 1 -- 172.21.15.155:0/369924898 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8980f6510 msgr2=0x7fa8981067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:01:53.231 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:53.229+0000 7fa89dd6e700 1 --2- 172.21.15.155:0/369924898 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8980f6510 0x7fa8981067e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fa88c005010 tx=0x7fa88c00b040).stop 2024-02-10T09:01:53.231 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:53.229+0000 7fa89dd6e700 1 -- 172.21.15.155:0/369924898 shutdown_connections 2024-02-10T09:01:53.231 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:53.229+0000 7fa89dd6e700 1 --2- 172.21.15.155:0/369924898 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fa88002ff10 0x7fa8800323d0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:01:53.231 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:53.229+0000 7fa89dd6e700 1 --2- 172.21.15.155:0/369924898 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8980f6510 0x7fa8981067e0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:01:53.231 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:53.229+0000 7fa89dd6e700 1 -- 172.21.15.155:0/369924898 >> 172.21.15.155:0/369924898 conn(0x7fa8980f1c70 msgr2=0x7fa8980f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:01:53.231 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:53.229+0000 7fa89dd6e700 1 -- 172.21.15.155:0/369924898 shutdown_connections 2024-02-10T09:01:53.231 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:53.229+0000 7fa89dd6e700 1 -- 172.21.15.155:0/369924898 wait complete. 2024-02-10T09:01:53.231 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:01:54.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:01:53 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:01:52.983361+0000 mgr.smithi077.dxntvw (mgr.14162) 94 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:01:54.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:01:53 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:01:53.229057+0000 mon.smithi077 (mon.0) 160 : audit [DBG] from='client.? 172.21.15.155:0/369924898' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:01:55.027 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:01:55.027 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:01:55.398 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:01:55 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:01:54.983649+0000 mgr.smithi077.dxntvw (mgr.14162) 95 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:01:57.012 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:56.990+0000 7f9fef59e700 1 -- 172.21.15.155:0/4049466408 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ff00f78e0 msgr2=0x7f9ff00f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:01:57.012 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:56.990+0000 7f9fef59e700 1 --2- 172.21.15.155:0/4049466408 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ff00f78e0 0x7f9ff00f7d00 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f9fe0009fb0 tx=0x7f9fe0009b30).stop 2024-02-10T09:01:57.012 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:56.990+0000 7f9fef59e700 1 -- 172.21.15.155:0/4049466408 shutdown_connections 2024-02-10T09:01:57.012 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:56.990+0000 7f9fef59e700 1 --2- 172.21.15.155:0/4049466408 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ff00f78e0 0x7f9ff00f7d00 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:01:57.012 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:56.990+0000 7f9fef59e700 1 -- 172.21.15.155:0/4049466408 >> 172.21.15.155:0/4049466408 conn(0x7f9ff00f0120 msgr2=0x7f9ff00f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:01:57.012 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:56.990+0000 7f9fef59e700 1 -- 172.21.15.155:0/4049466408 shutdown_connections 2024-02-10T09:01:57.013 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:56.990+0000 7f9fef59e700 1 -- 172.21.15.155:0/4049466408 wait complete. 2024-02-10T09:01:57.013 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:56.991+0000 7f9fef59e700 1 Processor -- start 2024-02-10T09:01:57.013 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:56.991+0000 7f9fef59e700 1 -- start start 2024-02-10T09:01:57.013 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:56.991+0000 7f9fef59e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ff00f78e0 0x7f9ff01065e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:01:57.013 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:56.991+0000 7f9fef59e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff0106ad0 con 0x7f9ff00f78e0 2024-02-10T09:01:57.013 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:56.991+0000 7f9fee59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ff00f78e0 0x7f9ff01065e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:01:57.013 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:56.991+0000 7f9fee59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ff00f78e0 0x7f9ff01065e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:44142/0 (socket says 172.21.15.155:44142) 2024-02-10T09:01:57.013 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:56.991+0000 7f9fee59c700 1 -- 172.21.15.155:0/1911177349 learned_addr learned my addr 172.21.15.155:0/1911177349 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:01:57.013 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:56.991+0000 7f9fee59c700 1 -- 172.21.15.155:0/1911177349 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fe0005040 con 0x7f9ff00f78e0 2024-02-10T09:01:57.013 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:56.991+0000 7f9fee59c700 1 --2- 172.21.15.155:0/1911177349 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ff00f78e0 0x7f9ff01065e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f9fe00089f0 tx=0x7f9fe000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:01:57.013 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:56.991+0000 7f9fdeffd700 1 -- 172.21.15.155:0/1911177349 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9fe0009320 con 0x7f9ff00f78e0 2024-02-10T09:01:57.013 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:56.992+0000 7f9fef59e700 1 -- 172.21.15.155:0/1911177349 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ff0106cd0 con 0x7f9ff00f78e0 2024-02-10T09:01:57.013 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:56.992+0000 7f9fdeffd700 1 -- 172.21.15.155:0/1911177349 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9fe0013380 con 0x7f9ff00f78e0 2024-02-10T09:01:57.013 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:56.992+0000 7f9fdeffd700 1 -- 172.21.15.155:0/1911177349 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9fe0014e20 con 0x7f9ff00f78e0 2024-02-10T09:01:57.014 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:56.992+0000 7f9fef59e700 1 -- 172.21.15.155:0/1911177349 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ff0107160 con 0x7f9ff00f78e0 2024-02-10T09:01:57.014 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:56.992+0000 7f9fdeffd700 1 -- 172.21.15.155:0/1911177349 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9fe00095f0 con 0x7f9ff00f78e0 2024-02-10T09:01:57.014 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:56.992+0000 7f9fef59e700 1 -- 172.21.15.155:0/1911177349 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ff0045ad0 con 0x7f9ff00f78e0 2024-02-10T09:01:57.014 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:56.992+0000 7f9fdeffd700 1 --2- 172.21.15.155:0/1911177349 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f9fd802ff10 0x7f9fd80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:01:57.014 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:56.993+0000 7f9fdeffd700 1 -- 172.21.15.155:0/1911177349 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9fe0040f10 con 0x7f9ff00f78e0 2024-02-10T09:01:57.014 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:56.993+0000 7f9fedd9b700 1 --2- 172.21.15.155:0/1911177349 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f9fd802ff10 0x7f9fd80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:01:57.014 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:56.993+0000 7f9fedd9b700 1 --2- 172.21.15.155:0/1911177349 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f9fd802ff10 0x7f9fd80323d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f9fe40096a0 tx=0x7f9fe4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:01:57.014 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:56.997+0000 7f9fdeffd700 1 -- 172.21.15.155:0/1911177349 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9fe0014420 con 0x7f9ff00f78e0 2024-02-10T09:01:57.203 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:57.202+0000 7f9fef59e700 1 -- 172.21.15.155:0/1911177349 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9ff0108010 con 0x7f9ff00f78e0 2024-02-10T09:01:57.204 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:57.203+0000 7f9fdeffd700 1 -- 172.21.15.155:0/1911177349 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9fe001d980 con 0x7f9ff00f78e0 2024-02-10T09:01:57.205 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:01:57.205 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:01:57.206 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:57.204+0000 7f9fef59e700 1 -- 172.21.15.155:0/1911177349 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f9fd802ff10 msgr2=0x7f9fd80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:01:57.206 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:57.204+0000 7f9fef59e700 1 --2- 172.21.15.155:0/1911177349 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f9fd802ff10 0x7f9fd80323d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f9fe40096a0 tx=0x7f9fe4006b40).stop 2024-02-10T09:01:57.207 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:57.204+0000 7f9fef59e700 1 -- 172.21.15.155:0/1911177349 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ff00f78e0 msgr2=0x7f9ff01065e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:01:57.207 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:57.204+0000 7f9fef59e700 1 --2- 172.21.15.155:0/1911177349 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ff00f78e0 0x7f9ff01065e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f9fe00089f0 tx=0x7f9fe000b040).stop 2024-02-10T09:01:57.207 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:57.204+0000 7f9fef59e700 1 -- 172.21.15.155:0/1911177349 shutdown_connections 2024-02-10T09:01:57.207 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:57.204+0000 7f9fef59e700 1 --2- 172.21.15.155:0/1911177349 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f9fd802ff10 0x7f9fd80323d0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:01:57.207 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:57.204+0000 7f9fef59e700 1 --2- 172.21.15.155:0/1911177349 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ff00f78e0 0x7f9ff01065e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:01:57.207 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:57.204+0000 7f9fef59e700 1 -- 172.21.15.155:0/1911177349 >> 172.21.15.155:0/1911177349 conn(0x7f9ff00f0120 msgr2=0x7f9ff00f0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:01:57.207 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:57.204+0000 7f9fef59e700 1 -- 172.21.15.155:0/1911177349 shutdown_connections 2024-02-10T09:01:57.208 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:01:57.204+0000 7f9fef59e700 1 -- 172.21.15.155:0/1911177349 wait complete. 2024-02-10T09:01:57.208 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:01:57.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:01:57 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:01:56.983958+0000 mgr.smithi077.dxntvw (mgr.14162) 96 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:01:58.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:01:58 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:01:57.204145+0000 mon.smithi077 (mon.0) 161 : audit [DBG] from='client.? 172.21.15.155:0/1911177349' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:01:58.987 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:01:58.988 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:01:59.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:01:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:01:58.984287+0000 mgr.smithi077.dxntvw (mgr.14162) 97 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:02:00.901 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:00.876+0000 7fb87f54d700 1 -- 172.21.15.155:0/2798519080 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8780f6510 msgr2=0x7fb8780f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:02:00.901 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:00.876+0000 7fb87f54d700 1 --2- 172.21.15.155:0/2798519080 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8780f6510 0x7fb8780f6930 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7fb868009fb0 tx=0x7fb868009b30).stop 2024-02-10T09:02:00.901 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:00.876+0000 7fb87f54d700 1 -- 172.21.15.155:0/2798519080 shutdown_connections 2024-02-10T09:02:00.902 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:00.876+0000 7fb87f54d700 1 --2- 172.21.15.155:0/2798519080 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8780f6510 0x7fb8780f6930 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:02:00.902 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:00.876+0000 7fb87f54d700 1 -- 172.21.15.155:0/2798519080 >> 172.21.15.155:0/2798519080 conn(0x7fb8780f1c70 msgr2=0x7fb8780f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:02:00.902 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:00.876+0000 7fb87f54d700 1 -- 172.21.15.155:0/2798519080 shutdown_connections 2024-02-10T09:02:00.902 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:00.876+0000 7fb87f54d700 1 -- 172.21.15.155:0/2798519080 wait complete. 2024-02-10T09:02:00.902 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:00.877+0000 7fb87f54d700 1 Processor -- start 2024-02-10T09:02:00.902 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:00.877+0000 7fb87f54d700 1 -- start start 2024-02-10T09:02:00.902 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:00.877+0000 7fb87f54d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8780f6510 0x7fb8781067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:02:00.902 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:00.877+0000 7fb87f54d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb878106cd0 con 0x7fb8780f6510 2024-02-10T09:02:00.902 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:00.877+0000 7fb87d2e9700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8780f6510 0x7fb8781067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:02:00.902 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:00.877+0000 7fb87d2e9700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8780f6510 0x7fb8781067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:44164/0 (socket says 172.21.15.155:44164) 2024-02-10T09:02:00.903 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:00.877+0000 7fb87d2e9700 1 -- 172.21.15.155:0/3799270120 learned_addr learned my addr 172.21.15.155:0/3799270120 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:02:00.903 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:00.877+0000 7fb87d2e9700 1 -- 172.21.15.155:0/3799270120 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb868005040 con 0x7fb8780f6510 2024-02-10T09:02:00.903 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:00.878+0000 7fb87d2e9700 1 --2- 172.21.15.155:0/3799270120 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8780f6510 0x7fb8781067e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fb8680089f0 tx=0x7fb86800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:02:00.903 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:00.878+0000 7fb86dffb700 1 -- 172.21.15.155:0/3799270120 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb868009320 con 0x7fb8780f6510 2024-02-10T09:02:00.903 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:00.878+0000 7fb86dffb700 1 -- 172.21.15.155:0/3799270120 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb868013380 con 0x7fb8780f6510 2024-02-10T09:02:00.903 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:00.878+0000 7fb87f54d700 1 -- 172.21.15.155:0/3799270120 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb878106ed0 con 0x7fb8780f6510 2024-02-10T09:02:00.903 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:00.878+0000 7fb86dffb700 1 -- 172.21.15.155:0/3799270120 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb86801ed80 con 0x7fb8780f6510 2024-02-10T09:02:00.903 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:00.878+0000 7fb87f54d700 1 -- 172.21.15.155:0/3799270120 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb878107360 con 0x7fb8780f6510 2024-02-10T09:02:00.904 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:00.879+0000 7fb86dffb700 1 -- 172.21.15.155:0/3799270120 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb86801e420 con 0x7fb8780f6510 2024-02-10T09:02:00.904 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:00.879+0000 7fb87f54d700 1 -- 172.21.15.155:0/3799270120 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb878045ad0 con 0x7fb8780f6510 2024-02-10T09:02:00.904 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:00.879+0000 7fb86dffb700 1 --2- 172.21.15.155:0/3799270120 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb86402ff60 0x7fb864032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:02:00.904 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:00.879+0000 7fb86dffb700 1 -- 172.21.15.155:0/3799270120 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb868040fc0 con 0x7fb8780f6510 2024-02-10T09:02:00.904 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:00.879+0000 7fb87cae8700 1 --2- 172.21.15.155:0/3799270120 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb86402ff60 0x7fb864032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:02:00.904 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:00.880+0000 7fb87cae8700 1 --2- 172.21.15.155:0/3799270120 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb86402ff60 0x7fb864032420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fb8740096a0 tx=0x7fb874006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:02:00.904 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:00.884+0000 7fb86dffb700 1 -- 172.21.15.155:0/3799270120 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb8680095f0 con 0x7fb8780f6510 2024-02-10T09:02:01.091 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:01.090+0000 7fb87f54d700 1 -- 172.21.15.155:0/3799270120 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb878108210 con 0x7fb8780f6510 2024-02-10T09:02:01.092 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:01.091+0000 7fb86dffb700 1 -- 172.21.15.155:0/3799270120 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb868004490 con 0x7fb8780f6510 2024-02-10T09:02:01.092 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:02:01.092 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:02:01.093 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:01.092+0000 7fb87f54d700 1 -- 172.21.15.155:0/3799270120 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb86402ff60 msgr2=0x7fb864032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:02:01.094 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:01.092+0000 7fb87f54d700 1 --2- 172.21.15.155:0/3799270120 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb86402ff60 0x7fb864032420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fb8740096a0 tx=0x7fb874006b40).stop 2024-02-10T09:02:01.094 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:01.092+0000 7fb87f54d700 1 -- 172.21.15.155:0/3799270120 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8780f6510 msgr2=0x7fb8781067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:02:01.094 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:01.092+0000 7fb87f54d700 1 --2- 172.21.15.155:0/3799270120 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8780f6510 0x7fb8781067e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fb8680089f0 tx=0x7fb86800b040).stop 2024-02-10T09:02:01.094 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:01.092+0000 7fb87f54d700 1 -- 172.21.15.155:0/3799270120 shutdown_connections 2024-02-10T09:02:01.094 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:01.092+0000 7fb87f54d700 1 --2- 172.21.15.155:0/3799270120 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb86402ff60 0x7fb864032420 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:02:01.094 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:01.092+0000 7fb87f54d700 1 --2- 172.21.15.155:0/3799270120 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8780f6510 0x7fb8781067e0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:02:01.094 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:01.092+0000 7fb87f54d700 1 -- 172.21.15.155:0/3799270120 >> 172.21.15.155:0/3799270120 conn(0x7fb8780f1c70 msgr2=0x7fb8780f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:02:01.094 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:01.092+0000 7fb87f54d700 1 -- 172.21.15.155:0/3799270120 shutdown_connections 2024-02-10T09:02:01.094 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:01.092+0000 7fb87f54d700 1 -- 172.21.15.155:0/3799270120 wait complete. 2024-02-10T09:02:01.095 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:02:01.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:02:01 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:02:00.984534+0000 mgr.smithi077.dxntvw (mgr.14162) 98 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:02:02.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:02:02 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:02:01.092146+0000 mon.smithi077 (mon.0) 162 : audit [DBG] from='client.? 172.21.15.155:0/3799270120' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:02:02.875 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:02:02.875 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:02:04.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:02:03 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:02:02.984935+0000 mgr.smithi077.dxntvw (mgr.14162) 99 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:02:04.820 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:04.809+0000 7f81f2aa3700 1 -- 172.21.15.155:0/1083535531 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81ec0f51a0 msgr2=0x7f81ec0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:02:04.821 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:04.809+0000 7f81f2aa3700 1 --2- 172.21.15.155:0/1083535531 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81ec0f51a0 0x7f81ec0f55c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f81dc009fb0 tx=0x7f81dc009b30).stop 2024-02-10T09:02:04.821 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:04.809+0000 7f81f2aa3700 1 -- 172.21.15.155:0/1083535531 shutdown_connections 2024-02-10T09:02:04.821 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:04.809+0000 7f81f2aa3700 1 --2- 172.21.15.155:0/1083535531 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81ec0f51a0 0x7f81ec0f55c0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:02:04.821 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:04.809+0000 7f81f2aa3700 1 -- 172.21.15.155:0/1083535531 >> 172.21.15.155:0/1083535531 conn(0x7f81ec0f09a0 msgr2=0x7f81ec0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:02:04.821 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:04.809+0000 7f81f2aa3700 1 -- 172.21.15.155:0/1083535531 shutdown_connections 2024-02-10T09:02:04.821 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:04.809+0000 7f81f2aa3700 1 -- 172.21.15.155:0/1083535531 wait complete. 2024-02-10T09:02:04.821 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:04.809+0000 7f81f2aa3700 1 Processor -- start 2024-02-10T09:02:04.821 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:04.809+0000 7f81f2aa3700 1 -- start start 2024-02-10T09:02:04.821 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:04.809+0000 7f81f2aa3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81ec0f51a0 0x7f81ec10b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:02:04.821 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:04.809+0000 7f81f2aa3700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81ec1070b0 con 0x7f81ec0f51a0 2024-02-10T09:02:04.821 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:04.810+0000 7f81f1aa1700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81ec0f51a0 0x7f81ec10b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:02:04.821 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:04.810+0000 7f81f1aa1700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81ec0f51a0 0x7f81ec10b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:55996/0 (socket says 172.21.15.155:55996) 2024-02-10T09:02:04.821 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:04.810+0000 7f81f1aa1700 1 -- 172.21.15.155:0/404053833 learned_addr learned my addr 172.21.15.155:0/404053833 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:02:04.822 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:04.810+0000 7f81f1aa1700 1 -- 172.21.15.155:0/404053833 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81dc005040 con 0x7f81ec0f51a0 2024-02-10T09:02:04.822 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:04.810+0000 7f81f1aa1700 1 --2- 172.21.15.155:0/404053833 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81ec0f51a0 0x7f81ec10b6c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f81dc0089f0 tx=0x7f81dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:02:04.822 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:04.810+0000 7f81e27fc700 1 -- 172.21.15.155:0/404053833 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f81dc009320 con 0x7f81ec0f51a0 2024-02-10T09:02:04.822 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:04.810+0000 7f81f2aa3700 1 -- 172.21.15.155:0/404053833 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81ec060220 con 0x7f81ec0f51a0 2024-02-10T09:02:04.822 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:04.810+0000 7f81e27fc700 1 -- 172.21.15.155:0/404053833 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f81dc013380 con 0x7f81ec0f51a0 2024-02-10T09:02:04.822 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:04.810+0000 7f81e27fc700 1 -- 172.21.15.155:0/404053833 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f81dc014e20 con 0x7f81ec0f51a0 2024-02-10T09:02:04.822 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:04.810+0000 7f81f2aa3700 1 -- 172.21.15.155:0/404053833 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81ec10bbb0 con 0x7f81ec0f51a0 2024-02-10T09:02:04.822 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:04.811+0000 7f81e27fc700 1 -- 172.21.15.155:0/404053833 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f81dc014420 con 0x7f81ec0f51a0 2024-02-10T09:02:04.822 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:04.811+0000 7f81f2aa3700 1 -- 172.21.15.155:0/404053833 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81ec045ad0 con 0x7f81ec0f51a0 2024-02-10T09:02:04.822 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:04.811+0000 7f81e27fc700 1 --2- 172.21.15.155:0/404053833 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f81d802ff10 0x7f81d80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:02:04.822 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:04.811+0000 7f81e27fc700 1 -- 172.21.15.155:0/404053833 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f81dc040fc0 con 0x7f81ec0f51a0 2024-02-10T09:02:04.822 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:04.811+0000 7f81f12a0700 1 --2- 172.21.15.155:0/404053833 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f81d802ff10 0x7f81d80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:02:04.822 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:04.812+0000 7f81f12a0700 1 --2- 172.21.15.155:0/404053833 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f81d802ff10 0x7f81d80323d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f81e80096a0 tx=0x7f81e8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:02:04.822 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:04.816+0000 7f81e27fc700 1 -- 172.21.15.155:0/404053833 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f81dc0095f0 con 0x7f81ec0f51a0 2024-02-10T09:02:05.026 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:05.026+0000 7f81f2aa3700 1 -- 172.21.15.155:0/404053833 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f81ec10c370 con 0x7f81ec0f51a0 2024-02-10T09:02:05.027 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:05.026+0000 7f81e27fc700 1 -- 172.21.15.155:0/404053833 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f81dc01db70 con 0x7f81ec0f51a0 2024-02-10T09:02:05.027 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:02:05.027 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:02:05.029 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:05.028+0000 7f81f2aa3700 1 -- 172.21.15.155:0/404053833 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f81d802ff10 msgr2=0x7f81d80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:02:05.029 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:05.028+0000 7f81f2aa3700 1 --2- 172.21.15.155:0/404053833 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f81d802ff10 0x7f81d80323d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f81e80096a0 tx=0x7f81e8006b40).stop 2024-02-10T09:02:05.029 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:05.028+0000 7f81f2aa3700 1 -- 172.21.15.155:0/404053833 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81ec0f51a0 msgr2=0x7f81ec10b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:02:05.029 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:05.028+0000 7f81f2aa3700 1 --2- 172.21.15.155:0/404053833 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81ec0f51a0 0x7f81ec10b6c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f81dc0089f0 tx=0x7f81dc00b040).stop 2024-02-10T09:02:05.029 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:05.028+0000 7f81f2aa3700 1 -- 172.21.15.155:0/404053833 shutdown_connections 2024-02-10T09:02:05.029 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:05.028+0000 7f81f2aa3700 1 --2- 172.21.15.155:0/404053833 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f81d802ff10 0x7f81d80323d0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:02:05.029 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:05.028+0000 7f81f2aa3700 1 --2- 172.21.15.155:0/404053833 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81ec0f51a0 0x7f81ec10b6c0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:02:05.029 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:05.028+0000 7f81f2aa3700 1 -- 172.21.15.155:0/404053833 >> 172.21.15.155:0/404053833 conn(0x7f81ec0f09a0 msgr2=0x7f81ec0f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:02:05.029 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:05.028+0000 7f81f2aa3700 1 -- 172.21.15.155:0/404053833 shutdown_connections 2024-02-10T09:02:05.030 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:05.028+0000 7f81f2aa3700 1 -- 172.21.15.155:0/404053833 wait complete. 2024-02-10T09:02:05.030 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:02:05.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:02:05 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:02:04.985382+0000 mgr.smithi077.dxntvw (mgr.14162) 100 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:02:05.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:02:05 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:02:05.027601+0000 mon.smithi077 (mon.0) 163 : audit [DBG] from='client.? 172.21.15.155:0/404053833' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:02:06.572 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:02:06.572 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:02:07.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:02:07 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:02:06.985722+0000 mgr.smithi077.dxntvw (mgr.14162) 101 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:02:08.666 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:08.635+0000 7f2c6493b700 1 -- 172.21.15.155:0/645802438 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c600f5410 msgr2=0x7f2c600f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:02:08.666 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:08.635+0000 7f2c6493b700 1 --2- 172.21.15.155:0/645802438 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c600f5410 0x7f2c600f5830 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f2c54009fb0 tx=0x7f2c54009b30).stop 2024-02-10T09:02:08.667 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:08.635+0000 7f2c6493b700 1 -- 172.21.15.155:0/645802438 shutdown_connections 2024-02-10T09:02:08.667 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:08.635+0000 7f2c6493b700 1 --2- 172.21.15.155:0/645802438 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c600f5410 0x7f2c600f5830 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:02:08.667 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:08.635+0000 7f2c6493b700 1 -- 172.21.15.155:0/645802438 >> 172.21.15.155:0/645802438 conn(0x7f2c600f0b70 msgr2=0x7f2c600f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:02:08.667 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:08.635+0000 7f2c6493b700 1 -- 172.21.15.155:0/645802438 shutdown_connections 2024-02-10T09:02:08.667 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:08.635+0000 7f2c6493b700 1 -- 172.21.15.155:0/645802438 wait complete. 2024-02-10T09:02:08.667 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:08.636+0000 7f2c6493b700 1 Processor -- start 2024-02-10T09:02:08.667 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:08.636+0000 7f2c6493b700 1 -- start start 2024-02-10T09:02:08.667 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:08.636+0000 7f2c6493b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c600f5410 0x7f2c60109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:02:08.667 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:08.636+0000 7f2c6493b700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c60060220 con 0x7f2c600f5410 2024-02-10T09:02:08.667 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:08.636+0000 7f2c5e59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c600f5410 0x7f2c60109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:02:08.667 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:08.636+0000 7f2c5e59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c600f5410 0x7f2c60109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:56008/0 (socket says 172.21.15.155:56008) 2024-02-10T09:02:08.667 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:08.636+0000 7f2c5e59c700 1 -- 172.21.15.155:0/2174109351 learned_addr learned my addr 172.21.15.155:0/2174109351 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:02:08.667 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:08.637+0000 7f2c5e59c700 1 -- 172.21.15.155:0/2174109351 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c54005040 con 0x7f2c600f5410 2024-02-10T09:02:08.667 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:08.637+0000 7f2c5e59c700 1 --2- 172.21.15.155:0/2174109351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c600f5410 0x7f2c60109500 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f2c54005010 tx=0x7f2c5400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:02:08.668 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:08.637+0000 7f2c4effd700 1 -- 172.21.15.155:0/2174109351 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c54009320 con 0x7f2c600f5410 2024-02-10T09:02:08.668 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:08.637+0000 7f2c6493b700 1 -- 172.21.15.155:0/2174109351 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c60060420 con 0x7f2c600f5410 2024-02-10T09:02:08.668 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:08.637+0000 7f2c4effd700 1 -- 172.21.15.155:0/2174109351 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2c54013380 con 0x7f2c600f5410 2024-02-10T09:02:08.668 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:08.637+0000 7f2c4effd700 1 -- 172.21.15.155:0/2174109351 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c54014e50 con 0x7f2c600f5410 2024-02-10T09:02:08.668 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:08.637+0000 7f2c6493b700 1 -- 172.21.15.155:0/2174109351 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c60109bb0 con 0x7f2c600f5410 2024-02-10T09:02:08.668 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:08.638+0000 7f2c6493b700 1 -- 172.21.15.155:0/2174109351 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c60059100 con 0x7f2c600f5410 2024-02-10T09:02:08.668 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:08.638+0000 7f2c4effd700 1 -- 172.21.15.155:0/2174109351 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2c540095f0 con 0x7f2c600f5410 2024-02-10T09:02:08.668 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:08.638+0000 7f2c4effd700 1 --2- 172.21.15.155:0/2174109351 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f2c4802ff10 0x7f2c480323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:02:08.668 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:08.638+0000 7f2c4effd700 1 -- 172.21.15.155:0/2174109351 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2c54040ed0 con 0x7f2c600f5410 2024-02-10T09:02:08.668 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:08.638+0000 7f2c5dd9b700 1 --2- 172.21.15.155:0/2174109351 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f2c4802ff10 0x7f2c480323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:02:08.668 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:08.638+0000 7f2c5dd9b700 1 --2- 172.21.15.155:0/2174109351 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f2c4802ff10 0x7f2c480323d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f2c500096a0 tx=0x7f2c50006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:02:08.668 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:08.642+0000 7f2c4effd700 1 -- 172.21.15.155:0/2174109351 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2c54014420 con 0x7f2c600f5410 2024-02-10T09:02:08.855 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:08.854+0000 7f2c6493b700 1 -- 172.21.15.155:0/2174109351 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2c60045ad0 con 0x7f2c600f5410 2024-02-10T09:02:08.855 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:08.855+0000 7f2c4effd700 1 -- 172.21.15.155:0/2174109351 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2c54010230 con 0x7f2c600f5410 2024-02-10T09:02:08.856 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:02:08.856 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:02:08.857 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:08.856+0000 7f2c6493b700 1 -- 172.21.15.155:0/2174109351 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f2c4802ff10 msgr2=0x7f2c480323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:02:08.857 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:08.856+0000 7f2c6493b700 1 --2- 172.21.15.155:0/2174109351 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f2c4802ff10 0x7f2c480323d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f2c500096a0 tx=0x7f2c50006b40).stop 2024-02-10T09:02:08.858 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:08.856+0000 7f2c6493b700 1 -- 172.21.15.155:0/2174109351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c600f5410 msgr2=0x7f2c60109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:02:08.858 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:08.856+0000 7f2c6493b700 1 --2- 172.21.15.155:0/2174109351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c600f5410 0x7f2c60109500 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f2c54005010 tx=0x7f2c5400b040).stop 2024-02-10T09:02:08.858 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:08.856+0000 7f2c6493b700 1 -- 172.21.15.155:0/2174109351 shutdown_connections 2024-02-10T09:02:08.858 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:08.856+0000 7f2c6493b700 1 --2- 172.21.15.155:0/2174109351 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f2c4802ff10 0x7f2c480323d0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:02:08.858 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:08.856+0000 7f2c6493b700 1 --2- 172.21.15.155:0/2174109351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2c600f5410 0x7f2c60109500 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:02:08.858 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:08.856+0000 7f2c6493b700 1 -- 172.21.15.155:0/2174109351 >> 172.21.15.155:0/2174109351 conn(0x7f2c600f0b70 msgr2=0x7f2c600f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:02:08.858 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:08.856+0000 7f2c6493b700 1 -- 172.21.15.155:0/2174109351 shutdown_connections 2024-02-10T09:02:08.858 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:08.856+0000 7f2c6493b700 1 -- 172.21.15.155:0/2174109351 wait complete. 2024-02-10T09:02:08.858 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:02:09.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:02:08 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:02:08.856040+0000 mon.smithi077 (mon.0) 164 : audit [DBG] from='client.? 172.21.15.155:0/2174109351' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:02:10.398 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:02:09 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:02:08.986181+0000 mgr.smithi077.dxntvw (mgr.14162) 102 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:02:10.598 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:02:10.598 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:02:11.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:02:11 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:02:10.986575+0000 mgr.smithi077.dxntvw (mgr.14162) 103 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:02:12.568 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:12.539+0000 7f3d7cb7f700 1 -- 172.21.15.155:0/365127336 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d780f5410 msgr2=0x7f3d780f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:02:12.569 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:12.539+0000 7f3d7cb7f700 1 --2- 172.21.15.155:0/365127336 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d780f5410 0x7f3d780f5830 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f3d6c009fb0 tx=0x7f3d6c009b30).stop 2024-02-10T09:02:12.569 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:12.539+0000 7f3d7cb7f700 1 -- 172.21.15.155:0/365127336 shutdown_connections 2024-02-10T09:02:12.569 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:12.539+0000 7f3d7cb7f700 1 --2- 172.21.15.155:0/365127336 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d780f5410 0x7f3d780f5830 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:02:12.569 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:12.539+0000 7f3d7cb7f700 1 -- 172.21.15.155:0/365127336 >> 172.21.15.155:0/365127336 conn(0x7f3d780f0b70 msgr2=0x7f3d780f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:02:12.569 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:12.539+0000 7f3d7cb7f700 1 -- 172.21.15.155:0/365127336 shutdown_connections 2024-02-10T09:02:12.569 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:12.540+0000 7f3d7cb7f700 1 -- 172.21.15.155:0/365127336 wait complete. 2024-02-10T09:02:12.569 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:12.540+0000 7f3d7cb7f700 1 Processor -- start 2024-02-10T09:02:12.569 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:12.540+0000 7f3d7cb7f700 1 -- start start 2024-02-10T09:02:12.569 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:12.540+0000 7f3d7cb7f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d780f5410 0x7f3d781056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:02:12.569 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:12.540+0000 7f3d7cb7f700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d78105bd0 con 0x7f3d780f5410 2024-02-10T09:02:12.569 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:12.540+0000 7f3d7659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d780f5410 0x7f3d781056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:02:12.569 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:12.540+0000 7f3d7659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d780f5410 0x7f3d781056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:36724/0 (socket says 172.21.15.155:36724) 2024-02-10T09:02:12.570 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:12.540+0000 7f3d7659c700 1 -- 172.21.15.155:0/3826858055 learned_addr learned my addr 172.21.15.155:0/3826858055 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:02:12.570 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:12.541+0000 7f3d7659c700 1 -- 172.21.15.155:0/3826858055 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d6c005040 con 0x7f3d780f5410 2024-02-10T09:02:12.570 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:12.541+0000 7f3d7659c700 1 --2- 172.21.15.155:0/3826858055 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d780f5410 0x7f3d781056e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f3d6c005010 tx=0x7f3d6c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:02:12.570 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:12.541+0000 7f3d66ffd700 1 -- 172.21.15.155:0/3826858055 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d6c009320 con 0x7f3d780f5410 2024-02-10T09:02:12.570 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:12.541+0000 7f3d7cb7f700 1 -- 172.21.15.155:0/3826858055 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d78105dd0 con 0x7f3d780f5410 2024-02-10T09:02:12.570 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:12.541+0000 7f3d66ffd700 1 -- 172.21.15.155:0/3826858055 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3d6c013380 con 0x7f3d780f5410 2024-02-10T09:02:12.570 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:12.541+0000 7f3d66ffd700 1 -- 172.21.15.155:0/3826858055 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d6c014e50 con 0x7f3d780f5410 2024-02-10T09:02:12.570 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:12.541+0000 7f3d7cb7f700 1 -- 172.21.15.155:0/3826858055 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d78106260 con 0x7f3d780f5410 2024-02-10T09:02:12.570 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:12.542+0000 7f3d66ffd700 1 -- 172.21.15.155:0/3826858055 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3d6c0263a0 con 0x7f3d780f5410 2024-02-10T09:02:12.570 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:12.542+0000 7f3d66ffd700 1 --2- 172.21.15.155:0/3826858055 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f3d6002ff10 0x7f3d600323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:02:12.570 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:12.542+0000 7f3d66ffd700 1 -- 172.21.15.155:0/3826858055 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3d6c040f70 con 0x7f3d780f5410 2024-02-10T09:02:12.570 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:12.542+0000 7f3d7cb7f700 1 -- 172.21.15.155:0/3826858055 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d78045ad0 con 0x7f3d780f5410 2024-02-10T09:02:12.570 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:12.542+0000 7f3d75d9b700 1 --2- 172.21.15.155:0/3826858055 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f3d6002ff10 0x7f3d600323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:02:12.571 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:12.543+0000 7f3d75d9b700 1 --2- 172.21.15.155:0/3826858055 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f3d6002ff10 0x7f3d600323d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f3d680096a0 tx=0x7f3d68006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:02:12.571 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:12.547+0000 7f3d66ffd700 1 -- 172.21.15.155:0/3826858055 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3d6c012920 con 0x7f3d780f5410 2024-02-10T09:02:12.754 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:12.753+0000 7f3d7cb7f700 1 -- 172.21.15.155:0/3826858055 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3d78106fa0 con 0x7f3d780f5410 2024-02-10T09:02:12.754 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:12.754+0000 7f3d66ffd700 1 -- 172.21.15.155:0/3826858055 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3d6c02b020 con 0x7f3d780f5410 2024-02-10T09:02:12.755 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:02:12.755 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:02:12.756 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:12.755+0000 7f3d7cb7f700 1 -- 172.21.15.155:0/3826858055 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f3d6002ff10 msgr2=0x7f3d600323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:02:12.756 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:12.755+0000 7f3d7cb7f700 1 --2- 172.21.15.155:0/3826858055 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f3d6002ff10 0x7f3d600323d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f3d680096a0 tx=0x7f3d68006b40).stop 2024-02-10T09:02:12.756 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:12.755+0000 7f3d7cb7f700 1 -- 172.21.15.155:0/3826858055 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d780f5410 msgr2=0x7f3d781056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:02:12.756 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:12.755+0000 7f3d7cb7f700 1 --2- 172.21.15.155:0/3826858055 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d780f5410 0x7f3d781056e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f3d6c005010 tx=0x7f3d6c00b040).stop 2024-02-10T09:02:12.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:12.755+0000 7f3d7cb7f700 1 -- 172.21.15.155:0/3826858055 shutdown_connections 2024-02-10T09:02:12.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:12.755+0000 7f3d7cb7f700 1 --2- 172.21.15.155:0/3826858055 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f3d6002ff10 0x7f3d600323d0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:02:12.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:12.755+0000 7f3d7cb7f700 1 --2- 172.21.15.155:0/3826858055 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d780f5410 0x7f3d781056e0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:02:12.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:12.755+0000 7f3d7cb7f700 1 -- 172.21.15.155:0/3826858055 >> 172.21.15.155:0/3826858055 conn(0x7f3d780f0b70 msgr2=0x7f3d780f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:02:12.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:12.755+0000 7f3d7cb7f700 1 -- 172.21.15.155:0/3826858055 shutdown_connections 2024-02-10T09:02:12.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:12.755+0000 7f3d7cb7f700 1 -- 172.21.15.155:0/3826858055 wait complete. 2024-02-10T09:02:12.757 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:02:14.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:02:13 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:02:12.755007+0000 mon.smithi077 (mon.0) 165 : audit [DBG] from='client.? 172.21.15.155:0/3826858055' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:02:14.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:02:13 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:02:12.986962+0000 mgr.smithi077.dxntvw (mgr.14162) 104 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:02:14.473 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:02:14.474 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:02:15.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:02:15 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:02:14.987362+0000 mgr.smithi077.dxntvw (mgr.14162) 105 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:02:16.463 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:16.431+0000 7f6f8359e700 1 -- 172.21.15.155:0/3403003657 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f840f51a0 msgr2=0x7f6f840f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:02:16.463 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:16.431+0000 7f6f8359e700 1 --2- 172.21.15.155:0/3403003657 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f840f51a0 0x7f6f840f55c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f6f78009fb0 tx=0x7f6f78009b30).stop 2024-02-10T09:02:16.464 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:16.432+0000 7f6f8359e700 1 -- 172.21.15.155:0/3403003657 shutdown_connections 2024-02-10T09:02:16.464 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:16.432+0000 7f6f8359e700 1 --2- 172.21.15.155:0/3403003657 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f840f51a0 0x7f6f840f55c0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:02:16.464 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:16.432+0000 7f6f8359e700 1 -- 172.21.15.155:0/3403003657 >> 172.21.15.155:0/3403003657 conn(0x7f6f840f09a0 msgr2=0x7f6f840f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:02:16.464 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:16.432+0000 7f6f8359e700 1 -- 172.21.15.155:0/3403003657 shutdown_connections 2024-02-10T09:02:16.464 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:16.432+0000 7f6f8359e700 1 -- 172.21.15.155:0/3403003657 wait complete. 2024-02-10T09:02:16.464 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:16.432+0000 7f6f8359e700 1 Processor -- start 2024-02-10T09:02:16.464 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:16.432+0000 7f6f8359e700 1 -- start start 2024-02-10T09:02:16.464 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:16.433+0000 7f6f8359e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f840f51a0 0x7f6f84109260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:02:16.464 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:16.433+0000 7f6f8359e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f84060280 con 0x7f6f840f51a0 2024-02-10T09:02:16.464 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:16.433+0000 7f6f8259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f840f51a0 0x7f6f84109260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:02:16.464 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:16.433+0000 7f6f8259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f840f51a0 0x7f6f84109260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:36734/0 (socket says 172.21.15.155:36734) 2024-02-10T09:02:16.464 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:16.433+0000 7f6f8259c700 1 -- 172.21.15.155:0/2983121261 learned_addr learned my addr 172.21.15.155:0/2983121261 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:02:16.464 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:16.433+0000 7f6f8259c700 1 -- 172.21.15.155:0/2983121261 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f78005040 con 0x7f6f840f51a0 2024-02-10T09:02:16.465 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:16.433+0000 7f6f8259c700 1 --2- 172.21.15.155:0/2983121261 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f840f51a0 0x7f6f84109260 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f6f78009830 tx=0x7f6f7800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:02:16.465 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:16.433+0000 7f6f72ffd700 1 -- 172.21.15.155:0/2983121261 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f78009320 con 0x7f6f840f51a0 2024-02-10T09:02:16.465 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:16.433+0000 7f6f8359e700 1 -- 172.21.15.155:0/2983121261 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f84060480 con 0x7f6f840f51a0 2024-02-10T09:02:16.465 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:16.433+0000 7f6f72ffd700 1 -- 172.21.15.155:0/2983121261 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6f78013380 con 0x7f6f840f51a0 2024-02-10T09:02:16.465 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:16.433+0000 7f6f72ffd700 1 -- 172.21.15.155:0/2983121261 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f78014420 con 0x7f6f840f51a0 2024-02-10T09:02:16.465 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:16.433+0000 7f6f8359e700 1 -- 172.21.15.155:0/2983121261 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f84109990 con 0x7f6f840f51a0 2024-02-10T09:02:16.465 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:16.434+0000 7f6f72ffd700 1 -- 172.21.15.155:0/2983121261 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6f78014580 con 0x7f6f840f51a0 2024-02-10T09:02:16.465 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:16.434+0000 7f6f72ffd700 1 --2- 172.21.15.155:0/2983121261 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f6f6c02ff60 0x7f6f6c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:02:16.465 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:16.434+0000 7f6f72ffd700 1 -- 172.21.15.155:0/2983121261 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6f78040f20 con 0x7f6f840f51a0 2024-02-10T09:02:16.465 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:16.434+0000 7f6f8359e700 1 -- 172.21.15.155:0/2983121261 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f84045ad0 con 0x7f6f840f51a0 2024-02-10T09:02:16.465 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:16.435+0000 7f6f81d9b700 1 --2- 172.21.15.155:0/2983121261 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f6f6c02ff60 0x7f6f6c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:02:16.465 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:16.435+0000 7f6f81d9b700 1 --2- 172.21.15.155:0/2983121261 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f6f6c02ff60 0x7f6f6c032420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f6f740096a0 tx=0x7f6f74006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:02:16.465 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:16.439+0000 7f6f72ffd700 1 -- 172.21.15.155:0/2983121261 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6f780273f0 con 0x7f6f840f51a0 2024-02-10T09:02:16.649 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:16.648+0000 7f6f8359e700 1 -- 172.21.15.155:0/2983121261 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6f8410a0e0 con 0x7f6f840f51a0 2024-02-10T09:02:16.650 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:16.648+0000 7f6f72ffd700 1 -- 172.21.15.155:0/2983121261 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6f7801c890 con 0x7f6f840f51a0 2024-02-10T09:02:16.650 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:02:16.650 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:02:16.652 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:16.650+0000 7f6f8359e700 1 -- 172.21.15.155:0/2983121261 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f6f6c02ff60 msgr2=0x7f6f6c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:02:16.652 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:16.650+0000 7f6f8359e700 1 --2- 172.21.15.155:0/2983121261 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f6f6c02ff60 0x7f6f6c032420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f6f740096a0 tx=0x7f6f74006b40).stop 2024-02-10T09:02:16.652 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:16.650+0000 7f6f8359e700 1 -- 172.21.15.155:0/2983121261 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f840f51a0 msgr2=0x7f6f84109260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:02:16.652 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:16.650+0000 7f6f8359e700 1 --2- 172.21.15.155:0/2983121261 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f840f51a0 0x7f6f84109260 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f6f78009830 tx=0x7f6f7800b040).stop 2024-02-10T09:02:16.652 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:16.650+0000 7f6f8359e700 1 -- 172.21.15.155:0/2983121261 shutdown_connections 2024-02-10T09:02:16.652 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:16.650+0000 7f6f8359e700 1 --2- 172.21.15.155:0/2983121261 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f6f6c02ff60 0x7f6f6c032420 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:02:16.652 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:16.650+0000 7f6f8359e700 1 --2- 172.21.15.155:0/2983121261 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f840f51a0 0x7f6f84109260 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:02:16.653 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:16.650+0000 7f6f8359e700 1 -- 172.21.15.155:0/2983121261 >> 172.21.15.155:0/2983121261 conn(0x7f6f840f09a0 msgr2=0x7f6f840f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:02:16.653 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:16.650+0000 7f6f8359e700 1 -- 172.21.15.155:0/2983121261 shutdown_connections 2024-02-10T09:02:16.653 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:16.650+0000 7f6f8359e700 1 -- 172.21.15.155:0/2983121261 wait complete. 2024-02-10T09:02:16.653 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:02:17.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:02:16 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:02:16.649686+0000 mon.smithi077 (mon.0) 166 : audit [DBG] from='client.? 172.21.15.155:0/2983121261' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:02:18.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:02:17 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:02:16.987723+0000 mgr.smithi077.dxntvw (mgr.14162) 106 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:02:18.329 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:02:18.329 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:02:19.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:02:19 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:02:18.988094+0000 mgr.smithi077.dxntvw (mgr.14162) 107 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:02:20.244 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:20.241+0000 7f91806cd700 1 -- 172.21.15.155:0/3656343404 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91780f5410 msgr2=0x7f91780f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:02:20.244 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:20.241+0000 7f91806cd700 1 --2- 172.21.15.155:0/3656343404 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91780f5410 0x7f91780f5830 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f9168009fb0 tx=0x7f9168009b30).stop 2024-02-10T09:02:20.244 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:20.241+0000 7f91806cd700 1 -- 172.21.15.155:0/3656343404 shutdown_connections 2024-02-10T09:02:20.245 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:20.241+0000 7f91806cd700 1 --2- 172.21.15.155:0/3656343404 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91780f5410 0x7f91780f5830 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:02:20.245 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:20.241+0000 7f91806cd700 1 -- 172.21.15.155:0/3656343404 >> 172.21.15.155:0/3656343404 conn(0x7f91780f0b70 msgr2=0x7f91780f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:02:20.245 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:20.241+0000 7f91806cd700 1 -- 172.21.15.155:0/3656343404 shutdown_connections 2024-02-10T09:02:20.245 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:20.241+0000 7f91806cd700 1 -- 172.21.15.155:0/3656343404 wait complete. 2024-02-10T09:02:20.245 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:20.242+0000 7f91806cd700 1 Processor -- start 2024-02-10T09:02:20.245 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:20.242+0000 7f91806cd700 1 -- start start 2024-02-10T09:02:20.245 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:20.242+0000 7f91806cd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91780f5410 0x7f91781056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:02:20.245 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:20.242+0000 7f91806cd700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9178105bd0 con 0x7f91780f5410 2024-02-10T09:02:20.245 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:20.242+0000 7f917e469700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91780f5410 0x7f91781056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:02:20.245 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:20.243+0000 7f917e469700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91780f5410 0x7f91781056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:36754/0 (socket says 172.21.15.155:36754) 2024-02-10T09:02:20.245 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:20.243+0000 7f917e469700 1 -- 172.21.15.155:0/1745223999 learned_addr learned my addr 172.21.15.155:0/1745223999 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:02:20.245 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:20.243+0000 7f917e469700 1 -- 172.21.15.155:0/1745223999 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9168005040 con 0x7f91780f5410 2024-02-10T09:02:20.245 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:20.243+0000 7f917e469700 1 --2- 172.21.15.155:0/1745223999 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91780f5410 0x7f91781056e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f9168005010 tx=0x7f916800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:02:20.246 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:20.243+0000 7f916effd700 1 -- 172.21.15.155:0/1745223999 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9168009320 con 0x7f91780f5410 2024-02-10T09:02:20.246 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:20.243+0000 7f916effd700 1 -- 172.21.15.155:0/1745223999 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9168013380 con 0x7f91780f5410 2024-02-10T09:02:20.246 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:20.243+0000 7f91806cd700 1 -- 172.21.15.155:0/1745223999 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9178105dd0 con 0x7f91780f5410 2024-02-10T09:02:20.246 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:20.243+0000 7f916effd700 1 -- 172.21.15.155:0/1745223999 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9168014e50 con 0x7f91780f5410 2024-02-10T09:02:20.246 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:20.243+0000 7f91806cd700 1 -- 172.21.15.155:0/1745223999 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9178106260 con 0x7f91780f5410 2024-02-10T09:02:20.247 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:20.244+0000 7f916effd700 1 -- 172.21.15.155:0/1745223999 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f91680095f0 con 0x7f91780f5410 2024-02-10T09:02:20.247 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:20.244+0000 7f91806cd700 1 -- 172.21.15.155:0/1745223999 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9178045ad0 con 0x7f91780f5410 2024-02-10T09:02:20.247 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:20.244+0000 7f916effd700 1 --2- 172.21.15.155:0/1745223999 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f916402ff10 0x7f91640323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:02:20.247 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:20.244+0000 7f916effd700 1 -- 172.21.15.155:0/1745223999 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9168040fa0 con 0x7f91780f5410 2024-02-10T09:02:20.247 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:20.244+0000 7f917dc68700 1 --2- 172.21.15.155:0/1745223999 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f916402ff10 0x7f91640323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:02:20.247 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:20.245+0000 7f917dc68700 1 --2- 172.21.15.155:0/1745223999 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f916402ff10 0x7f91640323d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f91740096a0 tx=0x7f9174006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:02:20.250 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:20.249+0000 7f916effd700 1 -- 172.21.15.155:0/1745223999 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9168014420 con 0x7f91780f5410 2024-02-10T09:02:20.456 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:20.455+0000 7f91806cd700 1 -- 172.21.15.155:0/1745223999 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9178107110 con 0x7f91780f5410 2024-02-10T09:02:20.457 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:20.456+0000 7f916effd700 1 -- 172.21.15.155:0/1745223999 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9168010230 con 0x7f91780f5410 2024-02-10T09:02:20.457 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:02:20.457 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:02:20.459 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:20.458+0000 7f91806cd700 1 -- 172.21.15.155:0/1745223999 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f916402ff10 msgr2=0x7f91640323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:02:20.459 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:20.458+0000 7f91806cd700 1 --2- 172.21.15.155:0/1745223999 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f916402ff10 0x7f91640323d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f91740096a0 tx=0x7f9174006b40).stop 2024-02-10T09:02:20.459 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:20.458+0000 7f91806cd700 1 -- 172.21.15.155:0/1745223999 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91780f5410 msgr2=0x7f91781056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:02:20.459 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:20.458+0000 7f91806cd700 1 --2- 172.21.15.155:0/1745223999 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91780f5410 0x7f91781056e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f9168005010 tx=0x7f916800b040).stop 2024-02-10T09:02:20.459 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:20.458+0000 7f91806cd700 1 -- 172.21.15.155:0/1745223999 shutdown_connections 2024-02-10T09:02:20.459 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:20.458+0000 7f91806cd700 1 --2- 172.21.15.155:0/1745223999 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f916402ff10 0x7f91640323d0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:02:20.459 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:20.458+0000 7f91806cd700 1 --2- 172.21.15.155:0/1745223999 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f91780f5410 0x7f91781056e0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:02:20.459 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:20.458+0000 7f91806cd700 1 -- 172.21.15.155:0/1745223999 >> 172.21.15.155:0/1745223999 conn(0x7f91780f0b70 msgr2=0x7f91780f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:02:20.459 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:20.458+0000 7f91806cd700 1 -- 172.21.15.155:0/1745223999 shutdown_connections 2024-02-10T09:02:20.460 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:20.458+0000 7f91806cd700 1 -- 172.21.15.155:0/1745223999 wait complete. 2024-02-10T09:02:20.460 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:02:20.899 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:02:20 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:02:20.457344+0000 mon.smithi077 (mon.0) 167 : audit [DBG] from='client.? 172.21.15.155:0/1745223999' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:02:21.899 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:02:21 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:02:20.988471+0000 mgr.smithi077.dxntvw (mgr.14162) 108 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:02:22.253 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:02:22.253 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:02:24.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:02:23 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:02:22.988820+0000 mgr.smithi077.dxntvw (mgr.14162) 109 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:02:24.298 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:24.296+0000 7f9ba8bd9700 1 -- 172.21.15.155:0/3897793416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ba40f5410 msgr2=0x7f9ba40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:02:24.298 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:24.296+0000 7f9ba8bd9700 1 --2- 172.21.15.155:0/3897793416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ba40f5410 0x7f9ba40f5830 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f9b94009fb0 tx=0x7f9b94009b30).stop 2024-02-10T09:02:24.298 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:24.296+0000 7f9ba8bd9700 1 -- 172.21.15.155:0/3897793416 shutdown_connections 2024-02-10T09:02:24.298 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:24.296+0000 7f9ba8bd9700 1 --2- 172.21.15.155:0/3897793416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ba40f5410 0x7f9ba40f5830 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:02:24.298 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:24.296+0000 7f9ba8bd9700 1 -- 172.21.15.155:0/3897793416 >> 172.21.15.155:0/3897793416 conn(0x7f9ba40f0b70 msgr2=0x7f9ba40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:02:24.298 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:24.296+0000 7f9ba8bd9700 1 -- 172.21.15.155:0/3897793416 shutdown_connections 2024-02-10T09:02:24.298 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:24.296+0000 7f9ba8bd9700 1 -- 172.21.15.155:0/3897793416 wait complete. 2024-02-10T09:02:24.298 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:24.297+0000 7f9ba8bd9700 1 Processor -- start 2024-02-10T09:02:24.299 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:24.297+0000 7f9ba8bd9700 1 -- start start 2024-02-10T09:02:24.299 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:24.297+0000 7f9ba8bd9700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ba40f5410 0x7f9ba41067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:02:24.299 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:24.297+0000 7f9ba8bd9700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba4106cd0 con 0x7f9ba40f5410 2024-02-10T09:02:24.299 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:24.297+0000 7f9ba259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ba40f5410 0x7f9ba41067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:02:24.299 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:24.297+0000 7f9ba259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ba40f5410 0x7f9ba41067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:38214/0 (socket says 172.21.15.155:38214) 2024-02-10T09:02:24.299 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:24.297+0000 7f9ba259c700 1 -- 172.21.15.155:0/934474375 learned_addr learned my addr 172.21.15.155:0/934474375 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:02:24.299 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:24.297+0000 7f9ba259c700 1 -- 172.21.15.155:0/934474375 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b94005040 con 0x7f9ba40f5410 2024-02-10T09:02:24.299 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:24.297+0000 7f9ba259c700 1 --2- 172.21.15.155:0/934474375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ba40f5410 0x7f9ba41067e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f9b940089f0 tx=0x7f9b9400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:02:24.299 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:24.298+0000 7f9b92ffd700 1 -- 172.21.15.155:0/934474375 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b94009320 con 0x7f9ba40f5410 2024-02-10T09:02:24.299 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:24.298+0000 7f9ba8bd9700 1 -- 172.21.15.155:0/934474375 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ba4106ed0 con 0x7f9ba40f5410 2024-02-10T09:02:24.299 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:24.298+0000 7f9b92ffd700 1 -- 172.21.15.155:0/934474375 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9b94013380 con 0x7f9ba40f5410 2024-02-10T09:02:24.299 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:24.298+0000 7f9b92ffd700 1 -- 172.21.15.155:0/934474375 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b9401ed80 con 0x7f9ba40f5410 2024-02-10T09:02:24.300 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:24.298+0000 7f9ba8bd9700 1 -- 172.21.15.155:0/934474375 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ba4107360 con 0x7f9ba40f5410 2024-02-10T09:02:24.300 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:24.298+0000 7f9b92ffd700 1 -- 172.21.15.155:0/934474375 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9b9401e420 con 0x7f9ba40f5410 2024-02-10T09:02:24.301 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:24.298+0000 7f9ba8bd9700 1 -- 172.21.15.155:0/934474375 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ba4045ad0 con 0x7f9ba40f5410 2024-02-10T09:02:24.301 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:24.299+0000 7f9b92ffd700 1 --2- 172.21.15.155:0/934474375 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f9b8c02ff60 0x7f9b8c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:02:24.301 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:24.299+0000 7f9b92ffd700 1 -- 172.21.15.155:0/934474375 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9b94041150 con 0x7f9ba40f5410 2024-02-10T09:02:24.301 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:24.299+0000 7f9ba1d9b700 1 --2- 172.21.15.155:0/934474375 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f9b8c02ff60 0x7f9b8c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:02:24.301 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:24.299+0000 7f9ba1d9b700 1 --2- 172.21.15.155:0/934474375 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f9b8c02ff60 0x7f9b8c032420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f9b980096a0 tx=0x7f9b98006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:02:24.303 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:24.303+0000 7f9b92ffd700 1 -- 172.21.15.155:0/934474375 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9b940095f0 con 0x7f9ba40f5410 2024-02-10T09:02:24.511 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:24.511+0000 7f9ba8bd9700 1 -- 172.21.15.155:0/934474375 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9ba4108210 con 0x7f9ba40f5410 2024-02-10T09:02:24.511 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:24.511+0000 7f9b92ffd700 1 -- 172.21.15.155:0/934474375 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9b94004490 con 0x7f9ba40f5410 2024-02-10T09:02:24.511 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:02:24.512 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:02:24.513 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:24.513+0000 7f9ba8bd9700 1 -- 172.21.15.155:0/934474375 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f9b8c02ff60 msgr2=0x7f9b8c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:02:24.513 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:24.513+0000 7f9ba8bd9700 1 --2- 172.21.15.155:0/934474375 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f9b8c02ff60 0x7f9b8c032420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f9b980096a0 tx=0x7f9b98006b40).stop 2024-02-10T09:02:24.513 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:24.513+0000 7f9ba8bd9700 1 -- 172.21.15.155:0/934474375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ba40f5410 msgr2=0x7f9ba41067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:02:24.513 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:24.513+0000 7f9ba8bd9700 1 --2- 172.21.15.155:0/934474375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ba40f5410 0x7f9ba41067e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f9b940089f0 tx=0x7f9b9400b040).stop 2024-02-10T09:02:24.513 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:24.513+0000 7f9ba8bd9700 1 -- 172.21.15.155:0/934474375 shutdown_connections 2024-02-10T09:02:24.513 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:24.513+0000 7f9ba8bd9700 1 --2- 172.21.15.155:0/934474375 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f9b8c02ff60 0x7f9b8c032420 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:02:24.513 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:24.513+0000 7f9ba8bd9700 1 --2- 172.21.15.155:0/934474375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9ba40f5410 0x7f9ba41067e0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:02:24.514 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:24.513+0000 7f9ba8bd9700 1 -- 172.21.15.155:0/934474375 >> 172.21.15.155:0/934474375 conn(0x7f9ba40f0b70 msgr2=0x7f9ba40f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:02:24.514 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:24.513+0000 7f9ba8bd9700 1 -- 172.21.15.155:0/934474375 shutdown_connections 2024-02-10T09:02:24.514 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:24.513+0000 7f9ba8bd9700 1 -- 172.21.15.155:0/934474375 wait complete. 2024-02-10T09:02:24.514 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:02:25.148 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:02:24 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:02:24.511766+0000 mon.smithi077 (mon.0) 168 : audit [DBG] from='client.? 172.21.15.155:0/934474375' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:02:26.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:02:25 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:02:24.989129+0000 mgr.smithi077.dxntvw (mgr.14162) 110 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:02:26.268 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:02:26.268 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:02:27.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:02:27 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:02:26.989382+0000 mgr.smithi077.dxntvw (mgr.14162) 111 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:02:28.175 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:28.174+0000 7fb9f012c700 1 -- 172.21.15.155:0/3883603115 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9e80f5410 msgr2=0x7fb9e80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:02:28.175 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:28.174+0000 7fb9f012c700 1 --2- 172.21.15.155:0/3883603115 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9e80f5410 0x7fb9e80f5830 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fb9d8009fb0 tx=0x7fb9d8009b30).stop 2024-02-10T09:02:28.175 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:28.174+0000 7fb9f012c700 1 -- 172.21.15.155:0/3883603115 shutdown_connections 2024-02-10T09:02:28.176 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:28.174+0000 7fb9f012c700 1 --2- 172.21.15.155:0/3883603115 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9e80f5410 0x7fb9e80f5830 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:02:28.176 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:28.174+0000 7fb9f012c700 1 -- 172.21.15.155:0/3883603115 >> 172.21.15.155:0/3883603115 conn(0x7fb9e80f0b70 msgr2=0x7fb9e80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:02:28.176 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:28.174+0000 7fb9f012c700 1 -- 172.21.15.155:0/3883603115 shutdown_connections 2024-02-10T09:02:28.176 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:28.174+0000 7fb9f012c700 1 -- 172.21.15.155:0/3883603115 wait complete. 2024-02-10T09:02:28.176 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:28.174+0000 7fb9f012c700 1 Processor -- start 2024-02-10T09:02:28.176 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:28.174+0000 7fb9f012c700 1 -- start start 2024-02-10T09:02:28.176 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:28.175+0000 7fb9f012c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9e80fd520 0x7fb9e80fb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:02:28.176 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:28.175+0000 7fb9f012c700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e80fd940 con 0x7fb9e80fd520 2024-02-10T09:02:28.176 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:28.175+0000 7fb9edec8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9e80fd520 0x7fb9e80fb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:02:28.176 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:28.175+0000 7fb9edec8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9e80fd520 0x7fb9e80fb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:38230/0 (socket says 172.21.15.155:38230) 2024-02-10T09:02:28.176 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:28.175+0000 7fb9edec8700 1 -- 172.21.15.155:0/3646858664 learned_addr learned my addr 172.21.15.155:0/3646858664 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:02:28.176 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:28.175+0000 7fb9edec8700 1 -- 172.21.15.155:0/3646858664 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9d8005040 con 0x7fb9e80fd520 2024-02-10T09:02:28.176 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:28.175+0000 7fb9edec8700 1 --2- 172.21.15.155:0/3646858664 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9e80fd520 0x7fb9e80fb9e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fb9e80f6510 tx=0x7fb9d8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:02:28.177 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:28.175+0000 7fb9de7fc700 1 -- 172.21.15.155:0/3646858664 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb9d8013380 con 0x7fb9e80fd520 2024-02-10T09:02:28.178 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:28.175+0000 7fb9f012c700 1 -- 172.21.15.155:0/3646858664 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9e80fbed0 con 0x7fb9e80fd520 2024-02-10T09:02:28.178 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:28.175+0000 7fb9de7fc700 1 -- 172.21.15.155:0/3646858664 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb9d8013790 con 0x7fb9e80fd520 2024-02-10T09:02:28.178 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:28.175+0000 7fb9de7fc700 1 -- 172.21.15.155:0/3646858664 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb9d801ee70 con 0x7fb9e80fd520 2024-02-10T09:02:28.178 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:28.176+0000 7fb9f012c700 1 -- 172.21.15.155:0/3646858664 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9e80fc3c0 con 0x7fb9e80fd520 2024-02-10T09:02:28.178 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:28.176+0000 7fb9de7fc700 1 -- 172.21.15.155:0/3646858664 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb9d8025060 con 0x7fb9e80fd520 2024-02-10T09:02:28.178 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:28.176+0000 7fb9f012c700 1 -- 172.21.15.155:0/3646858664 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9e8059100 con 0x7fb9e80fd520 2024-02-10T09:02:28.178 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:28.176+0000 7fb9de7fc700 1 --2- 172.21.15.155:0/3646858664 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb9d402ff60 0x7fb9d4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:02:28.178 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:28.176+0000 7fb9de7fc700 1 -- 172.21.15.155:0/3646858664 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb9d8040e70 con 0x7fb9e80fd520 2024-02-10T09:02:28.178 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:28.177+0000 7fb9ed6c7700 1 --2- 172.21.15.155:0/3646858664 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb9d402ff60 0x7fb9d4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:02:28.178 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:28.177+0000 7fb9ed6c7700 1 --2- 172.21.15.155:0/3646858664 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb9d402ff60 0x7fb9d4032420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fb9e40096a0 tx=0x7fb9e4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:02:28.181 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:28.181+0000 7fb9de7fc700 1 -- 172.21.15.155:0/3646858664 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb9d801c950 con 0x7fb9e80fd520 2024-02-10T09:02:28.387 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:28.386+0000 7fb9f012c700 1 -- 172.21.15.155:0/3646858664 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb9e80fc7b0 con 0x7fb9e80fd520 2024-02-10T09:02:28.387 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:28.387+0000 7fb9de7fc700 1 -- 172.21.15.155:0/3646858664 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb9d801cb60 con 0x7fb9e80fd520 2024-02-10T09:02:28.387 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:02:28.387 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:02:28.389 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:28.389+0000 7fb9f012c700 1 -- 172.21.15.155:0/3646858664 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb9d402ff60 msgr2=0x7fb9d4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:02:28.389 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:28.389+0000 7fb9f012c700 1 --2- 172.21.15.155:0/3646858664 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb9d402ff60 0x7fb9d4032420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fb9e40096a0 tx=0x7fb9e4006b40).stop 2024-02-10T09:02:28.389 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:28.389+0000 7fb9f012c700 1 -- 172.21.15.155:0/3646858664 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9e80fd520 msgr2=0x7fb9e80fb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:02:28.389 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:28.389+0000 7fb9f012c700 1 --2- 172.21.15.155:0/3646858664 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9e80fd520 0x7fb9e80fb9e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fb9e80f6510 tx=0x7fb9d8009b30).stop 2024-02-10T09:02:28.389 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:28.389+0000 7fb9f012c700 1 -- 172.21.15.155:0/3646858664 shutdown_connections 2024-02-10T09:02:28.389 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:28.389+0000 7fb9f012c700 1 --2- 172.21.15.155:0/3646858664 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb9d402ff60 0x7fb9d4032420 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:02:28.389 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:28.389+0000 7fb9f012c700 1 --2- 172.21.15.155:0/3646858664 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9e80fd520 0x7fb9e80fb9e0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:02:28.389 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:28.389+0000 7fb9f012c700 1 -- 172.21.15.155:0/3646858664 >> 172.21.15.155:0/3646858664 conn(0x7fb9e80f0b70 msgr2=0x7fb9e805fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:02:28.389 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:28.389+0000 7fb9f012c700 1 -- 172.21.15.155:0/3646858664 shutdown_connections 2024-02-10T09:02:28.389 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:28.389+0000 7fb9f012c700 1 -- 172.21.15.155:0/3646858664 wait complete. 2024-02-10T09:02:28.390 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:02:29.148 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:02:28 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:02:28.387538+0000 mon.smithi077 (mon.0) 169 : audit [DBG] from='client.? 172.21.15.155:0/3646858664' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:02:30.120 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:02:30.120 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:02:30.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:02:29 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:02:28.989632+0000 mgr.smithi077.dxntvw (mgr.14162) 112 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:02:31.398 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:02:31 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:02:30.989874+0000 mgr.smithi077.dxntvw (mgr.14162) 113 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:02:32.151 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:32.143+0000 7f3b55166700 1 -- 172.21.15.155:0/624838259 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b500f6510 msgr2=0x7f3b500f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:02:32.151 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:32.143+0000 7f3b55166700 1 --2- 172.21.15.155:0/624838259 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b500f6510 0x7f3b500f6930 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f3b44009fb0 tx=0x7f3b44009b30).stop 2024-02-10T09:02:32.151 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:32.143+0000 7f3b55166700 1 -- 172.21.15.155:0/624838259 shutdown_connections 2024-02-10T09:02:32.151 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:32.143+0000 7f3b55166700 1 --2- 172.21.15.155:0/624838259 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b500f6510 0x7f3b500f6930 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:02:32.151 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:32.143+0000 7f3b55166700 1 -- 172.21.15.155:0/624838259 >> 172.21.15.155:0/624838259 conn(0x7f3b500f1c70 msgr2=0x7f3b500f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:02:32.151 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:32.143+0000 7f3b55166700 1 -- 172.21.15.155:0/624838259 shutdown_connections 2024-02-10T09:02:32.152 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:32.143+0000 7f3b55166700 1 -- 172.21.15.155:0/624838259 wait complete. 2024-02-10T09:02:32.152 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:32.144+0000 7f3b55166700 1 Processor -- start 2024-02-10T09:02:32.152 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:32.144+0000 7f3b55166700 1 -- start start 2024-02-10T09:02:32.152 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:32.144+0000 7f3b55166700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b500f6510 0x7f3b501067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:02:32.152 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:32.144+0000 7f3b55166700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b50106cd0 con 0x7f3b500f6510 2024-02-10T09:02:32.152 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:32.144+0000 7f3b4ed9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b500f6510 0x7f3b501067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:02:32.152 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:32.144+0000 7f3b4ed9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b500f6510 0x7f3b501067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:38256/0 (socket says 172.21.15.155:38256) 2024-02-10T09:02:32.152 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:32.144+0000 7f3b4ed9d700 1 -- 172.21.15.155:0/2600668671 learned_addr learned my addr 172.21.15.155:0/2600668671 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:02:32.152 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:32.145+0000 7f3b4ed9d700 1 -- 172.21.15.155:0/2600668671 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b44005040 con 0x7f3b500f6510 2024-02-10T09:02:32.152 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:32.145+0000 7f3b4ed9d700 1 --2- 172.21.15.155:0/2600668671 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b500f6510 0x7f3b501067e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f3b44005010 tx=0x7f3b4400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:02:32.152 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:32.145+0000 7f3b377fe700 1 -- 172.21.15.155:0/2600668671 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b44009320 con 0x7f3b500f6510 2024-02-10T09:02:32.152 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:32.145+0000 7f3b55166700 1 -- 172.21.15.155:0/2600668671 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b50106ed0 con 0x7f3b500f6510 2024-02-10T09:02:32.152 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:32.145+0000 7f3b377fe700 1 -- 172.21.15.155:0/2600668671 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3b44013380 con 0x7f3b500f6510 2024-02-10T09:02:32.153 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:32.145+0000 7f3b377fe700 1 -- 172.21.15.155:0/2600668671 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b44014e50 con 0x7f3b500f6510 2024-02-10T09:02:32.153 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:32.145+0000 7f3b55166700 1 -- 172.21.15.155:0/2600668671 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b50107360 con 0x7f3b500f6510 2024-02-10T09:02:32.153 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:32.146+0000 7f3b377fe700 1 -- 172.21.15.155:0/2600668671 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3b44014420 con 0x7f3b500f6510 2024-02-10T09:02:32.153 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:32.146+0000 7f3b55166700 1 -- 172.21.15.155:0/2600668671 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b50045ad0 con 0x7f3b500f6510 2024-02-10T09:02:32.153 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:32.146+0000 7f3b377fe700 1 --2- 172.21.15.155:0/2600668671 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f3b3802ff10 0x7f3b380323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:02:32.153 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:32.146+0000 7f3b377fe700 1 -- 172.21.15.155:0/2600668671 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3b44040f70 con 0x7f3b500f6510 2024-02-10T09:02:32.153 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:32.146+0000 7f3b4e59c700 1 --2- 172.21.15.155:0/2600668671 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f3b3802ff10 0x7f3b380323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:02:32.153 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:32.147+0000 7f3b4e59c700 1 --2- 172.21.15.155:0/2600668671 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f3b3802ff10 0x7f3b380323d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f3b400096a0 tx=0x7f3b40006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:02:32.153 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:32.151+0000 7f3b377fe700 1 -- 172.21.15.155:0/2600668671 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3b440095f0 con 0x7f3b500f6510 2024-02-10T09:02:32.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:32.357+0000 7f3b55166700 1 -- 172.21.15.155:0/2600668671 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3b501081b0 con 0x7f3b500f6510 2024-02-10T09:02:32.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:32.358+0000 7f3b377fe700 1 -- 172.21.15.155:0/2600668671 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3b4401db70 con 0x7f3b500f6510 2024-02-10T09:02:32.358 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:02:32.359 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:02:32.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:32.360+0000 7f3b55166700 1 -- 172.21.15.155:0/2600668671 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f3b3802ff10 msgr2=0x7f3b380323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:02:32.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:32.360+0000 7f3b55166700 1 --2- 172.21.15.155:0/2600668671 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f3b3802ff10 0x7f3b380323d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f3b400096a0 tx=0x7f3b40006b40).stop 2024-02-10T09:02:32.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:32.360+0000 7f3b55166700 1 -- 172.21.15.155:0/2600668671 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b500f6510 msgr2=0x7f3b501067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:02:32.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:32.360+0000 7f3b55166700 1 --2- 172.21.15.155:0/2600668671 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b500f6510 0x7f3b501067e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f3b44005010 tx=0x7f3b4400b040).stop 2024-02-10T09:02:32.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:32.360+0000 7f3b55166700 1 -- 172.21.15.155:0/2600668671 shutdown_connections 2024-02-10T09:02:32.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:32.360+0000 7f3b55166700 1 --2- 172.21.15.155:0/2600668671 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f3b3802ff10 0x7f3b380323d0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:02:32.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:32.360+0000 7f3b55166700 1 --2- 172.21.15.155:0/2600668671 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b500f6510 0x7f3b501067e0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:02:32.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:32.360+0000 7f3b55166700 1 -- 172.21.15.155:0/2600668671 >> 172.21.15.155:0/2600668671 conn(0x7f3b500f1c70 msgr2=0x7f3b500f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:02:32.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:32.360+0000 7f3b55166700 1 -- 172.21.15.155:0/2600668671 shutdown_connections 2024-02-10T09:02:32.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:32.360+0000 7f3b55166700 1 -- 172.21.15.155:0/2600668671 wait complete. 2024-02-10T09:02:32.361 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:02:32.899 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:02:32 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:02:32.358522+0000 mon.smithi077 (mon.0) 170 : audit [DBG] from='client.? 172.21.15.155:0/2600668671' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:02:34.025 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:02:34.025 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:02:34.148 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:02:33 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:02:32.990173+0000 mgr.smithi077.dxntvw (mgr.14162) 114 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:02:35.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:02:35 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:02:34.990479+0000 mgr.smithi077.dxntvw (mgr.14162) 115 : cluster [DBG] pgmap v108: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:02:35.913 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:35.908+0000 7f5928574700 1 -- 172.21.15.155:0/3581137634 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59200f5410 msgr2=0x7f59200f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:02:35.913 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:35.908+0000 7f5928574700 1 --2- 172.21.15.155:0/3581137634 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59200f5410 0x7f59200f5830 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f5910009fb0 tx=0x7f5910009b30).stop 2024-02-10T09:02:35.913 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:35.908+0000 7f5928574700 1 -- 172.21.15.155:0/3581137634 shutdown_connections 2024-02-10T09:02:35.913 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:35.908+0000 7f5928574700 1 --2- 172.21.15.155:0/3581137634 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59200f5410 0x7f59200f5830 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:02:35.913 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:35.908+0000 7f5928574700 1 -- 172.21.15.155:0/3581137634 >> 172.21.15.155:0/3581137634 conn(0x7f59200f0b70 msgr2=0x7f59200f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:02:35.914 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:35.908+0000 7f5928574700 1 -- 172.21.15.155:0/3581137634 shutdown_connections 2024-02-10T09:02:35.914 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:35.908+0000 7f5928574700 1 -- 172.21.15.155:0/3581137634 wait complete. 2024-02-10T09:02:35.914 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:35.909+0000 7f5928574700 1 Processor -- start 2024-02-10T09:02:35.915 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:35.909+0000 7f5928574700 1 -- start start 2024-02-10T09:02:35.915 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:35.909+0000 7f5928574700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59200f5410 0x7f59201094a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:02:35.915 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:35.909+0000 7f5928574700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5920060280 con 0x7f59200f5410 2024-02-10T09:02:35.915 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:35.909+0000 7f5926310700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59200f5410 0x7f59201094a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:02:35.915 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:35.909+0000 7f5926310700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59200f5410 0x7f59201094a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:35434/0 (socket says 172.21.15.155:35434) 2024-02-10T09:02:35.915 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:35.909+0000 7f5926310700 1 -- 172.21.15.155:0/1018104234 learned_addr learned my addr 172.21.15.155:0/1018104234 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:02:35.916 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:35.910+0000 7f5926310700 1 -- 172.21.15.155:0/1018104234 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5910005040 con 0x7f59200f5410 2024-02-10T09:02:35.916 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:35.910+0000 7f5926310700 1 --2- 172.21.15.155:0/1018104234 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59200f5410 0x7f59201094a0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f5910005010 tx=0x7f591000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:02:35.916 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:35.910+0000 7f5916ffd700 1 -- 172.21.15.155:0/1018104234 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5910009320 con 0x7f59200f5410 2024-02-10T09:02:35.917 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:35.910+0000 7f5928574700 1 -- 172.21.15.155:0/1018104234 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5920060480 con 0x7f59200f5410 2024-02-10T09:02:35.917 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:35.910+0000 7f5916ffd700 1 -- 172.21.15.155:0/1018104234 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5910013380 con 0x7f59200f5410 2024-02-10T09:02:35.917 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:35.910+0000 7f5916ffd700 1 -- 172.21.15.155:0/1018104234 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5910014e50 con 0x7f59200f5410 2024-02-10T09:02:35.917 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:35.910+0000 7f5928574700 1 -- 172.21.15.155:0/1018104234 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5920109bd0 con 0x7f59200f5410 2024-02-10T09:02:35.917 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:35.911+0000 7f5916ffd700 1 -- 172.21.15.155:0/1018104234 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5910014420 con 0x7f59200f5410 2024-02-10T09:02:35.917 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:35.911+0000 7f5928574700 1 -- 172.21.15.155:0/1018104234 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5920045ad0 con 0x7f59200f5410 2024-02-10T09:02:35.918 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:35.911+0000 7f5916ffd700 1 --2- 172.21.15.155:0/1018104234 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f590c02ff10 0x7f590c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:02:35.918 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:35.911+0000 7f5916ffd700 1 -- 172.21.15.155:0/1018104234 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5910040f40 con 0x7f59200f5410 2024-02-10T09:02:35.918 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:35.911+0000 7f5925b0f700 1 --2- 172.21.15.155:0/1018104234 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f590c02ff10 0x7f590c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:02:35.918 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:35.911+0000 7f5925b0f700 1 --2- 172.21.15.155:0/1018104234 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f590c02ff10 0x7f590c0323d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f591c0096a0 tx=0x7f591c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:02:35.919 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:35.916+0000 7f5916ffd700 1 -- 172.21.15.155:0/1018104234 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f59100095f0 con 0x7f59200f5410 2024-02-10T09:02:36.120 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:36.120+0000 7f5928574700 1 -- 172.21.15.155:0/1018104234 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f59200fafa0 con 0x7f59200f5410 2024-02-10T09:02:36.121 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:36.121+0000 7f5916ffd700 1 -- 172.21.15.155:0/1018104234 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f591001db70 con 0x7f59200f5410 2024-02-10T09:02:36.121 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:02:36.121 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:02:36.122 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:36.122+0000 7f5928574700 1 -- 172.21.15.155:0/1018104234 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f590c02ff10 msgr2=0x7f590c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:02:36.123 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:36.122+0000 7f5928574700 1 --2- 172.21.15.155:0/1018104234 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f590c02ff10 0x7f590c0323d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f591c0096a0 tx=0x7f591c006b40).stop 2024-02-10T09:02:36.123 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:36.122+0000 7f5928574700 1 -- 172.21.15.155:0/1018104234 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59200f5410 msgr2=0x7f59201094a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:02:36.123 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:36.122+0000 7f5928574700 1 --2- 172.21.15.155:0/1018104234 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59200f5410 0x7f59201094a0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f5910005010 tx=0x7f591000b040).stop 2024-02-10T09:02:36.123 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:36.122+0000 7f5928574700 1 -- 172.21.15.155:0/1018104234 shutdown_connections 2024-02-10T09:02:36.123 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:36.122+0000 7f5928574700 1 --2- 172.21.15.155:0/1018104234 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f590c02ff10 0x7f590c0323d0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:02:36.123 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:36.122+0000 7f5928574700 1 --2- 172.21.15.155:0/1018104234 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59200f5410 0x7f59201094a0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:02:36.123 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:36.122+0000 7f5928574700 1 -- 172.21.15.155:0/1018104234 >> 172.21.15.155:0/1018104234 conn(0x7f59200f0b70 msgr2=0x7f59200f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:02:36.123 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:36.122+0000 7f5928574700 1 -- 172.21.15.155:0/1018104234 shutdown_connections 2024-02-10T09:02:36.123 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:36.122+0000 7f5928574700 1 -- 172.21.15.155:0/1018104234 wait complete. 2024-02-10T09:02:36.123 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:02:36.649 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:02:36 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:02:36.121131+0000 mon.smithi077 (mon.0) 171 : audit [DBG] from='client.? 172.21.15.155:0/1018104234' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:02:37.649 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:02:37 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:02:36.990758+0000 mgr.smithi077.dxntvw (mgr.14162) 116 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:02:37.752 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:02:37.752 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:02:39.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:02:39 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:02:38.991095+0000 mgr.smithi077.dxntvw (mgr.14162) 117 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:02:39.720 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:39.718+0000 7fe2724e8700 1 -- 172.21.15.155:0/1811016084 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe26c0f5410 msgr2=0x7fe26c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:02:39.720 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:39.718+0000 7fe2724e8700 1 --2- 172.21.15.155:0/1811016084 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe26c0f5410 0x7fe26c0f5830 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7fe25c009fb0 tx=0x7fe25c009b30).stop 2024-02-10T09:02:39.720 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:39.718+0000 7fe2724e8700 1 -- 172.21.15.155:0/1811016084 shutdown_connections 2024-02-10T09:02:39.720 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:39.718+0000 7fe2724e8700 1 --2- 172.21.15.155:0/1811016084 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe26c0f5410 0x7fe26c0f5830 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:02:39.720 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:39.718+0000 7fe2724e8700 1 -- 172.21.15.155:0/1811016084 >> 172.21.15.155:0/1811016084 conn(0x7fe26c0f0b70 msgr2=0x7fe26c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:02:39.720 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:39.718+0000 7fe2724e8700 1 -- 172.21.15.155:0/1811016084 shutdown_connections 2024-02-10T09:02:39.720 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:39.718+0000 7fe2724e8700 1 -- 172.21.15.155:0/1811016084 wait complete. 2024-02-10T09:02:39.720 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:39.719+0000 7fe2724e8700 1 Processor -- start 2024-02-10T09:02:39.720 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:39.719+0000 7fe2724e8700 1 -- start start 2024-02-10T09:02:39.722 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:39.719+0000 7fe2724e8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe26c1057e0 0x7fe26c105c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:02:39.722 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:39.719+0000 7fe2724e8700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe26c1060f0 con 0x7fe26c1057e0 2024-02-10T09:02:39.722 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:39.719+0000 7fe26bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe26c1057e0 0x7fe26c105c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:02:39.722 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:39.719+0000 7fe26bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe26c1057e0 0x7fe26c105c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:35450/0 (socket says 172.21.15.155:35450) 2024-02-10T09:02:39.722 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:39.719+0000 7fe26bfff700 1 -- 172.21.15.155:0/156192612 learned_addr learned my addr 172.21.15.155:0/156192612 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:02:39.723 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:39.720+0000 7fe26bfff700 1 -- 172.21.15.155:0/156192612 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe25c005040 con 0x7fe26c1057e0 2024-02-10T09:02:39.723 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:39.720+0000 7fe26bfff700 1 --2- 172.21.15.155:0/156192612 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe26c1057e0 0x7fe26c105c00 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fe25c009f80 tx=0x7fe25c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:02:39.723 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:39.720+0000 7fe268ff9700 1 -- 172.21.15.155:0/156192612 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe25c013380 con 0x7fe26c1057e0 2024-02-10T09:02:39.723 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:39.720+0000 7fe2724e8700 1 -- 172.21.15.155:0/156192612 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe26c1062f0 con 0x7fe26c1057e0 2024-02-10T09:02:39.723 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:39.720+0000 7fe268ff9700 1 -- 172.21.15.155:0/156192612 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe25c013790 con 0x7fe26c1057e0 2024-02-10T09:02:39.723 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:39.720+0000 7fe268ff9700 1 -- 172.21.15.155:0/156192612 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe25c014420 con 0x7fe26c1057e0 2024-02-10T09:02:39.723 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:39.720+0000 7fe2724e8700 1 -- 172.21.15.155:0/156192612 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe26c106e40 con 0x7fe26c1057e0 2024-02-10T09:02:39.723 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:39.721+0000 7fe2724e8700 1 -- 172.21.15.155:0/156192612 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe26c059100 con 0x7fe26c1057e0 2024-02-10T09:02:39.723 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:39.721+0000 7fe268ff9700 1 -- 172.21.15.155:0/156192612 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe25c025060 con 0x7fe26c1057e0 2024-02-10T09:02:39.723 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:39.721+0000 7fe268ff9700 1 --2- 172.21.15.155:0/156192612 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fe25402ffb0 0x7fe254032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:02:39.723 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:39.721+0000 7fe268ff9700 1 -- 172.21.15.155:0/156192612 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe25c040de0 con 0x7fe26c1057e0 2024-02-10T09:02:39.723 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:39.721+0000 7fe26b7fe700 1 --2- 172.21.15.155:0/156192612 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fe25402ffb0 0x7fe254032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:02:39.724 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:39.722+0000 7fe26b7fe700 1 --2- 172.21.15.155:0/156192612 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fe25402ffb0 0x7fe254032470 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fe2600096a0 tx=0x7fe260006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:02:39.726 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:39.726+0000 7fe268ff9700 1 -- 172.21.15.155:0/156192612 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe25c045070 con 0x7fe26c1057e0 2024-02-10T09:02:39.931 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:39.930+0000 7fe2724e8700 1 -- 172.21.15.155:0/156192612 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe26c045ad0 con 0x7fe26c1057e0 2024-02-10T09:02:39.932 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:39.931+0000 7fe268ff9700 1 -- 172.21.15.155:0/156192612 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe25c02d2d0 con 0x7fe26c1057e0 2024-02-10T09:02:39.932 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:02:39.932 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:02:39.934 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:39.933+0000 7fe2724e8700 1 -- 172.21.15.155:0/156192612 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fe25402ffb0 msgr2=0x7fe254032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:02:39.934 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:39.933+0000 7fe2724e8700 1 --2- 172.21.15.155:0/156192612 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fe25402ffb0 0x7fe254032470 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fe2600096a0 tx=0x7fe260006b40).stop 2024-02-10T09:02:39.934 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:39.933+0000 7fe2724e8700 1 -- 172.21.15.155:0/156192612 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe26c1057e0 msgr2=0x7fe26c105c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:02:39.934 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:39.933+0000 7fe2724e8700 1 --2- 172.21.15.155:0/156192612 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe26c1057e0 0x7fe26c105c00 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fe25c009f80 tx=0x7fe25c009b30).stop 2024-02-10T09:02:39.934 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:39.933+0000 7fe2724e8700 1 -- 172.21.15.155:0/156192612 shutdown_connections 2024-02-10T09:02:39.934 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:39.933+0000 7fe2724e8700 1 --2- 172.21.15.155:0/156192612 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fe25402ffb0 0x7fe254032470 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:02:39.934 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:39.933+0000 7fe2724e8700 1 --2- 172.21.15.155:0/156192612 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe26c1057e0 0x7fe26c105c00 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:02:39.934 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:39.933+0000 7fe2724e8700 1 -- 172.21.15.155:0/156192612 >> 172.21.15.155:0/156192612 conn(0x7fe26c0f0b70 msgr2=0x7fe26c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:02:39.934 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:39.933+0000 7fe2724e8700 1 -- 172.21.15.155:0/156192612 shutdown_connections 2024-02-10T09:02:39.934 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:39.933+0000 7fe2724e8700 1 -- 172.21.15.155:0/156192612 wait complete. 2024-02-10T09:02:39.935 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:02:40.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:02:40 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:02:39.932453+0000 mon.smithi077 (mon.0) 172 : audit [DBG] from='client.? 172.21.15.155:0/156192612' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:02:41.398 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:02:41 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:02:40.991432+0000 mgr.smithi077.dxntvw (mgr.14162) 118 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:02:41.565 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:02:41.565 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:02:43.544 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:43.541+0000 7f9470806700 1 -- 172.21.15.155:0/708468627 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f946c0f6510 msgr2=0x7f946c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:02:43.544 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:43.541+0000 7f9470806700 1 --2- 172.21.15.155:0/708468627 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f946c0f6510 0x7f946c0f6930 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f9464009fb0 tx=0x7f9464009b30).stop 2024-02-10T09:02:43.544 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:43.541+0000 7f9470806700 1 -- 172.21.15.155:0/708468627 shutdown_connections 2024-02-10T09:02:43.544 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:43.541+0000 7f9470806700 1 --2- 172.21.15.155:0/708468627 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f946c0f6510 0x7f946c0f6930 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:02:43.544 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:43.541+0000 7f9470806700 1 -- 172.21.15.155:0/708468627 >> 172.21.15.155:0/708468627 conn(0x7f946c0f1c70 msgr2=0x7f946c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:02:43.544 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:43.541+0000 7f9470806700 1 -- 172.21.15.155:0/708468627 shutdown_connections 2024-02-10T09:02:43.545 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:43.541+0000 7f9470806700 1 -- 172.21.15.155:0/708468627 wait complete. 2024-02-10T09:02:43.545 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:43.541+0000 7f9470806700 1 Processor -- start 2024-02-10T09:02:43.545 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:43.541+0000 7f9470806700 1 -- start start 2024-02-10T09:02:43.545 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:43.542+0000 7f9470806700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f946c0f6510 0x7f946c1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:02:43.545 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:43.542+0000 7f9470806700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f946c106cd0 con 0x7f946c0f6510 2024-02-10T09:02:43.545 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:43.542+0000 7f946a59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f946c0f6510 0x7f946c1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:02:43.545 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:43.542+0000 7f946a59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f946c0f6510 0x7f946c1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:52678/0 (socket says 172.21.15.155:52678) 2024-02-10T09:02:43.545 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:43.542+0000 7f946a59c700 1 -- 172.21.15.155:0/2987968392 learned_addr learned my addr 172.21.15.155:0/2987968392 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:02:43.545 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:43.542+0000 7f946a59c700 1 -- 172.21.15.155:0/2987968392 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9464005040 con 0x7f946c0f6510 2024-02-10T09:02:43.545 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:43.542+0000 7f946a59c700 1 --2- 172.21.15.155:0/2987968392 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f946c0f6510 0x7f946c1067e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f94640089f0 tx=0x7f946400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:02:43.545 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:43.542+0000 7f9462ffd700 1 -- 172.21.15.155:0/2987968392 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9464009320 con 0x7f946c0f6510 2024-02-10T09:02:43.545 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:43.543+0000 7f9470806700 1 -- 172.21.15.155:0/2987968392 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f946c106ed0 con 0x7f946c0f6510 2024-02-10T09:02:43.546 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:43.543+0000 7f9462ffd700 1 -- 172.21.15.155:0/2987968392 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9464013380 con 0x7f946c0f6510 2024-02-10T09:02:43.546 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:43.543+0000 7f9462ffd700 1 -- 172.21.15.155:0/2987968392 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f946401ed80 con 0x7f946c0f6510 2024-02-10T09:02:43.546 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:43.543+0000 7f9470806700 1 -- 172.21.15.155:0/2987968392 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f946c107360 con 0x7f946c0f6510 2024-02-10T09:02:43.547 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:43.543+0000 7f9462ffd700 1 -- 172.21.15.155:0/2987968392 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f946401e420 con 0x7f946c0f6510 2024-02-10T09:02:43.547 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:43.543+0000 7f9470806700 1 -- 172.21.15.155:0/2987968392 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f946c045ad0 con 0x7f946c0f6510 2024-02-10T09:02:43.547 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:43.544+0000 7f9462ffd700 1 --2- 172.21.15.155:0/2987968392 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f945802ff60 0x7f9458032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:02:43.547 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:43.544+0000 7f9462ffd700 1 -- 172.21.15.155:0/2987968392 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9464041000 con 0x7f946c0f6510 2024-02-10T09:02:43.547 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:43.544+0000 7f9469d9b700 1 --2- 172.21.15.155:0/2987968392 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f945802ff60 0x7f9458032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:02:43.547 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:43.544+0000 7f9469d9b700 1 --2- 172.21.15.155:0/2987968392 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f945802ff60 0x7f9458032420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f94540096a0 tx=0x7f9454006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:02:43.549 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:43.548+0000 7f9462ffd700 1 -- 172.21.15.155:0/2987968392 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f94640095f0 con 0x7f946c0f6510 2024-02-10T09:02:43.753 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:43.752+0000 7f9470806700 1 -- 172.21.15.155:0/2987968392 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f946c108210 con 0x7f946c0f6510 2024-02-10T09:02:43.754 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:43.753+0000 7f9462ffd700 1 -- 172.21.15.155:0/2987968392 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f946401cb10 con 0x7f946c0f6510 2024-02-10T09:02:43.755 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:02:43.755 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:02:43.756 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:43.755+0000 7f9470806700 1 -- 172.21.15.155:0/2987968392 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f945802ff60 msgr2=0x7f9458032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:02:43.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:43.755+0000 7f9470806700 1 --2- 172.21.15.155:0/2987968392 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f945802ff60 0x7f9458032420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f94540096a0 tx=0x7f9454006b40).stop 2024-02-10T09:02:43.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:43.755+0000 7f9470806700 1 -- 172.21.15.155:0/2987968392 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f946c0f6510 msgr2=0x7f946c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:02:43.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:43.755+0000 7f9470806700 1 --2- 172.21.15.155:0/2987968392 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f946c0f6510 0x7f946c1067e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f94640089f0 tx=0x7f946400b040).stop 2024-02-10T09:02:43.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:43.755+0000 7f9470806700 1 -- 172.21.15.155:0/2987968392 shutdown_connections 2024-02-10T09:02:43.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:43.755+0000 7f9470806700 1 --2- 172.21.15.155:0/2987968392 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f945802ff60 0x7f9458032420 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:02:43.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:43.755+0000 7f9470806700 1 --2- 172.21.15.155:0/2987968392 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f946c0f6510 0x7f946c1067e0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:02:43.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:43.755+0000 7f9470806700 1 -- 172.21.15.155:0/2987968392 >> 172.21.15.155:0/2987968392 conn(0x7f946c0f1c70 msgr2=0x7f946c0f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:02:43.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:43.755+0000 7f9470806700 1 -- 172.21.15.155:0/2987968392 shutdown_connections 2024-02-10T09:02:43.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:43.755+0000 7f9470806700 1 -- 172.21.15.155:0/2987968392 wait complete. 2024-02-10T09:02:43.757 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:02:44.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:02:43 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:02:42.991781+0000 mgr.smithi077.dxntvw (mgr.14162) 119 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:02:45.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:02:44 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:02:43.754588+0000 mon.smithi077 (mon.0) 173 : audit [DBG] from='client.? 172.21.15.155:0/2987968392' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:02:45.482 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:02:45.482 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:02:46.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:02:45 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:02:44.992124+0000 mgr.smithi077.dxntvw (mgr.14162) 120 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:02:47.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:02:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:02:46.992365+0000 mgr.smithi077.dxntvw (mgr.14162) 121 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:02:47.713 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:47.711+0000 7f8cb5017700 1 -- 172.21.15.155:0/281036699 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8cb00f5410 msgr2=0x7f8cb00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:02:47.713 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:47.711+0000 7f8cb5017700 1 --2- 172.21.15.155:0/281036699 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8cb00f5410 0x7f8cb00f5830 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f8c98009fb0 tx=0x7f8c98009b30).stop 2024-02-10T09:02:47.713 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:47.711+0000 7f8cb5017700 1 -- 172.21.15.155:0/281036699 shutdown_connections 2024-02-10T09:02:47.713 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:47.711+0000 7f8cb5017700 1 --2- 172.21.15.155:0/281036699 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8cb00f5410 0x7f8cb00f5830 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:02:47.713 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:47.711+0000 7f8cb5017700 1 -- 172.21.15.155:0/281036699 >> 172.21.15.155:0/281036699 conn(0x7f8cb00f0b70 msgr2=0x7f8cb00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:02:47.713 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:47.711+0000 7f8cb5017700 1 -- 172.21.15.155:0/281036699 shutdown_connections 2024-02-10T09:02:47.713 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:47.711+0000 7f8cb5017700 1 -- 172.21.15.155:0/281036699 wait complete. 2024-02-10T09:02:47.714 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:47.711+0000 7f8cb5017700 1 Processor -- start 2024-02-10T09:02:47.714 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:47.711+0000 7f8cb5017700 1 -- start start 2024-02-10T09:02:47.714 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:47.712+0000 7f8cb5017700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8cb00f5410 0x7f8cb00fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:02:47.714 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:47.712+0000 7f8cb5017700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb00fb960 con 0x7f8cb00f5410 2024-02-10T09:02:47.714 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:47.712+0000 7f8caed9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8cb00f5410 0x7f8cb00fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:02:47.714 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:47.712+0000 7f8caed9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8cb00f5410 0x7f8cb00fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:52694/0 (socket says 172.21.15.155:52694) 2024-02-10T09:02:47.714 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:47.712+0000 7f8caed9d700 1 -- 172.21.15.155:0/2019644705 learned_addr learned my addr 172.21.15.155:0/2019644705 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:02:47.714 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:47.712+0000 7f8caed9d700 1 -- 172.21.15.155:0/2019644705 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c98005040 con 0x7f8cb00f5410 2024-02-10T09:02:47.714 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:47.712+0000 7f8caed9d700 1 --2- 172.21.15.155:0/2019644705 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8cb00f5410 0x7f8cb00fd4e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f8c980089f0 tx=0x7f8c9800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:02:47.715 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:47.712+0000 7f8ca77fe700 1 -- 172.21.15.155:0/2019644705 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c98009320 con 0x7f8cb00f5410 2024-02-10T09:02:47.715 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:47.713+0000 7f8cb5017700 1 -- 172.21.15.155:0/2019644705 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cb00fbb00 con 0x7f8cb00f5410 2024-02-10T09:02:47.715 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:47.713+0000 7f8ca77fe700 1 -- 172.21.15.155:0/2019644705 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8c98013380 con 0x7f8cb00f5410 2024-02-10T09:02:47.715 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:47.713+0000 7f8cb5017700 1 -- 172.21.15.155:0/2019644705 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cb00fbf90 con 0x7f8cb00f5410 2024-02-10T09:02:47.715 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:47.713+0000 7f8ca77fe700 1 -- 172.21.15.155:0/2019644705 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c9801ed80 con 0x7f8cb00f5410 2024-02-10T09:02:47.715 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:47.713+0000 7f8cb5017700 1 -- 172.21.15.155:0/2019644705 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cb0059100 con 0x7f8cb00f5410 2024-02-10T09:02:47.715 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:47.713+0000 7f8ca77fe700 1 -- 172.21.15.155:0/2019644705 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8c9801e420 con 0x7f8cb00f5410 2024-02-10T09:02:47.715 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:47.713+0000 7f8ca77fe700 1 --2- 172.21.15.155:0/2019644705 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f8c9c02ffb0 0x7f8c9c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:02:47.716 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:47.714+0000 7f8ca77fe700 1 -- 172.21.15.155:0/2019644705 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8c980410d0 con 0x7f8cb00f5410 2024-02-10T09:02:47.716 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:47.714+0000 7f8cae59c700 1 --2- 172.21.15.155:0/2019644705 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f8c9c02ffb0 0x7f8c9c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:02:47.716 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:47.714+0000 7f8cae59c700 1 --2- 172.21.15.155:0/2019644705 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f8c9c02ffb0 0x7f8c9c032470 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f8ca00096a0 tx=0x7f8ca0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:02:47.719 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:47.718+0000 7f8ca77fe700 1 -- 172.21.15.155:0/2019644705 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8c980095f0 con 0x7f8cb00f5410 2024-02-10T09:02:47.925 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:47.924+0000 7f8cb5017700 1 -- 172.21.15.155:0/2019644705 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8cb0045ad0 con 0x7f8cb00f5410 2024-02-10T09:02:47.926 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:47.925+0000 7f8ca77fe700 1 -- 172.21.15.155:0/2019644705 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8c98004490 con 0x7f8cb00f5410 2024-02-10T09:02:47.926 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:02:47.926 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:02:47.928 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:47.926+0000 7f8cb5017700 1 -- 172.21.15.155:0/2019644705 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f8c9c02ffb0 msgr2=0x7f8c9c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:02:47.928 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:47.926+0000 7f8cb5017700 1 --2- 172.21.15.155:0/2019644705 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f8c9c02ffb0 0x7f8c9c032470 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f8ca00096a0 tx=0x7f8ca0006b40).stop 2024-02-10T09:02:47.928 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:47.926+0000 7f8cb5017700 1 -- 172.21.15.155:0/2019644705 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8cb00f5410 msgr2=0x7f8cb00fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:02:47.928 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:47.926+0000 7f8cb5017700 1 --2- 172.21.15.155:0/2019644705 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8cb00f5410 0x7f8cb00fd4e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f8c980089f0 tx=0x7f8c9800b040).stop 2024-02-10T09:02:47.928 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:47.926+0000 7f8cb5017700 1 -- 172.21.15.155:0/2019644705 shutdown_connections 2024-02-10T09:02:47.928 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:47.926+0000 7f8cb5017700 1 --2- 172.21.15.155:0/2019644705 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f8c9c02ffb0 0x7f8c9c032470 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:02:47.928 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:47.926+0000 7f8cb5017700 1 --2- 172.21.15.155:0/2019644705 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8cb00f5410 0x7f8cb00fd4e0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:02:47.928 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:47.926+0000 7f8cb5017700 1 -- 172.21.15.155:0/2019644705 >> 172.21.15.155:0/2019644705 conn(0x7f8cb00f0b70 msgr2=0x7f8cb005fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:02:47.928 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:47.926+0000 7f8cb5017700 1 -- 172.21.15.155:0/2019644705 shutdown_connections 2024-02-10T09:02:47.928 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:47.926+0000 7f8cb5017700 1 -- 172.21.15.155:0/2019644705 wait complete. 2024-02-10T09:02:47.928 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:02:48.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:02:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:02:47.091171+0000 mon.smithi077 (mon.0) 174 : audit [INF] from='mgr.14162 172.21.15.77:0/2029106299' entity='mgr.smithi077.dxntvw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi077.dxntvw/mirror_snapshot_schedule"}]: dispatch 2024-02-10T09:02:48.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:02:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:02:47.098532+0000 mon.smithi077 (mon.0) 175 : audit [INF] from='mgr.14162 172.21.15.77:0/2029106299' entity='mgr.smithi077.dxntvw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi077.dxntvw/trash_purge_schedule"}]: dispatch 2024-02-10T09:02:48.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:02:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:02:47.926134+0000 mon.smithi077 (mon.0) 176 : audit [DBG] from='client.? 172.21.15.155:0/2019644705' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:02:49.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:02:49 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:02:48.992672+0000 mgr.smithi077.dxntvw (mgr.14162) 122 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:02:49.658 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:02:49.659 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:02:51.398 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:02:51 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:02:50.993024+0000 mgr.smithi077.dxntvw (mgr.14162) 123 : cluster [DBG] pgmap v116: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:02:51.621 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:51.618+0000 7fc35206d700 1 -- 172.21.15.155:0/526709709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc34c0f6510 msgr2=0x7fc34c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:02:51.621 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:51.618+0000 7fc35206d700 1 --2- 172.21.15.155:0/526709709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc34c0f6510 0x7fc34c0f6930 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7fc33c009fb0 tx=0x7fc33c009b30).stop 2024-02-10T09:02:51.621 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:51.618+0000 7fc35206d700 1 -- 172.21.15.155:0/526709709 shutdown_connections 2024-02-10T09:02:51.621 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:51.618+0000 7fc35206d700 1 --2- 172.21.15.155:0/526709709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc34c0f6510 0x7fc34c0f6930 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:02:51.621 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:51.618+0000 7fc35206d700 1 -- 172.21.15.155:0/526709709 >> 172.21.15.155:0/526709709 conn(0x7fc34c0f1c70 msgr2=0x7fc34c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:02:51.621 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:51.618+0000 7fc35206d700 1 -- 172.21.15.155:0/526709709 shutdown_connections 2024-02-10T09:02:51.621 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:51.618+0000 7fc35206d700 1 -- 172.21.15.155:0/526709709 wait complete. 2024-02-10T09:02:51.621 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:51.619+0000 7fc35206d700 1 Processor -- start 2024-02-10T09:02:51.622 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:51.619+0000 7fc35206d700 1 -- start start 2024-02-10T09:02:51.622 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:51.619+0000 7fc35206d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc34c0f6510 0x7fc34c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:02:51.622 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:51.619+0000 7fc35206d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc34c069e20 con 0x7fc34c0f6510 2024-02-10T09:02:51.622 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:51.619+0000 7fc34b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc34c0f6510 0x7fc34c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:02:51.622 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:51.619+0000 7fc34b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc34c0f6510 0x7fc34c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:52704/0 (socket says 172.21.15.155:52704) 2024-02-10T09:02:51.622 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:51.619+0000 7fc34b7fe700 1 -- 172.21.15.155:0/1704506489 learned_addr learned my addr 172.21.15.155:0/1704506489 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:02:51.622 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:51.620+0000 7fc34b7fe700 1 -- 172.21.15.155:0/1704506489 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc33c005040 con 0x7fc34c0f6510 2024-02-10T09:02:51.622 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:51.620+0000 7fc34b7fe700 1 --2- 172.21.15.155:0/1704506489 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc34c0f6510 0x7fc34c06b9a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fc33c005010 tx=0x7fc33c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:02:51.622 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:51.620+0000 7fc333fff700 1 -- 172.21.15.155:0/1704506489 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc33c009320 con 0x7fc34c0f6510 2024-02-10T09:02:51.622 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:51.620+0000 7fc35206d700 1 -- 172.21.15.155:0/1704506489 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc34c069fc0 con 0x7fc34c0f6510 2024-02-10T09:02:51.622 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:51.620+0000 7fc333fff700 1 -- 172.21.15.155:0/1704506489 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc33c013380 con 0x7fc34c0f6510 2024-02-10T09:02:51.622 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:51.620+0000 7fc333fff700 1 -- 172.21.15.155:0/1704506489 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc33c014e50 con 0x7fc34c0f6510 2024-02-10T09:02:51.622 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:51.620+0000 7fc35206d700 1 -- 172.21.15.155:0/1704506489 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc34c06a450 con 0x7fc34c0f6510 2024-02-10T09:02:51.623 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:51.621+0000 7fc333fff700 1 -- 172.21.15.155:0/1704506489 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc33c0095f0 con 0x7fc34c0f6510 2024-02-10T09:02:51.623 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:51.621+0000 7fc35206d700 1 -- 172.21.15.155:0/1704506489 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc34c059100 con 0x7fc34c0f6510 2024-02-10T09:02:51.623 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:51.621+0000 7fc333fff700 1 --2- 172.21.15.155:0/1704506489 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fc33402ff10 0x7fc3340323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:02:51.623 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:51.621+0000 7fc333fff700 1 -- 172.21.15.155:0/1704506489 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc33c040e60 con 0x7fc34c0f6510 2024-02-10T09:02:51.624 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:51.621+0000 7fc34affd700 1 --2- 172.21.15.155:0/1704506489 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fc33402ff10 0x7fc3340323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:02:51.624 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:51.621+0000 7fc34affd700 1 --2- 172.21.15.155:0/1704506489 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fc33402ff10 0x7fc3340323d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fc3400096a0 tx=0x7fc340006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:02:51.626 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:51.625+0000 7fc333fff700 1 -- 172.21.15.155:0/1704506489 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc33c014420 con 0x7fc34c0f6510 2024-02-10T09:02:51.833 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:51.831+0000 7fc35206d700 1 -- 172.21.15.155:0/1704506489 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc34c045ad0 con 0x7fc34c0f6510 2024-02-10T09:02:51.833 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:51.832+0000 7fc333fff700 1 -- 172.21.15.155:0/1704506489 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc33c010230 con 0x7fc34c0f6510 2024-02-10T09:02:51.833 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:02:51.834 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:02:51.835 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:51.834+0000 7fc35206d700 1 -- 172.21.15.155:0/1704506489 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fc33402ff10 msgr2=0x7fc3340323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:02:51.835 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:51.834+0000 7fc35206d700 1 --2- 172.21.15.155:0/1704506489 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fc33402ff10 0x7fc3340323d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fc3400096a0 tx=0x7fc340006b40).stop 2024-02-10T09:02:51.835 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:51.834+0000 7fc35206d700 1 -- 172.21.15.155:0/1704506489 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc34c0f6510 msgr2=0x7fc34c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:02:51.835 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:51.834+0000 7fc35206d700 1 --2- 172.21.15.155:0/1704506489 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc34c0f6510 0x7fc34c06b9a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fc33c005010 tx=0x7fc33c00b040).stop 2024-02-10T09:02:51.835 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:51.834+0000 7fc35206d700 1 -- 172.21.15.155:0/1704506489 shutdown_connections 2024-02-10T09:02:51.835 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:51.834+0000 7fc35206d700 1 --2- 172.21.15.155:0/1704506489 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fc33402ff10 0x7fc3340323d0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:02:51.835 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:51.834+0000 7fc35206d700 1 --2- 172.21.15.155:0/1704506489 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc34c0f6510 0x7fc34c06b9a0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:02:51.835 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:51.834+0000 7fc35206d700 1 -- 172.21.15.155:0/1704506489 >> 172.21.15.155:0/1704506489 conn(0x7fc34c0f1c70 msgr2=0x7fc34c0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:02:51.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:51.834+0000 7fc35206d700 1 -- 172.21.15.155:0/1704506489 shutdown_connections 2024-02-10T09:02:51.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:51.834+0000 7fc35206d700 1 -- 172.21.15.155:0/1704506489 wait complete. 2024-02-10T09:02:51.836 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:02:52.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:02:52 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:02:51.833698+0000 mon.smithi077 (mon.0) 177 : audit [DBG] from='client.? 172.21.15.155:0/1704506489' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:02:53.585 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:02:53.585 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:02:54.148 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:02:53 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:02:52.993433+0000 mgr.smithi077.dxntvw (mgr.14162) 124 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:02:55.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:02:55 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:02:54.993770+0000 mgr.smithi077.dxntvw (mgr.14162) 125 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:02:55.636 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:55.608+0000 7f6fb659e700 1 -- 172.21.15.155:0/3882895117 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fb0060280 msgr2=0x7f6fb00f9010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:02:55.637 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:55.608+0000 7f6fb659e700 1 --2- 172.21.15.155:0/3882895117 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fb0060280 0x7f6fb00f9010 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f6fa0009fb0 tx=0x7f6fa0009b30).stop 2024-02-10T09:02:55.637 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:55.608+0000 7f6fb659e700 1 -- 172.21.15.155:0/3882895117 shutdown_connections 2024-02-10T09:02:55.637 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:55.608+0000 7f6fb659e700 1 --2- 172.21.15.155:0/3882895117 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fb0060280 0x7f6fb00f9010 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:02:55.637 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:55.608+0000 7f6fb659e700 1 -- 172.21.15.155:0/3882895117 >> 172.21.15.155:0/3882895117 conn(0x7f6fb00f0a90 msgr2=0x7f6fb00f2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:02:55.637 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:55.608+0000 7f6fb659e700 1 -- 172.21.15.155:0/3882895117 shutdown_connections 2024-02-10T09:02:55.637 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:55.608+0000 7f6fb659e700 1 -- 172.21.15.155:0/3882895117 wait complete. 2024-02-10T09:02:55.637 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:55.608+0000 7f6fb659e700 1 Processor -- start 2024-02-10T09:02:55.637 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:55.608+0000 7f6fb659e700 1 -- start start 2024-02-10T09:02:55.637 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:55.608+0000 7f6fb659e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fb0060280 0x7f6fb0106a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:02:55.637 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:55.609+0000 7f6fb659e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb0106ef0 con 0x7f6fb0060280 2024-02-10T09:02:55.637 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:55.609+0000 7f6faffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fb0060280 0x7f6fb0106a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:02:55.638 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:55.609+0000 7f6faffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fb0060280 0x7f6fb0106a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:33794/0 (socket says 172.21.15.155:33794) 2024-02-10T09:02:55.638 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:55.609+0000 7f6faffff700 1 -- 172.21.15.155:0/2407363080 learned_addr learned my addr 172.21.15.155:0/2407363080 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:02:55.638 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:55.609+0000 7f6faffff700 1 -- 172.21.15.155:0/2407363080 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fa0005040 con 0x7f6fb0060280 2024-02-10T09:02:55.638 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:55.609+0000 7f6faffff700 1 --2- 172.21.15.155:0/2407363080 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fb0060280 0x7f6fb0106a00 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f6fa00089f0 tx=0x7f6fa000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:02:55.638 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:55.609+0000 7f6facff9700 1 -- 172.21.15.155:0/2407363080 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6fa0009320 con 0x7f6fb0060280 2024-02-10T09:02:55.638 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:55.609+0000 7f6fb659e700 1 -- 172.21.15.155:0/2407363080 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fb01070f0 con 0x7f6fb0060280 2024-02-10T09:02:55.638 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:55.609+0000 7f6fb659e700 1 -- 172.21.15.155:0/2407363080 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fb0107580 con 0x7f6fb0060280 2024-02-10T09:02:55.638 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:55.610+0000 7f6facff9700 1 -- 172.21.15.155:0/2407363080 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6fa0013380 con 0x7f6fb0060280 2024-02-10T09:02:55.638 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:55.610+0000 7f6facff9700 1 -- 172.21.15.155:0/2407363080 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6fa0014e40 con 0x7f6fb0060280 2024-02-10T09:02:55.638 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:55.610+0000 7f6facff9700 1 -- 172.21.15.155:0/2407363080 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6fa0014420 con 0x7f6fb0060280 2024-02-10T09:02:55.638 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:55.610+0000 7f6fb659e700 1 -- 172.21.15.155:0/2407363080 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fb0045ad0 con 0x7f6fb0060280 2024-02-10T09:02:55.638 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:55.611+0000 7f6facff9700 1 --2- 172.21.15.155:0/2407363080 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f6f9802ff10 0x7f6f980323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:02:55.638 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:55.611+0000 7f6facff9700 1 -- 172.21.15.155:0/2407363080 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6fa0041180 con 0x7f6fb0060280 2024-02-10T09:02:55.638 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:55.611+0000 7f6faf7fe700 1 --2- 172.21.15.155:0/2407363080 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f6f9802ff10 0x7f6f980323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:02:55.638 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:55.611+0000 7f6faf7fe700 1 --2- 172.21.15.155:0/2407363080 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f6f9802ff10 0x7f6f980323d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f6fa40096a0 tx=0x7f6fa4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:02:55.639 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:55.615+0000 7f6facff9700 1 -- 172.21.15.155:0/2407363080 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6fa0012940 con 0x7f6fb0060280 2024-02-10T09:02:55.826 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:55.825+0000 7f6fb659e700 1 -- 172.21.15.155:0/2407363080 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6fb005ea50 con 0x7f6fb0060280 2024-02-10T09:02:55.827 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:55.826+0000 7f6facff9700 1 -- 172.21.15.155:0/2407363080 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6fa0012b50 con 0x7f6fb0060280 2024-02-10T09:02:55.827 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:02:55.827 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:02:55.829 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:55.828+0000 7f6fb659e700 1 -- 172.21.15.155:0/2407363080 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f6f9802ff10 msgr2=0x7f6f980323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:02:55.829 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:55.828+0000 7f6fb659e700 1 --2- 172.21.15.155:0/2407363080 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f6f9802ff10 0x7f6f980323d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f6fa40096a0 tx=0x7f6fa4006b40).stop 2024-02-10T09:02:55.829 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:55.828+0000 7f6fb659e700 1 -- 172.21.15.155:0/2407363080 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fb0060280 msgr2=0x7f6fb0106a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:02:55.829 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:55.828+0000 7f6fb659e700 1 --2- 172.21.15.155:0/2407363080 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fb0060280 0x7f6fb0106a00 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f6fa00089f0 tx=0x7f6fa000b040).stop 2024-02-10T09:02:55.829 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:55.828+0000 7f6fb659e700 1 -- 172.21.15.155:0/2407363080 shutdown_connections 2024-02-10T09:02:55.830 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:55.828+0000 7f6fb659e700 1 --2- 172.21.15.155:0/2407363080 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f6f9802ff10 0x7f6f980323d0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:02:55.830 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:55.828+0000 7f6fb659e700 1 --2- 172.21.15.155:0/2407363080 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6fb0060280 0x7f6fb0106a00 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:02:55.830 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:55.828+0000 7f6fb659e700 1 -- 172.21.15.155:0/2407363080 >> 172.21.15.155:0/2407363080 conn(0x7f6fb00f0a90 msgr2=0x7f6fb00f14a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:02:55.830 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:55.828+0000 7f6fb659e700 1 -- 172.21.15.155:0/2407363080 shutdown_connections 2024-02-10T09:02:55.830 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:55.828+0000 7f6fb659e700 1 -- 172.21.15.155:0/2407363080 wait complete. 2024-02-10T09:02:55.830 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:02:56.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:02:56 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:02:55.827233+0000 mon.smithi077 (mon.0) 178 : audit [DBG] from='client.? 172.21.15.155:0/2407363080' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:02:57.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:02:57 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:02:56.994118+0000 mgr.smithi077.dxntvw (mgr.14162) 126 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:02:57.516 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:02:57.516 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:02:59.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:02:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:02:58.994433+0000 mgr.smithi077.dxntvw (mgr.14162) 127 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:02:59.534 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:59.532+0000 7f9e12ac5700 1 -- 172.21.15.155:0/1621411805 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e0c0f51a0 msgr2=0x7f9e0c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:02:59.534 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:59.532+0000 7f9e12ac5700 1 --2- 172.21.15.155:0/1621411805 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e0c0f51a0 0x7f9e0c0f55c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f9e08009fb0 tx=0x7f9e08009b30).stop 2024-02-10T09:02:59.534 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:59.532+0000 7f9e12ac5700 1 -- 172.21.15.155:0/1621411805 shutdown_connections 2024-02-10T09:02:59.534 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:59.532+0000 7f9e12ac5700 1 --2- 172.21.15.155:0/1621411805 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e0c0f51a0 0x7f9e0c0f55c0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:02:59.534 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:59.532+0000 7f9e12ac5700 1 -- 172.21.15.155:0/1621411805 >> 172.21.15.155:0/1621411805 conn(0x7f9e0c0f09a0 msgr2=0x7f9e0c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:02:59.534 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:59.532+0000 7f9e12ac5700 1 -- 172.21.15.155:0/1621411805 shutdown_connections 2024-02-10T09:02:59.534 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:59.532+0000 7f9e12ac5700 1 -- 172.21.15.155:0/1621411805 wait complete. 2024-02-10T09:02:59.535 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:59.532+0000 7f9e12ac5700 1 Processor -- start 2024-02-10T09:02:59.535 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:59.532+0000 7f9e12ac5700 1 -- start start 2024-02-10T09:02:59.535 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:59.533+0000 7f9e12ac5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e0c0f51a0 0x7f9e0c1065c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:02:59.535 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:59.533+0000 7f9e12ac5700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e0c106ab0 con 0x7f9e0c0f51a0 2024-02-10T09:02:59.535 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:59.533+0000 7f9e11ac3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e0c0f51a0 0x7f9e0c1065c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:02:59.535 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:59.533+0000 7f9e11ac3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e0c0f51a0 0x7f9e0c1065c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:33808/0 (socket says 172.21.15.155:33808) 2024-02-10T09:02:59.535 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:59.533+0000 7f9e11ac3700 1 -- 172.21.15.155:0/3804916183 learned_addr learned my addr 172.21.15.155:0/3804916183 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:02:59.537 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:59.533+0000 7f9e11ac3700 1 -- 172.21.15.155:0/3804916183 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e08005040 con 0x7f9e0c0f51a0 2024-02-10T09:02:59.537 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:59.533+0000 7f9e11ac3700 1 --2- 172.21.15.155:0/3804916183 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e0c0f51a0 0x7f9e0c1065c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f9e080089f0 tx=0x7f9e0800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:02:59.537 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:59.534+0000 7f9e027fc700 1 -- 172.21.15.155:0/3804916183 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e08009320 con 0x7f9e0c0f51a0 2024-02-10T09:02:59.537 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:59.534+0000 7f9e12ac5700 1 -- 172.21.15.155:0/3804916183 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e0c106cb0 con 0x7f9e0c0f51a0 2024-02-10T09:02:59.537 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:59.534+0000 7f9e027fc700 1 -- 172.21.15.155:0/3804916183 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9e08013380 con 0x7f9e0c0f51a0 2024-02-10T09:02:59.537 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:59.534+0000 7f9e027fc700 1 -- 172.21.15.155:0/3804916183 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e08014e20 con 0x7f9e0c0f51a0 2024-02-10T09:02:59.537 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:59.534+0000 7f9e12ac5700 1 -- 172.21.15.155:0/3804916183 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e0c107140 con 0x7f9e0c0f51a0 2024-02-10T09:02:59.537 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:59.535+0000 7f9e027fc700 1 -- 172.21.15.155:0/3804916183 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9e080095f0 con 0x7f9e0c0f51a0 2024-02-10T09:02:59.537 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:59.535+0000 7f9e12ac5700 1 -- 172.21.15.155:0/3804916183 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e0c045ad0 con 0x7f9e0c0f51a0 2024-02-10T09:02:59.537 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:59.535+0000 7f9e027fc700 1 --2- 172.21.15.155:0/3804916183 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f9df802ff10 0x7f9df80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:02:59.537 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:59.535+0000 7f9e027fc700 1 -- 172.21.15.155:0/3804916183 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9e08040e90 con 0x7f9e0c0f51a0 2024-02-10T09:02:59.537 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:59.535+0000 7f9e112c2700 1 --2- 172.21.15.155:0/3804916183 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f9df802ff10 0x7f9df80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:02:59.538 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:59.535+0000 7f9e112c2700 1 --2- 172.21.15.155:0/3804916183 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f9df802ff10 0x7f9df80323d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f9dfc0096a0 tx=0x7f9dfc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:02:59.540 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:59.540+0000 7f9e027fc700 1 -- 172.21.15.155:0/3804916183 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9e08014420 con 0x7f9e0c0f51a0 2024-02-10T09:02:59.748 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:59.747+0000 7f9e12ac5700 1 -- 172.21.15.155:0/3804916183 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9e0c107ff0 con 0x7f9e0c0f51a0 2024-02-10T09:02:59.749 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:59.748+0000 7f9e027fc700 1 -- 172.21.15.155:0/3804916183 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9e0801d9d0 con 0x7f9e0c0f51a0 2024-02-10T09:02:59.749 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:02:59.749 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:02:59.750 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:59.749+0000 7f9e12ac5700 1 -- 172.21.15.155:0/3804916183 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f9df802ff10 msgr2=0x7f9df80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:02:59.750 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:59.749+0000 7f9e12ac5700 1 --2- 172.21.15.155:0/3804916183 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f9df802ff10 0x7f9df80323d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f9dfc0096a0 tx=0x7f9dfc006b40).stop 2024-02-10T09:02:59.751 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:59.749+0000 7f9e12ac5700 1 -- 172.21.15.155:0/3804916183 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e0c0f51a0 msgr2=0x7f9e0c1065c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:02:59.751 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:59.749+0000 7f9e12ac5700 1 --2- 172.21.15.155:0/3804916183 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e0c0f51a0 0x7f9e0c1065c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f9e080089f0 tx=0x7f9e0800b040).stop 2024-02-10T09:02:59.751 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:59.749+0000 7f9e12ac5700 1 -- 172.21.15.155:0/3804916183 shutdown_connections 2024-02-10T09:02:59.751 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:59.749+0000 7f9e12ac5700 1 --2- 172.21.15.155:0/3804916183 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f9df802ff10 0x7f9df80323d0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:02:59.751 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:59.749+0000 7f9e12ac5700 1 --2- 172.21.15.155:0/3804916183 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9e0c0f51a0 0x7f9e0c1065c0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:02:59.751 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:59.749+0000 7f9e12ac5700 1 -- 172.21.15.155:0/3804916183 >> 172.21.15.155:0/3804916183 conn(0x7f9e0c0f09a0 msgr2=0x7f9e0c05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:02:59.751 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:59.749+0000 7f9e12ac5700 1 -- 172.21.15.155:0/3804916183 shutdown_connections 2024-02-10T09:02:59.751 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:02:59.749+0000 7f9e12ac5700 1 -- 172.21.15.155:0/3804916183 wait complete. 2024-02-10T09:02:59.751 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:03:00.398 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:00 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:02:59.749091+0000 mon.smithi077 (mon.0) 179 : audit [DBG] from='client.? 172.21.15.155:0/3804916183' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:03:01.387 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:03:01.388 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:03:01.398 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:01 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:03:00.994797+0000 mgr.smithi077.dxntvw (mgr.14162) 128 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:03:03.291 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:03.276+0000 7f44e5f03700 1 -- 172.21.15.155:0/938845268 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44e0060220 msgr2=0x7f44e00f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:03:03.291 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:03.276+0000 7f44e5f03700 1 --2- 172.21.15.155:0/938845268 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44e0060220 0x7f44e00f9150 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f44d0009fb0 tx=0x7f44d0009b30).stop 2024-02-10T09:03:03.291 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:03.276+0000 7f44e5f03700 1 -- 172.21.15.155:0/938845268 shutdown_connections 2024-02-10T09:03:03.291 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:03.276+0000 7f44e5f03700 1 --2- 172.21.15.155:0/938845268 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44e0060220 0x7f44e00f9150 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:03:03.291 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:03.276+0000 7f44e5f03700 1 -- 172.21.15.155:0/938845268 >> 172.21.15.155:0/938845268 conn(0x7f44e00f0b70 msgr2=0x7f44e00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:03:03.291 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:03.276+0000 7f44e5f03700 1 -- 172.21.15.155:0/938845268 shutdown_connections 2024-02-10T09:03:03.291 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:03.276+0000 7f44e5f03700 1 -- 172.21.15.155:0/938845268 wait complete. 2024-02-10T09:03:03.291 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:03.277+0000 7f44e5f03700 1 Processor -- start 2024-02-10T09:03:03.291 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:03.277+0000 7f44e5f03700 1 -- start start 2024-02-10T09:03:03.291 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:03.277+0000 7f44e5f03700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44e0060220 0x7f44e01067a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:03:03.292 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:03.277+0000 7f44e5f03700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44e0106c90 con 0x7f44e0060220 2024-02-10T09:03:03.292 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:03.277+0000 7f44df7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44e0060220 0x7f44e01067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:03:03.292 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:03.277+0000 7f44df7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44e0060220 0x7f44e01067a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:37124/0 (socket says 172.21.15.155:37124) 2024-02-10T09:03:03.292 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:03.277+0000 7f44df7fe700 1 -- 172.21.15.155:0/993891942 learned_addr learned my addr 172.21.15.155:0/993891942 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:03:03.292 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:03.278+0000 7f44df7fe700 1 -- 172.21.15.155:0/993891942 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44d0005040 con 0x7f44e0060220 2024-02-10T09:03:03.292 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:03.278+0000 7f44df7fe700 1 --2- 172.21.15.155:0/993891942 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44e0060220 0x7f44e01067a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f44d00089f0 tx=0x7f44d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:03:03.292 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:03.278+0000 7f44d7fff700 1 -- 172.21.15.155:0/993891942 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f44d0009320 con 0x7f44e0060220 2024-02-10T09:03:03.292 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:03.278+0000 7f44e5f03700 1 -- 172.21.15.155:0/993891942 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44e0106e90 con 0x7f44e0060220 2024-02-10T09:03:03.292 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:03.278+0000 7f44d7fff700 1 -- 172.21.15.155:0/993891942 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f44d0013380 con 0x7f44e0060220 2024-02-10T09:03:03.292 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:03.278+0000 7f44d7fff700 1 -- 172.21.15.155:0/993891942 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f44d0014e20 con 0x7f44e0060220 2024-02-10T09:03:03.292 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:03.278+0000 7f44e5f03700 1 -- 172.21.15.155:0/993891942 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44e0107320 con 0x7f44e0060220 2024-02-10T09:03:03.292 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:03.279+0000 7f44d7fff700 1 -- 172.21.15.155:0/993891942 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f44d0014420 con 0x7f44e0060220 2024-02-10T09:03:03.292 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:03.279+0000 7f44e5f03700 1 -- 172.21.15.155:0/993891942 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44e0045ad0 con 0x7f44e0060220 2024-02-10T09:03:03.293 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:03.279+0000 7f44d7fff700 1 --2- 172.21.15.155:0/993891942 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f44cc02ff10 0x7f44cc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:03:03.293 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:03.279+0000 7f44d7fff700 1 -- 172.21.15.155:0/993891942 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f44d0040f00 con 0x7f44e0060220 2024-02-10T09:03:03.293 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:03.279+0000 7f44deffd700 1 --2- 172.21.15.155:0/993891942 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f44cc02ff10 0x7f44cc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:03:03.293 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:03.280+0000 7f44deffd700 1 --2- 172.21.15.155:0/993891942 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f44cc02ff10 0x7f44cc0323d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f44c80096a0 tx=0x7f44c8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:03:03.293 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:03.284+0000 7f44d7fff700 1 -- 172.21.15.155:0/993891942 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f44d00095f0 con 0x7f44e0060220 2024-02-10T09:03:03.489 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:03.488+0000 7f44e5f03700 1 -- 172.21.15.155:0/993891942 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f44e01081d0 con 0x7f44e0060220 2024-02-10T09:03:03.489 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:03.488+0000 7f44d7fff700 1 -- 172.21.15.155:0/993891942 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f44d001db70 con 0x7f44e0060220 2024-02-10T09:03:03.489 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:03:03.490 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:03:03.491 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:03.490+0000 7f44e5f03700 1 -- 172.21.15.155:0/993891942 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f44cc02ff10 msgr2=0x7f44cc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:03:03.491 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:03.490+0000 7f44e5f03700 1 --2- 172.21.15.155:0/993891942 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f44cc02ff10 0x7f44cc0323d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f44c80096a0 tx=0x7f44c8006b40).stop 2024-02-10T09:03:03.491 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:03.490+0000 7f44e5f03700 1 -- 172.21.15.155:0/993891942 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44e0060220 msgr2=0x7f44e01067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:03:03.491 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:03.490+0000 7f44e5f03700 1 --2- 172.21.15.155:0/993891942 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44e0060220 0x7f44e01067a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f44d00089f0 tx=0x7f44d000b040).stop 2024-02-10T09:03:03.491 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:03.490+0000 7f44e5f03700 1 -- 172.21.15.155:0/993891942 shutdown_connections 2024-02-10T09:03:03.491 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:03.490+0000 7f44e5f03700 1 --2- 172.21.15.155:0/993891942 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f44cc02ff10 0x7f44cc0323d0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:03:03.491 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:03.490+0000 7f44e5f03700 1 --2- 172.21.15.155:0/993891942 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f44e0060220 0x7f44e01067a0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:03:03.491 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:03.490+0000 7f44e5f03700 1 -- 172.21.15.155:0/993891942 >> 172.21.15.155:0/993891942 conn(0x7f44e00f0b70 msgr2=0x7f44e00f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:03:03.491 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:03.490+0000 7f44e5f03700 1 -- 172.21.15.155:0/993891942 shutdown_connections 2024-02-10T09:03:03.491 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:03.490+0000 7f44e5f03700 1 -- 172.21.15.155:0/993891942 wait complete. 2024-02-10T09:03:03.492 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:03:04.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:03 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:03:02.995186+0000 mgr.smithi077.dxntvw (mgr.14162) 129 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:03:04.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:03 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:03:03.489742+0000 mon.smithi077 (mon.0) 180 : audit [DBG] from='client.? 172.21.15.155:0/993891942' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:03:05.213 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:03:05.214 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:03:05.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:05 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:03:04.995495+0000 mgr.smithi077.dxntvw (mgr.14162) 130 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:03:07.071 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:07.070+0000 7f609d964700 1 -- 172.21.15.155:0/677426481 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60980f6510 msgr2=0x7f60980f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:03:07.072 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:07.070+0000 7f609d964700 1 --2- 172.21.15.155:0/677426481 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60980f6510 0x7f60980f6930 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f608c009fb0 tx=0x7f608c009b30).stop 2024-02-10T09:03:07.072 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:07.070+0000 7f609d964700 1 -- 172.21.15.155:0/677426481 shutdown_connections 2024-02-10T09:03:07.072 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:07.070+0000 7f609d964700 1 --2- 172.21.15.155:0/677426481 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60980f6510 0x7f60980f6930 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:03:07.072 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:07.070+0000 7f609d964700 1 -- 172.21.15.155:0/677426481 >> 172.21.15.155:0/677426481 conn(0x7f60980f1c70 msgr2=0x7f60980f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:03:07.072 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:07.070+0000 7f609d964700 1 -- 172.21.15.155:0/677426481 shutdown_connections 2024-02-10T09:03:07.072 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:07.070+0000 7f609d964700 1 -- 172.21.15.155:0/677426481 wait complete. 2024-02-10T09:03:07.074 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:07.071+0000 7f609d964700 1 Processor -- start 2024-02-10T09:03:07.074 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:07.071+0000 7f609d964700 1 -- start start 2024-02-10T09:03:07.074 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:07.071+0000 7f609d964700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60980f6510 0x7f60981067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:03:07.074 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:07.071+0000 7f609d964700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6098106cd0 con 0x7f60980f6510 2024-02-10T09:03:07.074 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:07.071+0000 7f6096ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60980f6510 0x7f60981067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:03:07.074 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:07.071+0000 7f6096ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60980f6510 0x7f60981067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:37138/0 (socket says 172.21.15.155:37138) 2024-02-10T09:03:07.075 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:07.071+0000 7f6096ffd700 1 -- 172.21.15.155:0/4140867986 learned_addr learned my addr 172.21.15.155:0/4140867986 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:03:07.075 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:07.072+0000 7f6096ffd700 1 -- 172.21.15.155:0/4140867986 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f608c005040 con 0x7f60980f6510 2024-02-10T09:03:07.075 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:07.072+0000 7f6096ffd700 1 --2- 172.21.15.155:0/4140867986 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60980f6510 0x7f60981067e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f608c0089f0 tx=0x7f608c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:03:07.075 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:07.072+0000 7f607f7fe700 1 -- 172.21.15.155:0/4140867986 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f608c009320 con 0x7f60980f6510 2024-02-10T09:03:07.075 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:07.072+0000 7f609d964700 1 -- 172.21.15.155:0/4140867986 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6098106ed0 con 0x7f60980f6510 2024-02-10T09:03:07.075 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:07.072+0000 7f607f7fe700 1 -- 172.21.15.155:0/4140867986 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f608c013380 con 0x7f60980f6510 2024-02-10T09:03:07.075 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:07.072+0000 7f607f7fe700 1 -- 172.21.15.155:0/4140867986 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f608c01ed80 con 0x7f60980f6510 2024-02-10T09:03:07.075 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:07.072+0000 7f609d964700 1 -- 172.21.15.155:0/4140867986 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6098107360 con 0x7f60980f6510 2024-02-10T09:03:07.075 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:07.073+0000 7f609d964700 1 -- 172.21.15.155:0/4140867986 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6098045ad0 con 0x7f60980f6510 2024-02-10T09:03:07.075 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:07.073+0000 7f607f7fe700 1 -- 172.21.15.155:0/4140867986 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f608c01e420 con 0x7f60980f6510 2024-02-10T09:03:07.075 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:07.073+0000 7f607f7fe700 1 --2- 172.21.15.155:0/4140867986 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f608002ff60 0x7f6080032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:03:07.075 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:07.073+0000 7f607f7fe700 1 -- 172.21.15.155:0/4140867986 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f608c040fc0 con 0x7f60980f6510 2024-02-10T09:03:07.076 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:07.073+0000 7f60967fc700 1 --2- 172.21.15.155:0/4140867986 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f608002ff60 0x7f6080032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:03:07.076 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:07.074+0000 7f60967fc700 1 --2- 172.21.15.155:0/4140867986 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f608002ff60 0x7f6080032420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f60880096a0 tx=0x7f6088006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:03:07.079 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:07.078+0000 7f607f7fe700 1 -- 172.21.15.155:0/4140867986 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f608c0095f0 con 0x7f60980f6510 2024-02-10T09:03:07.285 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:07.283+0000 7f609d964700 1 -- 172.21.15.155:0/4140867986 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f60981081b0 con 0x7f60980f6510 2024-02-10T09:03:07.285 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:07.284+0000 7f607f7fe700 1 -- 172.21.15.155:0/4140867986 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f608c01cb10 con 0x7f60980f6510 2024-02-10T09:03:07.285 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:03:07.286 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:03:07.287 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:07.286+0000 7f609d964700 1 -- 172.21.15.155:0/4140867986 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f608002ff60 msgr2=0x7f6080032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:03:07.287 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:07.286+0000 7f609d964700 1 --2- 172.21.15.155:0/4140867986 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f608002ff60 0x7f6080032420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f60880096a0 tx=0x7f6088006b40).stop 2024-02-10T09:03:07.287 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:07.286+0000 7f609d964700 1 -- 172.21.15.155:0/4140867986 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60980f6510 msgr2=0x7f60981067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:03:07.287 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:07.286+0000 7f609d964700 1 --2- 172.21.15.155:0/4140867986 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60980f6510 0x7f60981067e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f608c0089f0 tx=0x7f608c00b040).stop 2024-02-10T09:03:07.287 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:07.286+0000 7f609d964700 1 -- 172.21.15.155:0/4140867986 shutdown_connections 2024-02-10T09:03:07.287 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:07.286+0000 7f609d964700 1 --2- 172.21.15.155:0/4140867986 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f608002ff60 0x7f6080032420 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:03:07.287 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:07.286+0000 7f609d964700 1 --2- 172.21.15.155:0/4140867986 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f60980f6510 0x7f60981067e0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:03:07.287 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:07.286+0000 7f609d964700 1 -- 172.21.15.155:0/4140867986 >> 172.21.15.155:0/4140867986 conn(0x7f60980f1c70 msgr2=0x7f60980f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:03:07.288 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:07.286+0000 7f609d964700 1 -- 172.21.15.155:0/4140867986 shutdown_connections 2024-02-10T09:03:07.288 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:07.286+0000 7f609d964700 1 -- 172.21.15.155:0/4140867986 wait complete. 2024-02-10T09:03:07.288 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:03:07.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:07 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:03:06.995767+0000 mgr.smithi077.dxntvw (mgr.14162) 131 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:03:08.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:08 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:03:07.285511+0000 mon.smithi077 (mon.0) 181 : audit [DBG] from='client.? 172.21.15.155:0/4140867986' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:03:09.022 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:03:09.022 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:03:09.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:09 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:03:08.996032+0000 mgr.smithi077.dxntvw (mgr.14162) 132 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:03:11.116 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:11.092+0000 7f07dcae6700 1 -- 172.21.15.155:0/3641032351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07d80f5410 msgr2=0x7f07d80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:03:11.116 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:11.092+0000 7f07dcae6700 1 --2- 172.21.15.155:0/3641032351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07d80f5410 0x7f07d80f5830 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f07cc009fb0 tx=0x7f07cc009b30).stop 2024-02-10T09:03:11.116 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:11.092+0000 7f07dcae6700 1 -- 172.21.15.155:0/3641032351 shutdown_connections 2024-02-10T09:03:11.116 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:11.092+0000 7f07dcae6700 1 --2- 172.21.15.155:0/3641032351 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07d80f5410 0x7f07d80f5830 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:03:11.116 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:11.092+0000 7f07dcae6700 1 -- 172.21.15.155:0/3641032351 >> 172.21.15.155:0/3641032351 conn(0x7f07d80f0b70 msgr2=0x7f07d80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:03:11.116 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:11.093+0000 7f07dcae6700 1 -- 172.21.15.155:0/3641032351 shutdown_connections 2024-02-10T09:03:11.116 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:11.093+0000 7f07dcae6700 1 -- 172.21.15.155:0/3641032351 wait complete. 2024-02-10T09:03:11.116 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:11.093+0000 7f07dcae6700 1 Processor -- start 2024-02-10T09:03:11.116 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:11.093+0000 7f07dcae6700 1 -- start start 2024-02-10T09:03:11.116 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:11.093+0000 7f07dcae6700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07d80f5410 0x7f07d80fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:03:11.116 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:11.093+0000 7f07dcae6700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07d80fb960 con 0x7f07d80f5410 2024-02-10T09:03:11.116 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:11.093+0000 7f07d659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07d80f5410 0x7f07d80fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:03:11.116 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:11.093+0000 7f07d659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07d80f5410 0x7f07d80fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:37146/0 (socket says 172.21.15.155:37146) 2024-02-10T09:03:11.117 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:11.093+0000 7f07d659c700 1 -- 172.21.15.155:0/881879324 learned_addr learned my addr 172.21.15.155:0/881879324 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:03:11.117 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:11.094+0000 7f07d659c700 1 -- 172.21.15.155:0/881879324 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07cc005040 con 0x7f07d80f5410 2024-02-10T09:03:11.117 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:11.094+0000 7f07d659c700 1 --2- 172.21.15.155:0/881879324 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07d80f5410 0x7f07d80fd4e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f07cc005010 tx=0x7f07cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:03:11.117 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:11.094+0000 7f07c6ffd700 1 -- 172.21.15.155:0/881879324 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07cc009320 con 0x7f07d80f5410 2024-02-10T09:03:11.117 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:11.094+0000 7f07dcae6700 1 -- 172.21.15.155:0/881879324 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07d80fbb00 con 0x7f07d80f5410 2024-02-10T09:03:11.117 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:11.094+0000 7f07c6ffd700 1 -- 172.21.15.155:0/881879324 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f07cc013380 con 0x7f07d80f5410 2024-02-10T09:03:11.117 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:11.094+0000 7f07c6ffd700 1 -- 172.21.15.155:0/881879324 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07cc014e50 con 0x7f07d80f5410 2024-02-10T09:03:11.117 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:11.094+0000 7f07dcae6700 1 -- 172.21.15.155:0/881879324 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07d80fbf90 con 0x7f07d80f5410 2024-02-10T09:03:11.117 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:11.095+0000 7f07c6ffd700 1 -- 172.21.15.155:0/881879324 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f07cc0095f0 con 0x7f07d80f5410 2024-02-10T09:03:11.117 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:11.095+0000 7f07dcae6700 1 -- 172.21.15.155:0/881879324 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07d8059100 con 0x7f07d80f5410 2024-02-10T09:03:11.117 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:11.095+0000 7f07c6ffd700 1 --2- 172.21.15.155:0/881879324 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f07c002ff10 0x7f07c00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:03:11.117 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:11.095+0000 7f07c6ffd700 1 -- 172.21.15.155:0/881879324 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f07cc040fa0 con 0x7f07d80f5410 2024-02-10T09:03:11.117 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:11.095+0000 7f07d5d9b700 1 --2- 172.21.15.155:0/881879324 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f07c002ff10 0x7f07c00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:03:11.117 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:11.096+0000 7f07d5d9b700 1 --2- 172.21.15.155:0/881879324 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f07c002ff10 0x7f07c00323d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f07c80096a0 tx=0x7f07c8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:03:11.117 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:11.100+0000 7f07c6ffd700 1 -- 172.21.15.155:0/881879324 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f07cc014420 con 0x7f07d80f5410 2024-02-10T09:03:11.309 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:11.309+0000 7f07dcae6700 1 -- 172.21.15.155:0/881879324 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f07d80fafa0 con 0x7f07d80f5410 2024-02-10T09:03:11.310 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:11.310+0000 7f07c6ffd700 1 -- 172.21.15.155:0/881879324 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f07cc010230 con 0x7f07d80f5410 2024-02-10T09:03:11.310 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:03:11.310 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:03:11.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:11.312+0000 7f07dcae6700 1 -- 172.21.15.155:0/881879324 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f07c002ff10 msgr2=0x7f07c00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:03:11.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:11.312+0000 7f07dcae6700 1 --2- 172.21.15.155:0/881879324 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f07c002ff10 0x7f07c00323d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f07c80096a0 tx=0x7f07c8006b40).stop 2024-02-10T09:03:11.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:11.312+0000 7f07dcae6700 1 -- 172.21.15.155:0/881879324 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07d80f5410 msgr2=0x7f07d80fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:03:11.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:11.312+0000 7f07dcae6700 1 --2- 172.21.15.155:0/881879324 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07d80f5410 0x7f07d80fd4e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f07cc005010 tx=0x7f07cc00b040).stop 2024-02-10T09:03:11.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:11.312+0000 7f07dcae6700 1 -- 172.21.15.155:0/881879324 shutdown_connections 2024-02-10T09:03:11.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:11.312+0000 7f07dcae6700 1 --2- 172.21.15.155:0/881879324 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f07c002ff10 0x7f07c00323d0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:03:11.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:11.312+0000 7f07dcae6700 1 --2- 172.21.15.155:0/881879324 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07d80f5410 0x7f07d80fd4e0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:03:11.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:11.312+0000 7f07dcae6700 1 -- 172.21.15.155:0/881879324 >> 172.21.15.155:0/881879324 conn(0x7f07d80f0b70 msgr2=0x7f07d805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:03:11.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:11.312+0000 7f07dcae6700 1 -- 172.21.15.155:0/881879324 shutdown_connections 2024-02-10T09:03:11.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:11.312+0000 7f07dcae6700 1 -- 172.21.15.155:0/881879324 wait complete. 2024-02-10T09:03:11.313 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:03:11.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:11 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:03:10.996245+0000 mgr.smithi077.dxntvw (mgr.14162) 133 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:03:12.398 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:12 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:03:11.310445+0000 mon.smithi077 (mon.0) 182 : audit [DBG] from='client.? 172.21.15.155:0/881879324' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:03:13.011 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:03:13.012 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:03:14.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:13 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:03:12.996500+0000 mgr.smithi077.dxntvw (mgr.14162) 134 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:03:14.991 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:14.972+0000 7f84904c2700 1 -- 172.21.15.155:0/1555541443 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f84880f5410 msgr2=0x7f84880f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:03:14.991 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:14.972+0000 7f84904c2700 1 --2- 172.21.15.155:0/1555541443 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f84880f5410 0x7f84880f5830 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f8478009fb0 tx=0x7f8478009b30).stop 2024-02-10T09:03:14.991 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:14.972+0000 7f84904c2700 1 -- 172.21.15.155:0/1555541443 shutdown_connections 2024-02-10T09:03:14.991 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:14.972+0000 7f84904c2700 1 --2- 172.21.15.155:0/1555541443 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f84880f5410 0x7f84880f5830 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:03:14.991 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:14.972+0000 7f84904c2700 1 -- 172.21.15.155:0/1555541443 >> 172.21.15.155:0/1555541443 conn(0x7f84880f0b70 msgr2=0x7f84880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:03:14.991 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:14.972+0000 7f84904c2700 1 -- 172.21.15.155:0/1555541443 shutdown_connections 2024-02-10T09:03:14.991 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:14.972+0000 7f84904c2700 1 -- 172.21.15.155:0/1555541443 wait complete. 2024-02-10T09:03:14.991 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:14.973+0000 7f84904c2700 1 Processor -- start 2024-02-10T09:03:14.991 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:14.973+0000 7f84904c2700 1 -- start start 2024-02-10T09:03:14.991 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:14.973+0000 7f84904c2700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f84880f5410 0x7f84881094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:03:14.992 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:14.973+0000 7f84904c2700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8488060240 con 0x7f84880f5410 2024-02-10T09:03:14.992 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:14.973+0000 7f848e25e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f84880f5410 0x7f84881094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:03:14.992 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:14.973+0000 7f848e25e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f84880f5410 0x7f84881094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:45982/0 (socket says 172.21.15.155:45982) 2024-02-10T09:03:14.992 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:14.973+0000 7f848e25e700 1 -- 172.21.15.155:0/3883741306 learned_addr learned my addr 172.21.15.155:0/3883741306 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:03:14.992 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:14.974+0000 7f848e25e700 1 -- 172.21.15.155:0/3883741306 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8478005040 con 0x7f84880f5410 2024-02-10T09:03:14.992 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:14.974+0000 7f848e25e700 1 --2- 172.21.15.155:0/3883741306 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f84880f5410 0x7f84881094e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f84780089f0 tx=0x7f847800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:03:14.992 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:14.974+0000 7f847effd700 1 -- 172.21.15.155:0/3883741306 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8478009320 con 0x7f84880f5410 2024-02-10T09:03:14.992 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:14.974+0000 7f84904c2700 1 -- 172.21.15.155:0/3883741306 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8488060440 con 0x7f84880f5410 2024-02-10T09:03:14.992 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:14.974+0000 7f847effd700 1 -- 172.21.15.155:0/3883741306 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8478013380 con 0x7f84880f5410 2024-02-10T09:03:14.992 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:14.974+0000 7f847effd700 1 -- 172.21.15.155:0/3883741306 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f847801ed80 con 0x7f84880f5410 2024-02-10T09:03:14.992 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:14.974+0000 7f84904c2700 1 -- 172.21.15.155:0/3883741306 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8488109b90 con 0x7f84880f5410 2024-02-10T09:03:14.992 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:14.975+0000 7f847effd700 1 -- 172.21.15.155:0/3883741306 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f847801e420 con 0x7f84880f5410 2024-02-10T09:03:14.992 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:14.975+0000 7f84904c2700 1 -- 172.21.15.155:0/3883741306 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8488045ad0 con 0x7f84880f5410 2024-02-10T09:03:14.992 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:14.975+0000 7f847effd700 1 --2- 172.21.15.155:0/3883741306 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f847402ffb0 0x7f8474032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:03:14.993 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:14.975+0000 7f847effd700 1 -- 172.21.15.155:0/3883741306 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f84780410c0 con 0x7f84880f5410 2024-02-10T09:03:14.993 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:14.975+0000 7f848da5d700 1 --2- 172.21.15.155:0/3883741306 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f847402ffb0 0x7f8474032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:03:14.993 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:14.976+0000 7f848da5d700 1 --2- 172.21.15.155:0/3883741306 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f847402ffb0 0x7f8474032470 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f84840096a0 tx=0x7f8484006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:03:14.993 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:14.980+0000 7f847effd700 1 -- 172.21.15.155:0/3883741306 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f847801cb10 con 0x7f84880f5410 2024-02-10T09:03:15.185 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:15.185+0000 7f84904c2700 1 -- 172.21.15.155:0/3883741306 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f848810a3b0 con 0x7f84880f5410 2024-02-10T09:03:15.186 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:15.186+0000 7f847effd700 1 -- 172.21.15.155:0/3883741306 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f847801cb10 con 0x7f84880f5410 2024-02-10T09:03:15.186 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:03:15.187 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:03:15.188 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:15.187+0000 7f84904c2700 1 -- 172.21.15.155:0/3883741306 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f847402ffb0 msgr2=0x7f8474032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:03:15.188 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:15.187+0000 7f84904c2700 1 --2- 172.21.15.155:0/3883741306 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f847402ffb0 0x7f8474032470 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f84840096a0 tx=0x7f8484006b40).stop 2024-02-10T09:03:15.189 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:15.187+0000 7f84904c2700 1 -- 172.21.15.155:0/3883741306 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f84880f5410 msgr2=0x7f84881094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:03:15.189 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:15.187+0000 7f84904c2700 1 --2- 172.21.15.155:0/3883741306 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f84880f5410 0x7f84881094e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f84780089f0 tx=0x7f847800b040).stop 2024-02-10T09:03:15.189 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:15.187+0000 7f84904c2700 1 -- 172.21.15.155:0/3883741306 shutdown_connections 2024-02-10T09:03:15.189 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:15.188+0000 7f84904c2700 1 --2- 172.21.15.155:0/3883741306 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f847402ffb0 0x7f8474032470 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:03:15.189 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:15.188+0000 7f84904c2700 1 --2- 172.21.15.155:0/3883741306 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f84880f5410 0x7f84881094e0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:03:15.189 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:15.188+0000 7f84904c2700 1 -- 172.21.15.155:0/3883741306 >> 172.21.15.155:0/3883741306 conn(0x7f84880f0b70 msgr2=0x7f84880f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:03:15.190 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:15.188+0000 7f84904c2700 1 -- 172.21.15.155:0/3883741306 shutdown_connections 2024-02-10T09:03:15.190 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:15.188+0000 7f84904c2700 1 -- 172.21.15.155:0/3883741306 wait complete. 2024-02-10T09:03:15.190 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:03:15.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:15 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:03:14.996814+0000 mgr.smithi077.dxntvw (mgr.14162) 135 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:03:16.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:16 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:03:15.186387+0000 mon.smithi077 (mon.0) 183 : audit [DBG] from='client.? 172.21.15.155:0/3883741306' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:03:16.916 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:03:16.916 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:03:17.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:17 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:03:16.997105+0000 mgr.smithi077.dxntvw (mgr.14162) 136 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:03:18.851 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:18.849+0000 7fa302600700 1 -- 172.21.15.155:0/1622363035 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2fc0601e0 msgr2=0x7fa2fc0f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:03:18.851 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:18.849+0000 7fa302600700 1 --2- 172.21.15.155:0/1622363035 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2fc0601e0 0x7fa2fc0f8f20 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7fa2ec009fb0 tx=0x7fa2ec009b30).stop 2024-02-10T09:03:18.851 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:18.849+0000 7fa302600700 1 -- 172.21.15.155:0/1622363035 shutdown_connections 2024-02-10T09:03:18.851 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:18.849+0000 7fa302600700 1 --2- 172.21.15.155:0/1622363035 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2fc0601e0 0x7fa2fc0f8f20 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:03:18.851 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:18.849+0000 7fa302600700 1 -- 172.21.15.155:0/1622363035 >> 172.21.15.155:0/1622363035 conn(0x7fa2fc0f09a0 msgr2=0x7fa2fc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:03:18.851 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:18.849+0000 7fa302600700 1 -- 172.21.15.155:0/1622363035 shutdown_connections 2024-02-10T09:03:18.851 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:18.849+0000 7fa302600700 1 -- 172.21.15.155:0/1622363035 wait complete. 2024-02-10T09:03:18.851 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:18.850+0000 7fa302600700 1 Processor -- start 2024-02-10T09:03:18.851 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:18.850+0000 7fa302600700 1 -- start start 2024-02-10T09:03:18.851 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:18.850+0000 7fa302600700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2fc0601e0 0x7fa2fc106600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:03:18.852 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:18.850+0000 7fa302600700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2fc106af0 con 0x7fa2fc0601e0 2024-02-10T09:03:18.852 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:18.850+0000 7fa3015fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2fc0601e0 0x7fa2fc106600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:03:18.852 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:18.850+0000 7fa3015fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2fc0601e0 0x7fa2fc106600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:45996/0 (socket says 172.21.15.155:45996) 2024-02-10T09:03:18.852 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:18.850+0000 7fa3015fe700 1 -- 172.21.15.155:0/49137962 learned_addr learned my addr 172.21.15.155:0/49137962 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:03:18.852 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:18.850+0000 7fa3015fe700 1 -- 172.21.15.155:0/49137962 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2ec005040 con 0x7fa2fc0601e0 2024-02-10T09:03:18.852 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:18.850+0000 7fa3015fe700 1 --2- 172.21.15.155:0/49137962 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2fc0601e0 0x7fa2fc106600 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7fa2ec005010 tx=0x7fa2ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:03:18.852 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:18.851+0000 7fa2f1ffb700 1 -- 172.21.15.155:0/49137962 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa2ec009320 con 0x7fa2fc0601e0 2024-02-10T09:03:18.852 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:18.851+0000 7fa302600700 1 -- 172.21.15.155:0/49137962 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2fc106cf0 con 0x7fa2fc0601e0 2024-02-10T09:03:18.852 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:18.851+0000 7fa2f1ffb700 1 -- 172.21.15.155:0/49137962 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa2ec013380 con 0x7fa2fc0601e0 2024-02-10T09:03:18.852 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:18.851+0000 7fa2f1ffb700 1 -- 172.21.15.155:0/49137962 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa2ec014e50 con 0x7fa2fc0601e0 2024-02-10T09:03:18.852 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:18.851+0000 7fa302600700 1 -- 172.21.15.155:0/49137962 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2fc107180 con 0x7fa2fc0601e0 2024-02-10T09:03:18.853 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:18.852+0000 7fa302600700 1 -- 172.21.15.155:0/49137962 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2fc045ad0 con 0x7fa2fc0601e0 2024-02-10T09:03:18.853 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:18.852+0000 7fa2f1ffb700 1 -- 172.21.15.155:0/49137962 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa2ec0095f0 con 0x7fa2fc0601e0 2024-02-10T09:03:18.853 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:18.852+0000 7fa2f1ffb700 1 --2- 172.21.15.155:0/49137962 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fa2e802ff10 0x7fa2e80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:03:18.853 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:18.852+0000 7fa2f1ffb700 1 -- 172.21.15.155:0/49137962 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa2ec040e70 con 0x7fa2fc0601e0 2024-02-10T09:03:18.853 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:18.852+0000 7fa300dfd700 1 --2- 172.21.15.155:0/49137962 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fa2e802ff10 0x7fa2e80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:03:18.853 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:18.852+0000 7fa300dfd700 1 --2- 172.21.15.155:0/49137962 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fa2e802ff10 0x7fa2e80323d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7fa2f80096a0 tx=0x7fa2f8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:03:18.857 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:18.856+0000 7fa2f1ffb700 1 -- 172.21.15.155:0/49137962 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa2ec014420 con 0x7fa2fc0601e0 2024-02-10T09:03:19.061 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:19.061+0000 7fa302600700 1 -- 172.21.15.155:0/49137962 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa2fc108030 con 0x7fa2fc0601e0 2024-02-10T09:03:19.062 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:19.062+0000 7fa2f1ffb700 1 -- 172.21.15.155:0/49137962 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa2ec010230 con 0x7fa2fc0601e0 2024-02-10T09:03:19.062 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:03:19.062 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:03:19.064 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:19.064+0000 7fa302600700 1 -- 172.21.15.155:0/49137962 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fa2e802ff10 msgr2=0x7fa2e80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:03:19.064 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:19.064+0000 7fa302600700 1 --2- 172.21.15.155:0/49137962 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fa2e802ff10 0x7fa2e80323d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7fa2f80096a0 tx=0x7fa2f8006b40).stop 2024-02-10T09:03:19.064 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:19.064+0000 7fa302600700 1 -- 172.21.15.155:0/49137962 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2fc0601e0 msgr2=0x7fa2fc106600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:03:19.064 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:19.064+0000 7fa302600700 1 --2- 172.21.15.155:0/49137962 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2fc0601e0 0x7fa2fc106600 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7fa2ec005010 tx=0x7fa2ec00b040).stop 2024-02-10T09:03:19.064 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:19.064+0000 7fa302600700 1 -- 172.21.15.155:0/49137962 shutdown_connections 2024-02-10T09:03:19.064 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:19.064+0000 7fa302600700 1 --2- 172.21.15.155:0/49137962 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fa2e802ff10 0x7fa2e80323d0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:03:19.065 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:19.064+0000 7fa302600700 1 --2- 172.21.15.155:0/49137962 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2fc0601e0 0x7fa2fc106600 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:03:19.065 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:19.064+0000 7fa302600700 1 -- 172.21.15.155:0/49137962 >> 172.21.15.155:0/49137962 conn(0x7fa2fc0f09a0 msgr2=0x7fa2fc0f13d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:03:19.065 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:19.064+0000 7fa302600700 1 -- 172.21.15.155:0/49137962 shutdown_connections 2024-02-10T09:03:19.065 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:19.064+0000 7fa302600700 1 -- 172.21.15.155:0/49137962 wait complete. 2024-02-10T09:03:19.065 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:03:19.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:19 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:03:18.997450+0000 mgr.smithi077.dxntvw (mgr.14162) 137 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:03:19.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:19 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:03:19.062543+0000 mon.smithi077 (mon.0) 184 : audit [DBG] from='client.? 172.21.15.155:0/49137962' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:03:20.776 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:03:20.776 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:03:21.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:21 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:03:20.997706+0000 mgr.smithi077.dxntvw (mgr.14162) 138 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:03:22.704 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:22.703+0000 7f26ce462700 1 -- 172.21.15.155:0/1561576072 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26c80f5410 msgr2=0x7f26c80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:03:22.705 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:22.703+0000 7f26ce462700 1 --2- 172.21.15.155:0/1561576072 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26c80f5410 0x7f26c80f5830 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f26b0009fb0 tx=0x7f26b0009b30).stop 2024-02-10T09:03:22.705 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:22.703+0000 7f26ce462700 1 -- 172.21.15.155:0/1561576072 shutdown_connections 2024-02-10T09:03:22.705 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:22.703+0000 7f26ce462700 1 --2- 172.21.15.155:0/1561576072 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26c80f5410 0x7f26c80f5830 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:03:22.705 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:22.703+0000 7f26ce462700 1 -- 172.21.15.155:0/1561576072 >> 172.21.15.155:0/1561576072 conn(0x7f26c80f0b70 msgr2=0x7f26c80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:03:22.705 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:22.703+0000 7f26ce462700 1 -- 172.21.15.155:0/1561576072 shutdown_connections 2024-02-10T09:03:22.705 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:22.703+0000 7f26ce462700 1 -- 172.21.15.155:0/1561576072 wait complete. 2024-02-10T09:03:22.705 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:22.703+0000 7f26ce462700 1 Processor -- start 2024-02-10T09:03:22.705 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:22.704+0000 7f26ce462700 1 -- start start 2024-02-10T09:03:22.705 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:22.704+0000 7f26ce462700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26c80f5410 0x7f26c81056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:03:22.705 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:22.704+0000 7f26ce462700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26c8105bd0 con 0x7f26c80f5410 2024-02-10T09:03:22.705 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:22.704+0000 7f26c7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26c80f5410 0x7f26c81056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:03:22.706 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:22.704+0000 7f26c7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26c80f5410 0x7f26c81056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:55116/0 (socket says 172.21.15.155:55116) 2024-02-10T09:03:22.706 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:22.704+0000 7f26c7fff700 1 -- 172.21.15.155:0/3707366704 learned_addr learned my addr 172.21.15.155:0/3707366704 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:03:22.706 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:22.704+0000 7f26c7fff700 1 -- 172.21.15.155:0/3707366704 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26b0005040 con 0x7f26c80f5410 2024-02-10T09:03:22.706 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:22.704+0000 7f26c7fff700 1 --2- 172.21.15.155:0/3707366704 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26c80f5410 0x7f26c81056e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f26b00089f0 tx=0x7f26b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:03:22.706 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:22.705+0000 7f26c4ff9700 1 -- 172.21.15.155:0/3707366704 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f26b0009320 con 0x7f26c80f5410 2024-02-10T09:03:22.706 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:22.705+0000 7f26ce462700 1 -- 172.21.15.155:0/3707366704 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26c8105dd0 con 0x7f26c80f5410 2024-02-10T09:03:22.706 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:22.705+0000 7f26c4ff9700 1 -- 172.21.15.155:0/3707366704 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f26b0013380 con 0x7f26c80f5410 2024-02-10T09:03:22.706 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:22.705+0000 7f26c4ff9700 1 -- 172.21.15.155:0/3707366704 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f26b001ed80 con 0x7f26c80f5410 2024-02-10T09:03:22.707 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:22.705+0000 7f26ce462700 1 -- 172.21.15.155:0/3707366704 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26c8106260 con 0x7f26c80f5410 2024-02-10T09:03:22.707 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:22.706+0000 7f26c4ff9700 1 -- 172.21.15.155:0/3707366704 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f26b0014420 con 0x7f26c80f5410 2024-02-10T09:03:22.707 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:22.706+0000 7f26ce462700 1 -- 172.21.15.155:0/3707366704 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26c8045ad0 con 0x7f26c80f5410 2024-02-10T09:03:22.707 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:22.706+0000 7f26c4ff9700 1 --2- 172.21.15.155:0/3707366704 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f26b402ff60 0x7f26b4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:03:22.708 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:22.706+0000 7f26c4ff9700 1 -- 172.21.15.155:0/3707366704 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f26b00410d0 con 0x7f26c80f5410 2024-02-10T09:03:22.708 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:22.706+0000 7f26c77fe700 1 --2- 172.21.15.155:0/3707366704 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f26b402ff60 0x7f26b4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:03:22.708 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:22.706+0000 7f26c77fe700 1 --2- 172.21.15.155:0/3707366704 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f26b402ff60 0x7f26b4032420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f26b80096a0 tx=0x7f26b8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:03:22.710 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:22.710+0000 7f26c4ff9700 1 -- 172.21.15.155:0/3707366704 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f26b000faf0 con 0x7f26c80f5410 2024-02-10T09:03:22.918 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:22.917+0000 7f26ce462700 1 -- 172.21.15.155:0/3707366704 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f26c8107110 con 0x7f26c80f5410 2024-02-10T09:03:22.919 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:22.918+0000 7f26c4ff9700 1 -- 172.21.15.155:0/3707366704 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f26b000faf0 con 0x7f26c80f5410 2024-02-10T09:03:22.919 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:03:22.919 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:03:22.920 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:22.920+0000 7f26ce462700 1 -- 172.21.15.155:0/3707366704 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f26b402ff60 msgr2=0x7f26b4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:03:22.920 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:22.920+0000 7f26ce462700 1 --2- 172.21.15.155:0/3707366704 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f26b402ff60 0x7f26b4032420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f26b80096a0 tx=0x7f26b8006b40).stop 2024-02-10T09:03:22.920 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:22.920+0000 7f26ce462700 1 -- 172.21.15.155:0/3707366704 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26c80f5410 msgr2=0x7f26c81056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:03:22.920 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:22.920+0000 7f26ce462700 1 --2- 172.21.15.155:0/3707366704 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26c80f5410 0x7f26c81056e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f26b00089f0 tx=0x7f26b000b040).stop 2024-02-10T09:03:22.920 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:22.920+0000 7f26ce462700 1 -- 172.21.15.155:0/3707366704 shutdown_connections 2024-02-10T09:03:22.920 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:22.920+0000 7f26ce462700 1 --2- 172.21.15.155:0/3707366704 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f26b402ff60 0x7f26b4032420 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:03:22.920 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:22.920+0000 7f26ce462700 1 --2- 172.21.15.155:0/3707366704 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f26c80f5410 0x7f26c81056e0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:03:22.920 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:22.920+0000 7f26ce462700 1 -- 172.21.15.155:0/3707366704 >> 172.21.15.155:0/3707366704 conn(0x7f26c80f0b70 msgr2=0x7f26c80f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:03:22.920 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:22.920+0000 7f26ce462700 1 -- 172.21.15.155:0/3707366704 shutdown_connections 2024-02-10T09:03:22.921 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:22.920+0000 7f26ce462700 1 -- 172.21.15.155:0/3707366704 wait complete. 2024-02-10T09:03:22.921 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:03:24.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:23 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:03:22.918225+0000 mon.smithi077 (mon.0) 185 : audit [DBG] from='client.? 172.21.15.155:0/3707366704' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:03:24.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:23 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:03:22.998053+0000 mgr.smithi077.dxntvw (mgr.14162) 139 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:03:24.596 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:03:24.597 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:03:25.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:25 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:03:24.998436+0000 mgr.smithi077.dxntvw (mgr.14162) 140 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:03:26.770 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:26.765+0000 7f01f76d3700 1 -- 172.21.15.155:0/2019623886 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01f0060280 msgr2=0x7f01f00f9010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:03:26.770 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:26.765+0000 7f01f76d3700 1 --2- 172.21.15.155:0/2019623886 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01f0060280 0x7f01f00f9010 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f01e0009fb0 tx=0x7f01e0009b30).stop 2024-02-10T09:03:26.770 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:26.765+0000 7f01f76d3700 1 -- 172.21.15.155:0/2019623886 shutdown_connections 2024-02-10T09:03:26.770 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:26.765+0000 7f01f76d3700 1 --2- 172.21.15.155:0/2019623886 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01f0060280 0x7f01f00f9010 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:03:26.770 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:26.765+0000 7f01f76d3700 1 -- 172.21.15.155:0/2019623886 >> 172.21.15.155:0/2019623886 conn(0x7f01f00f0a90 msgr2=0x7f01f00f2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:03:26.770 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:26.765+0000 7f01f76d3700 1 -- 172.21.15.155:0/2019623886 shutdown_connections 2024-02-10T09:03:26.770 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:26.765+0000 7f01f76d3700 1 -- 172.21.15.155:0/2019623886 wait complete. 2024-02-10T09:03:26.770 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:26.766+0000 7f01f76d3700 1 Processor -- start 2024-02-10T09:03:26.771 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:26.766+0000 7f01f76d3700 1 -- start start 2024-02-10T09:03:26.771 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:26.766+0000 7f01f76d3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01f0060280 0x7f01f00f6a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:03:26.771 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:26.766+0000 7f01f76d3700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01f00f4f10 con 0x7f01f0060280 2024-02-10T09:03:26.771 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:26.766+0000 7f01f546f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01f0060280 0x7f01f00f6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:03:26.771 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:26.767+0000 7f01f546f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01f0060280 0x7f01f00f6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:55128/0 (socket says 172.21.15.155:55128) 2024-02-10T09:03:26.771 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:26.767+0000 7f01f546f700 1 -- 172.21.15.155:0/3398617105 learned_addr learned my addr 172.21.15.155:0/3398617105 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:03:26.771 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:26.767+0000 7f01f546f700 1 -- 172.21.15.155:0/3398617105 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01e0005040 con 0x7f01f0060280 2024-02-10T09:03:26.771 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:26.767+0000 7f01f546f700 1 --2- 172.21.15.155:0/3398617105 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01f0060280 0x7f01f00f6a90 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f01e00089f0 tx=0x7f01e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:03:26.771 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:26.767+0000 7f01e5ffb700 1 -- 172.21.15.155:0/3398617105 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01e0009320 con 0x7f01f0060280 2024-02-10T09:03:26.771 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:26.767+0000 7f01f76d3700 1 -- 172.21.15.155:0/3398617105 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01f00f50b0 con 0x7f01f0060280 2024-02-10T09:03:26.771 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:26.767+0000 7f01e5ffb700 1 -- 172.21.15.155:0/3398617105 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f01e0013380 con 0x7f01f0060280 2024-02-10T09:03:26.772 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:26.767+0000 7f01e5ffb700 1 -- 172.21.15.155:0/3398617105 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01e0014e20 con 0x7f01f0060280 2024-02-10T09:03:26.772 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:26.767+0000 7f01f76d3700 1 -- 172.21.15.155:0/3398617105 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01f00f5540 con 0x7f01f0060280 2024-02-10T09:03:26.772 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:26.768+0000 7f01e5ffb700 1 -- 172.21.15.155:0/3398617105 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f01e00095f0 con 0x7f01f0060280 2024-02-10T09:03:26.772 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:26.768+0000 7f01f76d3700 1 -- 172.21.15.155:0/3398617105 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01f0045ad0 con 0x7f01f0060280 2024-02-10T09:03:26.772 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:26.768+0000 7f01e5ffb700 1 --2- 172.21.15.155:0/3398617105 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f01dc02ff10 0x7f01dc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:03:26.772 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:26.768+0000 7f01e5ffb700 1 -- 172.21.15.155:0/3398617105 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f01e0040e90 con 0x7f01f0060280 2024-02-10T09:03:26.772 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:26.769+0000 7f01f4c6e700 1 --2- 172.21.15.155:0/3398617105 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f01dc02ff10 0x7f01dc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:03:26.772 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:26.769+0000 7f01f4c6e700 1 --2- 172.21.15.155:0/3398617105 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f01dc02ff10 0x7f01dc0323d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f01ec0096a0 tx=0x7f01ec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:03:26.774 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:26.773+0000 7f01e5ffb700 1 -- 172.21.15.155:0/3398617105 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f01e0014420 con 0x7f01f0060280 2024-02-10T09:03:26.983 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:26.982+0000 7f01f76d3700 1 -- 172.21.15.155:0/3398617105 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f01f005ea50 con 0x7f01f0060280 2024-02-10T09:03:26.983 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:26.982+0000 7f01e5ffb700 1 -- 172.21.15.155:0/3398617105 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f01e001d980 con 0x7f01f0060280 2024-02-10T09:03:26.984 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:03:26.984 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:03:26.985 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:26.984+0000 7f01f76d3700 1 -- 172.21.15.155:0/3398617105 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f01dc02ff10 msgr2=0x7f01dc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:03:26.985 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:26.984+0000 7f01f76d3700 1 --2- 172.21.15.155:0/3398617105 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f01dc02ff10 0x7f01dc0323d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f01ec0096a0 tx=0x7f01ec006b40).stop 2024-02-10T09:03:26.985 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:26.984+0000 7f01f76d3700 1 -- 172.21.15.155:0/3398617105 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01f0060280 msgr2=0x7f01f00f6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:03:26.985 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:26.984+0000 7f01f76d3700 1 --2- 172.21.15.155:0/3398617105 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01f0060280 0x7f01f00f6a90 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f01e00089f0 tx=0x7f01e000b040).stop 2024-02-10T09:03:26.986 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:26.984+0000 7f01f76d3700 1 -- 172.21.15.155:0/3398617105 shutdown_connections 2024-02-10T09:03:26.986 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:26.984+0000 7f01f76d3700 1 --2- 172.21.15.155:0/3398617105 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f01dc02ff10 0x7f01dc0323d0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:03:26.986 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:26.984+0000 7f01f76d3700 1 --2- 172.21.15.155:0/3398617105 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01f0060280 0x7f01f00f6a90 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:03:26.986 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:26.984+0000 7f01f76d3700 1 -- 172.21.15.155:0/3398617105 >> 172.21.15.155:0/3398617105 conn(0x7f01f00f0a90 msgr2=0x7f01f00f14a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:03:26.986 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:26.984+0000 7f01f76d3700 1 -- 172.21.15.155:0/3398617105 shutdown_connections 2024-02-10T09:03:26.986 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:26.984+0000 7f01f76d3700 1 -- 172.21.15.155:0/3398617105 wait complete. 2024-02-10T09:03:26.986 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:03:27.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:27 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:03:26.983870+0000 mon.smithi077 (mon.0) 186 : audit [DBG] from='client.? 172.21.15.155:0/3398617105' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:03:27.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:27 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:03:26.998711+0000 mgr.smithi077.dxntvw (mgr.14162) 141 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:03:28.718 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:03:28.718 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:03:29.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:29 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:03:28.999005+0000 mgr.smithi077.dxntvw (mgr.14162) 142 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:03:30.681 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:30.680+0000 7f828d7c0700 1 -- 172.21.15.155:0/2059470533 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82880f78e0 msgr2=0x7f82880f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:03:30.681 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:30.680+0000 7f828d7c0700 1 --2- 172.21.15.155:0/2059470533 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82880f78e0 0x7f82880f7d00 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f8270009fb0 tx=0x7f8270009b30).stop 2024-02-10T09:03:30.681 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:30.680+0000 7f828d7c0700 1 -- 172.21.15.155:0/2059470533 shutdown_connections 2024-02-10T09:03:30.681 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:30.680+0000 7f828d7c0700 1 --2- 172.21.15.155:0/2059470533 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82880f78e0 0x7f82880f7d00 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:03:30.681 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:30.680+0000 7f828d7c0700 1 -- 172.21.15.155:0/2059470533 >> 172.21.15.155:0/2059470533 conn(0x7f82880f0120 msgr2=0x7f82880f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:03:30.682 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:30.680+0000 7f828d7c0700 1 -- 172.21.15.155:0/2059470533 shutdown_connections 2024-02-10T09:03:30.682 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:30.680+0000 7f828d7c0700 1 -- 172.21.15.155:0/2059470533 wait complete. 2024-02-10T09:03:30.682 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:30.680+0000 7f828d7c0700 1 Processor -- start 2024-02-10T09:03:30.682 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:30.680+0000 7f828d7c0700 1 -- start start 2024-02-10T09:03:30.683 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:30.681+0000 7f828d7c0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82880f78e0 0x7f8288106600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:03:30.683 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:30.681+0000 7f828d7c0700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8288106af0 con 0x7f82880f78e0 2024-02-10T09:03:30.683 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:30.681+0000 7f8287fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82880f78e0 0x7f8288106600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:03:30.683 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:30.681+0000 7f8287fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82880f78e0 0x7f8288106600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:55148/0 (socket says 172.21.15.155:55148) 2024-02-10T09:03:30.683 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:30.681+0000 7f8287fff700 1 -- 172.21.15.155:0/225506995 learned_addr learned my addr 172.21.15.155:0/225506995 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:03:30.683 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:30.681+0000 7f8287fff700 1 -- 172.21.15.155:0/225506995 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8270005040 con 0x7f82880f78e0 2024-02-10T09:03:30.683 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:30.681+0000 7f8287fff700 1 --2- 172.21.15.155:0/225506995 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82880f78e0 0x7f8288106600 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f82700089f0 tx=0x7f827000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:03:30.683 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:30.681+0000 7f8284ff9700 1 -- 172.21.15.155:0/225506995 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8270009320 con 0x7f82880f78e0 2024-02-10T09:03:30.684 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:30.682+0000 7f828d7c0700 1 -- 172.21.15.155:0/225506995 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8288106cf0 con 0x7f82880f78e0 2024-02-10T09:03:30.684 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:30.682+0000 7f828d7c0700 1 -- 172.21.15.155:0/225506995 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8288107180 con 0x7f82880f78e0 2024-02-10T09:03:30.684 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:30.682+0000 7f8284ff9700 1 -- 172.21.15.155:0/225506995 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8270013380 con 0x7f82880f78e0 2024-02-10T09:03:30.684 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:30.682+0000 7f8284ff9700 1 -- 172.21.15.155:0/225506995 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8270014e40 con 0x7f82880f78e0 2024-02-10T09:03:30.685 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:30.682+0000 7f8284ff9700 1 -- 172.21.15.155:0/225506995 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8270014420 con 0x7f82880f78e0 2024-02-10T09:03:30.685 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:30.682+0000 7f828d7c0700 1 -- 172.21.15.155:0/225506995 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8288045ad0 con 0x7f82880f78e0 2024-02-10T09:03:30.685 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:30.683+0000 7f8284ff9700 1 --2- 172.21.15.155:0/225506995 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f827402ff10 0x7f82740323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:03:30.685 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:30.683+0000 7f8284ff9700 1 -- 172.21.15.155:0/225506995 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8270041060 con 0x7f82880f78e0 2024-02-10T09:03:30.685 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:30.683+0000 7f82877fe700 1 --2- 172.21.15.155:0/225506995 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f827402ff10 0x7f82740323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:03:30.685 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:30.683+0000 7f82877fe700 1 --2- 172.21.15.155:0/225506995 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f827402ff10 0x7f82740323d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f82780096a0 tx=0x7f8278006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:03:30.688 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:30.687+0000 7f8284ff9700 1 -- 172.21.15.155:0/225506995 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f827006d2b0 con 0x7f82880f78e0 2024-02-10T09:03:30.892 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:30.891+0000 7f828d7c0700 1 -- 172.21.15.155:0/225506995 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8288107fc0 con 0x7f82880f78e0 2024-02-10T09:03:30.892 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:30.891+0000 7f8284ff9700 1 -- 172.21.15.155:0/225506995 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8270009680 con 0x7f82880f78e0 2024-02-10T09:03:30.892 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:03:30.893 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:03:30.894 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:30.893+0000 7f828d7c0700 1 -- 172.21.15.155:0/225506995 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f827402ff10 msgr2=0x7f82740323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:03:30.894 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:30.893+0000 7f828d7c0700 1 --2- 172.21.15.155:0/225506995 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f827402ff10 0x7f82740323d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f82780096a0 tx=0x7f8278006b40).stop 2024-02-10T09:03:30.894 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:30.893+0000 7f828d7c0700 1 -- 172.21.15.155:0/225506995 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82880f78e0 msgr2=0x7f8288106600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:03:30.894 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:30.893+0000 7f828d7c0700 1 --2- 172.21.15.155:0/225506995 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82880f78e0 0x7f8288106600 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f82700089f0 tx=0x7f827000b040).stop 2024-02-10T09:03:30.894 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:30.893+0000 7f828d7c0700 1 -- 172.21.15.155:0/225506995 shutdown_connections 2024-02-10T09:03:30.894 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:30.893+0000 7f828d7c0700 1 --2- 172.21.15.155:0/225506995 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f827402ff10 0x7f82740323d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:03:30.894 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:30.893+0000 7f828d7c0700 1 --2- 172.21.15.155:0/225506995 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82880f78e0 0x7f8288106600 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:03:30.894 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:30.893+0000 7f828d7c0700 1 -- 172.21.15.155:0/225506995 >> 172.21.15.155:0/225506995 conn(0x7f82880f0120 msgr2=0x7f82880f0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:03:30.894 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:30.893+0000 7f828d7c0700 1 -- 172.21.15.155:0/225506995 shutdown_connections 2024-02-10T09:03:30.894 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:30.893+0000 7f828d7c0700 1 -- 172.21.15.155:0/225506995 wait complete. 2024-02-10T09:03:30.895 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:03:31.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:30 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:03:30.892707+0000 mon.smithi077 (mon.0) 187 : audit [DBG] from='client.? 172.21.15.155:0/225506995' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:03:32.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:31 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:03:30.999358+0000 mgr.smithi077.dxntvw (mgr.14162) 143 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:03:32.677 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:03:32.678 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:03:34.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:33 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:03:32.999610+0000 mgr.smithi077.dxntvw (mgr.14162) 144 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:03:34.755 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:34.748+0000 7f07ffd09700 1 -- 172.21.15.155:0/3764767408 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07f80f5410 msgr2=0x7f07f80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:03:34.755 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:34.748+0000 7f07ffd09700 1 --2- 172.21.15.155:0/3764767408 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07f80f5410 0x7f07f80f5830 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7f07e8009fb0 tx=0x7f07e8009b30).stop 2024-02-10T09:03:34.755 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:34.748+0000 7f07ffd09700 1 -- 172.21.15.155:0/3764767408 shutdown_connections 2024-02-10T09:03:34.755 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:34.748+0000 7f07ffd09700 1 --2- 172.21.15.155:0/3764767408 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07f80f5410 0x7f07f80f5830 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:03:34.756 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:34.748+0000 7f07ffd09700 1 -- 172.21.15.155:0/3764767408 >> 172.21.15.155:0/3764767408 conn(0x7f07f80f0b70 msgr2=0x7f07f80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:03:34.756 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:34.748+0000 7f07ffd09700 1 -- 172.21.15.155:0/3764767408 shutdown_connections 2024-02-10T09:03:34.756 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:34.749+0000 7f07ffd09700 1 -- 172.21.15.155:0/3764767408 wait complete. 2024-02-10T09:03:34.756 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:34.749+0000 7f07ffd09700 1 Processor -- start 2024-02-10T09:03:34.756 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:34.749+0000 7f07ffd09700 1 -- start start 2024-02-10T09:03:34.756 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:34.749+0000 7f07ffd09700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07f80f5410 0x7f07f80fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:03:34.756 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:34.749+0000 7f07ffd09700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07f80fb960 con 0x7f07f80f5410 2024-02-10T09:03:34.756 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:34.749+0000 7f07fdaa5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07f80f5410 0x7f07f80fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:03:34.756 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:34.749+0000 7f07fdaa5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07f80f5410 0x7f07f80fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:59524/0 (socket says 172.21.15.155:59524) 2024-02-10T09:03:34.756 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:34.749+0000 7f07fdaa5700 1 -- 172.21.15.155:0/2146499184 learned_addr learned my addr 172.21.15.155:0/2146499184 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:03:34.756 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:34.750+0000 7f07fdaa5700 1 -- 172.21.15.155:0/2146499184 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07e8005040 con 0x7f07f80f5410 2024-02-10T09:03:34.756 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:34.750+0000 7f07fdaa5700 1 --2- 172.21.15.155:0/2146499184 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07f80f5410 0x7f07f80fd4e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f07e8005010 tx=0x7f07e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:03:34.756 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:34.750+0000 7f07ee7fc700 1 -- 172.21.15.155:0/2146499184 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07e8009320 con 0x7f07f80f5410 2024-02-10T09:03:34.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:34.750+0000 7f07ffd09700 1 -- 172.21.15.155:0/2146499184 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07f80fbb00 con 0x7f07f80f5410 2024-02-10T09:03:34.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:34.750+0000 7f07ee7fc700 1 -- 172.21.15.155:0/2146499184 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f07e8013380 con 0x7f07f80f5410 2024-02-10T09:03:34.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:34.750+0000 7f07ee7fc700 1 -- 172.21.15.155:0/2146499184 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07e8014e50 con 0x7f07f80f5410 2024-02-10T09:03:34.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:34.750+0000 7f07ffd09700 1 -- 172.21.15.155:0/2146499184 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07f80fbf90 con 0x7f07f80f5410 2024-02-10T09:03:34.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:34.751+0000 7f07ee7fc700 1 -- 172.21.15.155:0/2146499184 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f07e80095f0 con 0x7f07f80f5410 2024-02-10T09:03:34.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:34.751+0000 7f07ffd09700 1 -- 172.21.15.155:0/2146499184 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07f8059100 con 0x7f07f80f5410 2024-02-10T09:03:34.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:34.751+0000 7f07ee7fc700 1 --2- 172.21.15.155:0/2146499184 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f07e402ff10 0x7f07e40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:03:34.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:34.751+0000 7f07ee7fc700 1 -- 172.21.15.155:0/2146499184 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f07e8040e70 con 0x7f07f80f5410 2024-02-10T09:03:34.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:34.752+0000 7f07fd2a4700 1 --2- 172.21.15.155:0/2146499184 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f07e402ff10 0x7f07e40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:03:34.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:34.752+0000 7f07fd2a4700 1 --2- 172.21.15.155:0/2146499184 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f07e402ff10 0x7f07e40323d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f07f40096a0 tx=0x7f07f4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:03:34.758 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:34.756+0000 7f07ee7fc700 1 -- 172.21.15.155:0/2146499184 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f07e8014420 con 0x7f07f80f5410 2024-02-10T09:03:34.963 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:34.962+0000 7f07ffd09700 1 -- 172.21.15.155:0/2146499184 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f07f8045ad0 con 0x7f07f80f5410 2024-02-10T09:03:34.964 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:34.962+0000 7f07ee7fc700 1 -- 172.21.15.155:0/2146499184 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f07e8010230 con 0x7f07f80f5410 2024-02-10T09:03:34.964 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:03:34.964 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:03:34.966 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:34.964+0000 7f07ffd09700 1 -- 172.21.15.155:0/2146499184 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f07e402ff10 msgr2=0x7f07e40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:03:34.966 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:34.964+0000 7f07ffd09700 1 --2- 172.21.15.155:0/2146499184 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f07e402ff10 0x7f07e40323d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f07f40096a0 tx=0x7f07f4006b40).stop 2024-02-10T09:03:34.966 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:34.964+0000 7f07ffd09700 1 -- 172.21.15.155:0/2146499184 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07f80f5410 msgr2=0x7f07f80fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:03:34.966 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:34.964+0000 7f07ffd09700 1 --2- 172.21.15.155:0/2146499184 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07f80f5410 0x7f07f80fd4e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f07e8005010 tx=0x7f07e800b040).stop 2024-02-10T09:03:34.966 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:34.964+0000 7f07ffd09700 1 -- 172.21.15.155:0/2146499184 shutdown_connections 2024-02-10T09:03:34.966 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:34.964+0000 7f07ffd09700 1 --2- 172.21.15.155:0/2146499184 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f07e402ff10 0x7f07e40323d0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:03:34.966 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:34.964+0000 7f07ffd09700 1 --2- 172.21.15.155:0/2146499184 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07f80f5410 0x7f07f80fd4e0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:03:34.966 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:34.964+0000 7f07ffd09700 1 -- 172.21.15.155:0/2146499184 >> 172.21.15.155:0/2146499184 conn(0x7f07f80f0b70 msgr2=0x7f07f80f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:03:34.966 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:34.964+0000 7f07ffd09700 1 -- 172.21.15.155:0/2146499184 shutdown_connections 2024-02-10T09:03:34.966 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:34.964+0000 7f07ffd09700 1 -- 172.21.15.155:0/2146499184 wait complete. 2024-02-10T09:03:34.967 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:03:35.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:35 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:03:34.964061+0000 mon.smithi077 (mon.0) 188 : audit [DBG] from='client.? 172.21.15.155:0/2146499184' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:03:36.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:36 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:03:34.999960+0000 mgr.smithi077.dxntvw (mgr.14162) 145 : cluster [DBG] pgmap v138: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:03:36.644 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:03:36.644 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:03:37.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:37 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:03:37.000251+0000 mgr.smithi077.dxntvw (mgr.14162) 146 : cluster [DBG] pgmap v139: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:03:38.629 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:38.627+0000 7f18d99d0700 1 -- 172.21.15.155:0/993635301 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18d40f62a0 msgr2=0x7f18d40f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:03:38.629 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:38.627+0000 7f18d99d0700 1 --2- 172.21.15.155:0/993635301 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18d40f62a0 0x7f18d40f66c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f18c8009fb0 tx=0x7f18c8009b30).stop 2024-02-10T09:03:38.629 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:38.627+0000 7f18d99d0700 1 -- 172.21.15.155:0/993635301 shutdown_connections 2024-02-10T09:03:38.629 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:38.627+0000 7f18d99d0700 1 --2- 172.21.15.155:0/993635301 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18d40f62a0 0x7f18d40f66c0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:03:38.629 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:38.627+0000 7f18d99d0700 1 -- 172.21.15.155:0/993635301 >> 172.21.15.155:0/993635301 conn(0x7f18d40f1ac0 msgr2=0x7f18d40f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:03:38.629 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:38.627+0000 7f18d99d0700 1 -- 172.21.15.155:0/993635301 shutdown_connections 2024-02-10T09:03:38.629 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:38.627+0000 7f18d99d0700 1 -- 172.21.15.155:0/993635301 wait complete. 2024-02-10T09:03:38.629 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:38.627+0000 7f18d99d0700 1 Processor -- start 2024-02-10T09:03:38.629 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:38.627+0000 7f18d99d0700 1 -- start start 2024-02-10T09:03:38.630 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:38.628+0000 7f18d99d0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18d40f62a0 0x7f18d406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:03:38.630 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:38.628+0000 7f18d99d0700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18d4069e20 con 0x7f18d40f62a0 2024-02-10T09:03:38.631 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:38.628+0000 7f18d89ce700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18d40f62a0 0x7f18d406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:03:38.631 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:38.628+0000 7f18d89ce700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18d40f62a0 0x7f18d406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:59542/0 (socket says 172.21.15.155:59542) 2024-02-10T09:03:38.632 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:38.628+0000 7f18d89ce700 1 -- 172.21.15.155:0/478527842 learned_addr learned my addr 172.21.15.155:0/478527842 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:03:38.632 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:38.628+0000 7f18d89ce700 1 -- 172.21.15.155:0/478527842 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18c8005040 con 0x7f18d40f62a0 2024-02-10T09:03:38.632 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:38.629+0000 7f18d89ce700 1 --2- 172.21.15.155:0/478527842 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18d40f62a0 0x7f18d406b9a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f18c8009830 tx=0x7f18c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:03:38.632 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:38.629+0000 7f18d17fa700 1 -- 172.21.15.155:0/478527842 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f18c8009320 con 0x7f18d40f62a0 2024-02-10T09:03:38.632 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:38.629+0000 7f18d99d0700 1 -- 172.21.15.155:0/478527842 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18d4069fc0 con 0x7f18d40f62a0 2024-02-10T09:03:38.632 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:38.629+0000 7f18d17fa700 1 -- 172.21.15.155:0/478527842 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f18c8013380 con 0x7f18d40f62a0 2024-02-10T09:03:38.632 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:38.629+0000 7f18d17fa700 1 -- 172.21.15.155:0/478527842 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f18c8014420 con 0x7f18d40f62a0 2024-02-10T09:03:38.632 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:38.629+0000 7f18d99d0700 1 -- 172.21.15.155:0/478527842 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18d406a450 con 0x7f18d40f62a0 2024-02-10T09:03:38.632 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:38.630+0000 7f18d17fa700 1 -- 172.21.15.155:0/478527842 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f18c801e420 con 0x7f18d40f62a0 2024-02-10T09:03:38.632 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:38.630+0000 7f18d99d0700 1 -- 172.21.15.155:0/478527842 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18d4059100 con 0x7f18d40f62a0 2024-02-10T09:03:38.632 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:38.630+0000 7f18d17fa700 1 --2- 172.21.15.155:0/478527842 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f18c402ff60 0x7f18c4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:03:38.632 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:38.630+0000 7f18d17fa700 1 -- 172.21.15.155:0/478527842 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f18c8040e20 con 0x7f18d40f62a0 2024-02-10T09:03:38.633 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:38.630+0000 7f18d3fff700 1 --2- 172.21.15.155:0/478527842 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f18c402ff60 0x7f18c4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:03:38.633 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:38.630+0000 7f18d3fff700 1 --2- 172.21.15.155:0/478527842 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f18c402ff60 0x7f18c4032420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f18c00096a0 tx=0x7f18c0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:03:38.635 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:38.635+0000 7f18d17fa700 1 -- 172.21.15.155:0/478527842 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f18c801c970 con 0x7f18d40f62a0 2024-02-10T09:03:38.839 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:38.837+0000 7f18d99d0700 1 -- 172.21.15.155:0/478527842 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f18d4045ad0 con 0x7f18d40f62a0 2024-02-10T09:03:38.839 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:38.838+0000 7f18d17fa700 1 -- 172.21.15.155:0/478527842 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f18c8004490 con 0x7f18d40f62a0 2024-02-10T09:03:38.839 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:03:38.840 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:03:38.841 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:38.840+0000 7f18d99d0700 1 -- 172.21.15.155:0/478527842 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f18c402ff60 msgr2=0x7f18c4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:03:38.841 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:38.840+0000 7f18d99d0700 1 --2- 172.21.15.155:0/478527842 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f18c402ff60 0x7f18c4032420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f18c00096a0 tx=0x7f18c0006b40).stop 2024-02-10T09:03:38.841 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:38.840+0000 7f18d99d0700 1 -- 172.21.15.155:0/478527842 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18d40f62a0 msgr2=0x7f18d406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:03:38.841 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:38.840+0000 7f18d99d0700 1 --2- 172.21.15.155:0/478527842 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18d40f62a0 0x7f18d406b9a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f18c8009830 tx=0x7f18c800b040).stop 2024-02-10T09:03:38.841 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:38.840+0000 7f18d99d0700 1 -- 172.21.15.155:0/478527842 shutdown_connections 2024-02-10T09:03:38.841 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:38.840+0000 7f18d99d0700 1 --2- 172.21.15.155:0/478527842 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f18c402ff60 0x7f18c4032420 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:03:38.841 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:38.840+0000 7f18d99d0700 1 --2- 172.21.15.155:0/478527842 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18d40f62a0 0x7f18d406b9a0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:03:38.841 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:38.840+0000 7f18d99d0700 1 -- 172.21.15.155:0/478527842 >> 172.21.15.155:0/478527842 conn(0x7f18d40f1ac0 msgr2=0x7f18d40f24c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:03:38.842 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:38.840+0000 7f18d99d0700 1 -- 172.21.15.155:0/478527842 shutdown_connections 2024-02-10T09:03:38.842 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:38.840+0000 7f18d99d0700 1 -- 172.21.15.155:0/478527842 wait complete. 2024-02-10T09:03:38.842 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:03:39.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:38 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:03:38.839664+0000 mon.smithi077 (mon.0) 189 : audit [DBG] from='client.? 172.21.15.155:0/478527842' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:03:40.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:39 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:03:39.000553+0000 mgr.smithi077.dxntvw (mgr.14162) 147 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:03:40.666 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:03:40.666 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:03:41.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:41 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:03:41.000802+0000 mgr.smithi077.dxntvw (mgr.14162) 148 : cluster [DBG] pgmap v141: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:03:42.490 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:42.489+0000 7f1ce2971700 1 -- 172.21.15.155:0/36943709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1cdc0f5410 msgr2=0x7f1cdc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:03:42.491 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:42.489+0000 7f1ce2971700 1 --2- 172.21.15.155:0/36943709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1cdc0f5410 0x7f1cdc0f5830 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f1ccc009fb0 tx=0x7f1ccc009b30).stop 2024-02-10T09:03:42.491 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:42.489+0000 7f1ce2971700 1 -- 172.21.15.155:0/36943709 shutdown_connections 2024-02-10T09:03:42.491 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:42.489+0000 7f1ce2971700 1 --2- 172.21.15.155:0/36943709 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1cdc0f5410 0x7f1cdc0f5830 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:03:42.491 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:42.489+0000 7f1ce2971700 1 -- 172.21.15.155:0/36943709 >> 172.21.15.155:0/36943709 conn(0x7f1cdc0f0b70 msgr2=0x7f1cdc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:03:42.491 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:42.489+0000 7f1ce2971700 1 -- 172.21.15.155:0/36943709 shutdown_connections 2024-02-10T09:03:42.491 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:42.489+0000 7f1ce2971700 1 -- 172.21.15.155:0/36943709 wait complete. 2024-02-10T09:03:42.491 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:42.490+0000 7f1ce2971700 1 Processor -- start 2024-02-10T09:03:42.492 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:42.490+0000 7f1ce2971700 1 -- start start 2024-02-10T09:03:42.493 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:42.490+0000 7f1ce2971700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1cdc0f5410 0x7f1cdc1056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:03:42.493 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:42.490+0000 7f1ce2971700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cdc105bd0 con 0x7f1cdc0f5410 2024-02-10T09:03:42.493 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:42.490+0000 7f1cdbfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1cdc0f5410 0x7f1cdc1056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:03:42.493 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:42.491+0000 7f1cdbfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1cdc0f5410 0x7f1cdc1056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:58846/0 (socket says 172.21.15.155:58846) 2024-02-10T09:03:42.493 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:42.491+0000 7f1cdbfff700 1 -- 172.21.15.155:0/1210214505 learned_addr learned my addr 172.21.15.155:0/1210214505 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:03:42.493 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:42.491+0000 7f1cdbfff700 1 -- 172.21.15.155:0/1210214505 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ccc005040 con 0x7f1cdc0f5410 2024-02-10T09:03:42.493 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:42.491+0000 7f1cdbfff700 1 --2- 172.21.15.155:0/1210214505 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1cdc0f5410 0x7f1cdc1056e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f1ccc005010 tx=0x7f1ccc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:03:42.493 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:42.491+0000 7f1cd8ff9700 1 -- 172.21.15.155:0/1210214505 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ccc009320 con 0x7f1cdc0f5410 2024-02-10T09:03:42.493 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:42.491+0000 7f1ce2971700 1 -- 172.21.15.155:0/1210214505 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cdc105dd0 con 0x7f1cdc0f5410 2024-02-10T09:03:42.493 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:42.491+0000 7f1cd8ff9700 1 -- 172.21.15.155:0/1210214505 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1ccc013380 con 0x7f1cdc0f5410 2024-02-10T09:03:42.493 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:42.491+0000 7f1cd8ff9700 1 -- 172.21.15.155:0/1210214505 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ccc014e50 con 0x7f1cdc0f5410 2024-02-10T09:03:42.494 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:42.491+0000 7f1ce2971700 1 -- 172.21.15.155:0/1210214505 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cdc106260 con 0x7f1cdc0f5410 2024-02-10T09:03:42.494 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:42.492+0000 7f1cd8ff9700 1 -- 172.21.15.155:0/1210214505 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1ccc0095f0 con 0x7f1cdc0f5410 2024-02-10T09:03:42.495 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:42.492+0000 7f1ce2971700 1 -- 172.21.15.155:0/1210214505 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cdc045ad0 con 0x7f1cdc0f5410 2024-02-10T09:03:42.495 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:42.492+0000 7f1cd8ff9700 1 --2- 172.21.15.155:0/1210214505 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f1cc402ff10 0x7f1cc40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:03:42.495 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:42.492+0000 7f1cd8ff9700 1 -- 172.21.15.155:0/1210214505 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1ccc040fa0 con 0x7f1cdc0f5410 2024-02-10T09:03:42.495 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:42.492+0000 7f1cdb7fe700 1 --2- 172.21.15.155:0/1210214505 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f1cc402ff10 0x7f1cc40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:03:42.495 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:42.493+0000 7f1cdb7fe700 1 --2- 172.21.15.155:0/1210214505 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f1cc402ff10 0x7f1cc40323d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f1cd00096a0 tx=0x7f1cd0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:03:42.497 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:42.497+0000 7f1cd8ff9700 1 -- 172.21.15.155:0/1210214505 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1ccc014420 con 0x7f1cdc0f5410 2024-02-10T09:03:42.702 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:42.701+0000 7f1ce2971700 1 -- 172.21.15.155:0/1210214505 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1cdc107110 con 0x7f1cdc0f5410 2024-02-10T09:03:42.703 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:42.702+0000 7f1cd8ff9700 1 -- 172.21.15.155:0/1210214505 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1ccc010230 con 0x7f1cdc0f5410 2024-02-10T09:03:42.703 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:03:42.703 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:03:42.704 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:42.703+0000 7f1ce2971700 1 -- 172.21.15.155:0/1210214505 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f1cc402ff10 msgr2=0x7f1cc40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:03:42.704 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:42.703+0000 7f1ce2971700 1 --2- 172.21.15.155:0/1210214505 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f1cc402ff10 0x7f1cc40323d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f1cd00096a0 tx=0x7f1cd0006b40).stop 2024-02-10T09:03:42.705 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:42.703+0000 7f1ce2971700 1 -- 172.21.15.155:0/1210214505 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1cdc0f5410 msgr2=0x7f1cdc1056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:03:42.705 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:42.703+0000 7f1ce2971700 1 --2- 172.21.15.155:0/1210214505 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1cdc0f5410 0x7f1cdc1056e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f1ccc005010 tx=0x7f1ccc00b040).stop 2024-02-10T09:03:42.705 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:42.703+0000 7f1ce2971700 1 -- 172.21.15.155:0/1210214505 shutdown_connections 2024-02-10T09:03:42.705 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:42.703+0000 7f1ce2971700 1 --2- 172.21.15.155:0/1210214505 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f1cc402ff10 0x7f1cc40323d0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:03:42.705 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:42.703+0000 7f1ce2971700 1 --2- 172.21.15.155:0/1210214505 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1cdc0f5410 0x7f1cdc1056e0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:03:42.705 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:42.703+0000 7f1ce2971700 1 -- 172.21.15.155:0/1210214505 >> 172.21.15.155:0/1210214505 conn(0x7f1cdc0f0b70 msgr2=0x7f1cdc0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:03:42.705 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:42.703+0000 7f1ce2971700 1 -- 172.21.15.155:0/1210214505 shutdown_connections 2024-02-10T09:03:42.705 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:42.703+0000 7f1ce2971700 1 -- 172.21.15.155:0/1210214505 wait complete. 2024-02-10T09:03:42.705 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:03:43.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:42 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:03:42.703293+0000 mon.smithi077 (mon.0) 190 : audit [DBG] from='client.? 172.21.15.155:0/1210214505' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:03:44.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:43 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:03:43.001048+0000 mgr.smithi077.dxntvw (mgr.14162) 149 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:03:44.409 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:03:44.410 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:03:45.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:45 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:03:45.001318+0000 mgr.smithi077.dxntvw (mgr.14162) 150 : cluster [DBG] pgmap v143: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:03:46.466 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:46.449+0000 7f131fe5c700 1 -- 172.21.15.155:0/631066654 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13180f6480 msgr2=0x7f13180f68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:03:46.466 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:46.449+0000 7f131fe5c700 1 --2- 172.21.15.155:0/631066654 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13180f6480 0x7f13180f68a0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f1308009fb0 tx=0x7f1308009b30).stop 2024-02-10T09:03:46.467 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:46.449+0000 7f131fe5c700 1 -- 172.21.15.155:0/631066654 shutdown_connections 2024-02-10T09:03:46.467 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:46.449+0000 7f131fe5c700 1 --2- 172.21.15.155:0/631066654 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13180f6480 0x7f13180f68a0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:03:46.467 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:46.449+0000 7f131fe5c700 1 -- 172.21.15.155:0/631066654 >> 172.21.15.155:0/631066654 conn(0x7f13180f1c00 msgr2=0x7f13180f4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:03:46.467 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:46.449+0000 7f131fe5c700 1 -- 172.21.15.155:0/631066654 shutdown_connections 2024-02-10T09:03:46.467 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:46.449+0000 7f131fe5c700 1 -- 172.21.15.155:0/631066654 wait complete. 2024-02-10T09:03:46.467 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:46.449+0000 7f131fe5c700 1 Processor -- start 2024-02-10T09:03:46.467 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:46.449+0000 7f131fe5c700 1 -- start start 2024-02-10T09:03:46.467 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:46.449+0000 7f131fe5c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13180f6480 0x7f13181067f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:03:46.467 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:46.449+0000 7f131fe5c700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1318106ce0 con 0x7f13180f6480 2024-02-10T09:03:46.467 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:46.450+0000 7f131dbf8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13180f6480 0x7f13181067f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:03:46.467 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:46.450+0000 7f131dbf8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13180f6480 0x7f13181067f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:58866/0 (socket says 172.21.15.155:58866) 2024-02-10T09:03:46.467 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:46.450+0000 7f131dbf8700 1 -- 172.21.15.155:0/3579009636 learned_addr learned my addr 172.21.15.155:0/3579009636 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:03:46.467 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:46.450+0000 7f131dbf8700 1 -- 172.21.15.155:0/3579009636 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1308005040 con 0x7f13180f6480 2024-02-10T09:03:46.467 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:46.450+0000 7f131dbf8700 1 --2- 172.21.15.155:0/3579009636 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13180f6480 0x7f13181067f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f1308005010 tx=0x7f130800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:03:46.468 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:46.450+0000 7f130e7fc700 1 -- 172.21.15.155:0/3579009636 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1308009320 con 0x7f13180f6480 2024-02-10T09:03:46.468 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:46.450+0000 7f131fe5c700 1 -- 172.21.15.155:0/3579009636 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1318106ee0 con 0x7f13180f6480 2024-02-10T09:03:46.468 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:46.450+0000 7f130e7fc700 1 -- 172.21.15.155:0/3579009636 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1308013380 con 0x7f13180f6480 2024-02-10T09:03:46.468 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:46.450+0000 7f130e7fc700 1 -- 172.21.15.155:0/3579009636 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1308014e50 con 0x7f13180f6480 2024-02-10T09:03:46.468 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:46.450+0000 7f131fe5c700 1 -- 172.21.15.155:0/3579009636 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1318107370 con 0x7f13180f6480 2024-02-10T09:03:46.468 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:46.451+0000 7f130e7fc700 1 -- 172.21.15.155:0/3579009636 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f13080095f0 con 0x7f13180f6480 2024-02-10T09:03:46.468 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:46.451+0000 7f131fe5c700 1 -- 172.21.15.155:0/3579009636 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1318045ad0 con 0x7f13180f6480 2024-02-10T09:03:46.468 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:46.451+0000 7f130e7fc700 1 --2- 172.21.15.155:0/3579009636 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f130402ff10 0x7f13040323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:03:46.468 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:46.451+0000 7f130e7fc700 1 -- 172.21.15.155:0/3579009636 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1308040e90 con 0x7f13180f6480 2024-02-10T09:03:46.468 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:46.451+0000 7f131d3f7700 1 --2- 172.21.15.155:0/3579009636 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f130402ff10 0x7f13040323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:03:46.468 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:46.452+0000 7f131d3f7700 1 --2- 172.21.15.155:0/3579009636 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f130402ff10 0x7f13040323d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f13140096a0 tx=0x7f1314006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:03:46.468 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:46.456+0000 7f130e7fc700 1 -- 172.21.15.155:0/3579009636 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1308014420 con 0x7f13180f6480 2024-02-10T09:03:46.665 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:46.664+0000 7f131fe5c700 1 -- 172.21.15.155:0/3579009636 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f13180fc090 con 0x7f13180f6480 2024-02-10T09:03:46.666 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:46.665+0000 7f130e7fc700 1 -- 172.21.15.155:0/3579009636 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1308010230 con 0x7f13180f6480 2024-02-10T09:03:46.666 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:03:46.666 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:03:46.668 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:46.667+0000 7f131fe5c700 1 -- 172.21.15.155:0/3579009636 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f130402ff10 msgr2=0x7f13040323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:03:46.668 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:46.667+0000 7f131fe5c700 1 --2- 172.21.15.155:0/3579009636 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f130402ff10 0x7f13040323d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f13140096a0 tx=0x7f1314006b40).stop 2024-02-10T09:03:46.668 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:46.667+0000 7f131fe5c700 1 -- 172.21.15.155:0/3579009636 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13180f6480 msgr2=0x7f13181067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:03:46.668 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:46.667+0000 7f131fe5c700 1 --2- 172.21.15.155:0/3579009636 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13180f6480 0x7f13181067f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f1308005010 tx=0x7f130800b040).stop 2024-02-10T09:03:46.668 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:46.667+0000 7f131fe5c700 1 -- 172.21.15.155:0/3579009636 shutdown_connections 2024-02-10T09:03:46.668 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:46.667+0000 7f131fe5c700 1 --2- 172.21.15.155:0/3579009636 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f130402ff10 0x7f13040323d0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:03:46.669 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:46.667+0000 7f131fe5c700 1 --2- 172.21.15.155:0/3579009636 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13180f6480 0x7f13181067f0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:03:46.669 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:46.667+0000 7f131fe5c700 1 -- 172.21.15.155:0/3579009636 >> 172.21.15.155:0/3579009636 conn(0x7f13180f1c00 msgr2=0x7f13180f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:03:46.669 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:46.667+0000 7f131fe5c700 1 -- 172.21.15.155:0/3579009636 shutdown_connections 2024-02-10T09:03:46.669 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:46.667+0000 7f131fe5c700 1 -- 172.21.15.155:0/3579009636 wait complete. 2024-02-10T09:03:46.669 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:03:47.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:46 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:03:46.666440+0000 mon.smithi077 (mon.0) 191 : audit [DBG] from='client.? 172.21.15.155:0/3579009636' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:03:48.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:03:47.001599+0000 mgr.smithi077.dxntvw (mgr.14162) 151 : cluster [DBG] pgmap v144: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:03:48.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:03:47.092136+0000 mon.smithi077 (mon.0) 192 : audit [INF] from='mgr.14162 172.21.15.77:0/2029106299' entity='mgr.smithi077.dxntvw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi077.dxntvw/mirror_snapshot_schedule"}]: dispatch 2024-02-10T09:03:48.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:03:47.099548+0000 mon.smithi077 (mon.0) 193 : audit [INF] from='mgr.14162 172.21.15.77:0/2029106299' entity='mgr.smithi077.dxntvw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi077.dxntvw/trash_purge_schedule"}]: dispatch 2024-02-10T09:03:48.372 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:03:48.372 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:03:49.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:49 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:03:49.001932+0000 mgr.smithi077.dxntvw (mgr.14162) 152 : cluster [DBG] pgmap v145: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:03:50.487 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:50.457+0000 7f7b0db79700 1 -- 172.21.15.155:0/1005859359 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b080f62a0 msgr2=0x7f7b080f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:03:50.487 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:50.457+0000 7f7b0db79700 1 --2- 172.21.15.155:0/1005859359 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b080f62a0 0x7f7b080f66c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f7afc009fb0 tx=0x7f7afc009b30).stop 2024-02-10T09:03:50.487 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:50.457+0000 7f7b0db79700 1 -- 172.21.15.155:0/1005859359 shutdown_connections 2024-02-10T09:03:50.487 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:50.457+0000 7f7b0db79700 1 --2- 172.21.15.155:0/1005859359 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b080f62a0 0x7f7b080f66c0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:03:50.487 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:50.457+0000 7f7b0db79700 1 -- 172.21.15.155:0/1005859359 >> 172.21.15.155:0/1005859359 conn(0x7f7b080f1aa0 msgr2=0x7f7b080f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:03:50.487 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:50.457+0000 7f7b0db79700 1 -- 172.21.15.155:0/1005859359 shutdown_connections 2024-02-10T09:03:50.487 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:50.457+0000 7f7b0db79700 1 -- 172.21.15.155:0/1005859359 wait complete. 2024-02-10T09:03:50.488 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:50.458+0000 7f7b0db79700 1 Processor -- start 2024-02-10T09:03:50.488 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:50.458+0000 7f7b0db79700 1 -- start start 2024-02-10T09:03:50.488 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:50.458+0000 7f7b0db79700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b080f62a0 0x7f7b081065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:03:50.488 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:50.458+0000 7f7b0db79700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b08106ac0 con 0x7f7b080f62a0 2024-02-10T09:03:50.488 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:50.458+0000 7f7b0cb77700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b080f62a0 0x7f7b081065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:03:50.488 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:50.458+0000 7f7b0cb77700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b080f62a0 0x7f7b081065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:58884/0 (socket says 172.21.15.155:58884) 2024-02-10T09:03:50.488 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:50.458+0000 7f7b0cb77700 1 -- 172.21.15.155:0/2662099533 learned_addr learned my addr 172.21.15.155:0/2662099533 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:03:50.488 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:50.459+0000 7f7b0cb77700 1 -- 172.21.15.155:0/2662099533 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7afc005040 con 0x7f7b080f62a0 2024-02-10T09:03:50.488 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:50.459+0000 7f7b0cb77700 1 --2- 172.21.15.155:0/2662099533 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b080f62a0 0x7f7b081065d0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f7afc0089f0 tx=0x7f7afc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:03:50.488 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:50.459+0000 7f7b057fa700 1 -- 172.21.15.155:0/2662099533 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7afc009320 con 0x7f7b080f62a0 2024-02-10T09:03:50.488 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:50.459+0000 7f7b0db79700 1 -- 172.21.15.155:0/2662099533 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b08106cc0 con 0x7f7b080f62a0 2024-02-10T09:03:50.489 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:50.459+0000 7f7b057fa700 1 -- 172.21.15.155:0/2662099533 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7afc013380 con 0x7f7b080f62a0 2024-02-10T09:03:50.489 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:50.459+0000 7f7b057fa700 1 -- 172.21.15.155:0/2662099533 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7afc014e20 con 0x7f7b080f62a0 2024-02-10T09:03:50.489 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:50.459+0000 7f7b0db79700 1 -- 172.21.15.155:0/2662099533 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b08107150 con 0x7f7b080f62a0 2024-02-10T09:03:50.489 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:50.460+0000 7f7b057fa700 1 -- 172.21.15.155:0/2662099533 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7afc0095f0 con 0x7f7b080f62a0 2024-02-10T09:03:50.489 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:50.460+0000 7f7b0db79700 1 -- 172.21.15.155:0/2662099533 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b08045ad0 con 0x7f7b080f62a0 2024-02-10T09:03:50.489 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:50.460+0000 7f7b057fa700 1 --2- 172.21.15.155:0/2662099533 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f7af802ff10 0x7f7af80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:03:50.489 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:50.460+0000 7f7b057fa700 1 -- 172.21.15.155:0/2662099533 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7afc040e90 con 0x7f7b080f62a0 2024-02-10T09:03:50.489 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:50.460+0000 7f7b07fff700 1 --2- 172.21.15.155:0/2662099533 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f7af802ff10 0x7f7af80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:03:50.489 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:50.461+0000 7f7b07fff700 1 --2- 172.21.15.155:0/2662099533 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f7af802ff10 0x7f7af80323d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f7af40096a0 tx=0x7f7af4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:03:50.489 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:50.465+0000 7f7b057fa700 1 -- 172.21.15.155:0/2662099533 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7afc014420 con 0x7f7b080f62a0 2024-02-10T09:03:50.674 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:50.672+0000 7f7b0db79700 1 -- 172.21.15.155:0/2662099533 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7b08108000 con 0x7f7b080f62a0 2024-02-10T09:03:50.674 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:50.673+0000 7f7b057fa700 1 -- 172.21.15.155:0/2662099533 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7afc01d980 con 0x7f7b080f62a0 2024-02-10T09:03:50.674 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:03:50.674 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:03:50.676 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:50.674+0000 7f7b0db79700 1 -- 172.21.15.155:0/2662099533 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f7af802ff10 msgr2=0x7f7af80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:03:50.676 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:50.674+0000 7f7b0db79700 1 --2- 172.21.15.155:0/2662099533 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f7af802ff10 0x7f7af80323d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f7af40096a0 tx=0x7f7af4006b40).stop 2024-02-10T09:03:50.676 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:50.674+0000 7f7b0db79700 1 -- 172.21.15.155:0/2662099533 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b080f62a0 msgr2=0x7f7b081065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:03:50.676 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:50.674+0000 7f7b0db79700 1 --2- 172.21.15.155:0/2662099533 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b080f62a0 0x7f7b081065d0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f7afc0089f0 tx=0x7f7afc00b040).stop 2024-02-10T09:03:50.676 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:50.675+0000 7f7b0db79700 1 -- 172.21.15.155:0/2662099533 shutdown_connections 2024-02-10T09:03:50.676 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:50.675+0000 7f7b0db79700 1 --2- 172.21.15.155:0/2662099533 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f7af802ff10 0x7f7af80323d0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:03:50.676 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:50.675+0000 7f7b0db79700 1 --2- 172.21.15.155:0/2662099533 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7b080f62a0 0x7f7b081065d0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:03:50.676 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:50.675+0000 7f7b0db79700 1 -- 172.21.15.155:0/2662099533 >> 172.21.15.155:0/2662099533 conn(0x7f7b080f1aa0 msgr2=0x7f7b080f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:03:50.676 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:50.675+0000 7f7b0db79700 1 -- 172.21.15.155:0/2662099533 shutdown_connections 2024-02-10T09:03:50.676 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:50.675+0000 7f7b0db79700 1 -- 172.21.15.155:0/2662099533 wait complete. 2024-02-10T09:03:50.677 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:03:51.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:50 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:03:50.674425+0000 mon.smithi077 (mon.0) 194 : audit [DBG] from='client.? 172.21.15.155:0/2662099533' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:03:52.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:51 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:03:51.002192+0000 mgr.smithi077.dxntvw (mgr.14162) 153 : cluster [DBG] pgmap v146: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:03:52.490 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:03:52.491 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:03:54.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:53 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:03:53.002503+0000 mgr.smithi077.dxntvw (mgr.14162) 154 : cluster [DBG] pgmap v147: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:03:54.448 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:54.432+0000 7f1d10a3b700 1 -- 172.21.15.155:0/370234390 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d0c060280 msgr2=0x7f1d0c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:03:54.449 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:54.432+0000 7f1d10a3b700 1 --2- 172.21.15.155:0/370234390 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d0c060280 0x7f1d0c0f90f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f1d00009fb0 tx=0x7f1d00009b30).stop 2024-02-10T09:03:54.449 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:54.432+0000 7f1d10a3b700 1 -- 172.21.15.155:0/370234390 shutdown_connections 2024-02-10T09:03:54.449 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:54.432+0000 7f1d10a3b700 1 --2- 172.21.15.155:0/370234390 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d0c060280 0x7f1d0c0f90f0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:03:54.449 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:54.432+0000 7f1d10a3b700 1 -- 172.21.15.155:0/370234390 >> 172.21.15.155:0/370234390 conn(0x7f1d0c0f0b70 msgr2=0x7f1d0c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:03:54.449 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:54.432+0000 7f1d10a3b700 1 -- 172.21.15.155:0/370234390 shutdown_connections 2024-02-10T09:03:54.449 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:54.432+0000 7f1d10a3b700 1 -- 172.21.15.155:0/370234390 wait complete. 2024-02-10T09:03:54.449 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:54.433+0000 7f1d10a3b700 1 Processor -- start 2024-02-10T09:03:54.449 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:54.433+0000 7f1d10a3b700 1 -- start start 2024-02-10T09:03:54.449 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:54.433+0000 7f1d10a3b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d0c060280 0x7f1d0c106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:03:54.449 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:54.433+0000 7f1d10a3b700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d0c106c70 con 0x7f1d0c060280 2024-02-10T09:03:54.449 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:54.433+0000 7f1d0a59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d0c060280 0x7f1d0c106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:03:54.450 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:54.433+0000 7f1d0a59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d0c060280 0x7f1d0c106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:43798/0 (socket says 172.21.15.155:43798) 2024-02-10T09:03:54.450 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:54.433+0000 7f1d0a59c700 1 -- 172.21.15.155:0/3062116245 learned_addr learned my addr 172.21.15.155:0/3062116245 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:03:54.450 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:54.433+0000 7f1d0a59c700 1 -- 172.21.15.155:0/3062116245 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d00005040 con 0x7f1d0c060280 2024-02-10T09:03:54.450 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:54.434+0000 7f1d0a59c700 1 --2- 172.21.15.155:0/3062116245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d0c060280 0x7f1d0c106780 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f1d000089f0 tx=0x7f1d0000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:03:54.450 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:54.434+0000 7f1cfaffd700 1 -- 172.21.15.155:0/3062116245 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d00009320 con 0x7f1d0c060280 2024-02-10T09:03:54.450 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:54.434+0000 7f1d10a3b700 1 -- 172.21.15.155:0/3062116245 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d0c106e70 con 0x7f1d0c060280 2024-02-10T09:03:54.450 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:54.434+0000 7f1cfaffd700 1 -- 172.21.15.155:0/3062116245 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1d00013380 con 0x7f1d0c060280 2024-02-10T09:03:54.450 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:54.434+0000 7f1cfaffd700 1 -- 172.21.15.155:0/3062116245 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d00014e20 con 0x7f1d0c060280 2024-02-10T09:03:54.450 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:54.434+0000 7f1d10a3b700 1 -- 172.21.15.155:0/3062116245 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d0c107300 con 0x7f1d0c060280 2024-02-10T09:03:54.450 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:54.435+0000 7f1cfaffd700 1 -- 172.21.15.155:0/3062116245 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1d00014420 con 0x7f1d0c060280 2024-02-10T09:03:54.450 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:54.435+0000 7f1d10a3b700 1 -- 172.21.15.155:0/3062116245 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d0c045ad0 con 0x7f1d0c060280 2024-02-10T09:03:54.450 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:54.435+0000 7f1cfaffd700 1 --2- 172.21.15.155:0/3062116245 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f1cf402ff10 0x7f1cf40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:03:54.450 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:54.435+0000 7f1cfaffd700 1 -- 172.21.15.155:0/3062116245 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1d00040fc0 con 0x7f1d0c060280 2024-02-10T09:03:54.450 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:54.435+0000 7f1d09d9b700 1 --2- 172.21.15.155:0/3062116245 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f1cf402ff10 0x7f1cf40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:03:54.451 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:54.435+0000 7f1d09d9b700 1 --2- 172.21.15.155:0/3062116245 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f1cf402ff10 0x7f1cf40323d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f1cfc0096a0 tx=0x7f1cfc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:03:54.451 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:54.440+0000 7f1cfaffd700 1 -- 172.21.15.155:0/3062116245 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1d0006d2b0 con 0x7f1d0c060280 2024-02-10T09:03:54.645 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:54.644+0000 7f1d10a3b700 1 -- 172.21.15.155:0/3062116245 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1d0c1081b0 con 0x7f1d0c060280 2024-02-10T09:03:54.646 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:54.645+0000 7f1cfaffd700 1 -- 172.21.15.155:0/3062116245 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1d000095f0 con 0x7f1d0c060280 2024-02-10T09:03:54.646 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:03:54.646 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:03:54.647 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:54.646+0000 7f1d10a3b700 1 -- 172.21.15.155:0/3062116245 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f1cf402ff10 msgr2=0x7f1cf40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:03:54.647 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:54.646+0000 7f1d10a3b700 1 --2- 172.21.15.155:0/3062116245 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f1cf402ff10 0x7f1cf40323d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f1cfc0096a0 tx=0x7f1cfc006b40).stop 2024-02-10T09:03:54.647 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:54.646+0000 7f1d10a3b700 1 -- 172.21.15.155:0/3062116245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d0c060280 msgr2=0x7f1d0c106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:03:54.648 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:54.646+0000 7f1d10a3b700 1 --2- 172.21.15.155:0/3062116245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d0c060280 0x7f1d0c106780 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f1d000089f0 tx=0x7f1d0000b040).stop 2024-02-10T09:03:54.648 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:54.646+0000 7f1d10a3b700 1 -- 172.21.15.155:0/3062116245 shutdown_connections 2024-02-10T09:03:54.648 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:54.646+0000 7f1d10a3b700 1 --2- 172.21.15.155:0/3062116245 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f1cf402ff10 0x7f1cf40323d0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:03:54.648 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:54.646+0000 7f1d10a3b700 1 --2- 172.21.15.155:0/3062116245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d0c060280 0x7f1d0c106780 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:03:54.648 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:54.646+0000 7f1d10a3b700 1 -- 172.21.15.155:0/3062116245 >> 172.21.15.155:0/3062116245 conn(0x7f1d0c0f0b70 msgr2=0x7f1d0c0f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:03:54.648 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:54.646+0000 7f1d10a3b700 1 -- 172.21.15.155:0/3062116245 shutdown_connections 2024-02-10T09:03:54.648 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:54.646+0000 7f1d10a3b700 1 -- 172.21.15.155:0/3062116245 wait complete. 2024-02-10T09:03:54.648 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:03:55.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:54 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:03:54.646133+0000 mon.smithi077 (mon.0) 195 : audit [DBG] from='client.? 172.21.15.155:0/3062116245' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:03:56.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:55 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:03:55.002769+0000 mgr.smithi077.dxntvw (mgr.14162) 155 : cluster [DBG] pgmap v148: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:03:56.397 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:03:56.397 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:03:57.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:57 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:03:57.003020+0000 mgr.smithi077.dxntvw (mgr.14162) 156 : cluster [DBG] pgmap v149: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:03:58.426 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:58.424+0000 7fa337c84700 1 -- 172.21.15.155:0/1721870630 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa33006b960 msgr2=0x7fa330069e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:03:58.426 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:58.424+0000 7fa337c84700 1 --2- 172.21.15.155:0/1721870630 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa33006b960 0x7fa330069e60 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7fa320009fb0 tx=0x7fa320009b30).stop 2024-02-10T09:03:58.426 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:58.424+0000 7fa337c84700 1 -- 172.21.15.155:0/1721870630 shutdown_connections 2024-02-10T09:03:58.426 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:58.424+0000 7fa337c84700 1 --2- 172.21.15.155:0/1721870630 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa33006b960 0x7fa330069e60 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:03:58.427 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:58.424+0000 7fa337c84700 1 -- 172.21.15.155:0/1721870630 >> 172.21.15.155:0/1721870630 conn(0x7fa3300f1c70 msgr2=0x7fa3300f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:03:58.427 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:58.424+0000 7fa337c84700 1 -- 172.21.15.155:0/1721870630 shutdown_connections 2024-02-10T09:03:58.427 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:58.424+0000 7fa337c84700 1 -- 172.21.15.155:0/1721870630 wait complete. 2024-02-10T09:03:58.427 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:58.425+0000 7fa337c84700 1 Processor -- start 2024-02-10T09:03:58.427 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:58.425+0000 7fa337c84700 1 -- start start 2024-02-10T09:03:58.427 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:58.425+0000 7fa337c84700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa33006b960 0x7fa3301089a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:03:58.427 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:58.425+0000 7fa337c84700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa330108e90 con 0x7fa33006b960 2024-02-10T09:03:58.427 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:58.425+0000 7fa335a20700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa33006b960 0x7fa3301089a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:03:58.427 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:58.425+0000 7fa335a20700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa33006b960 0x7fa3301089a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:43820/0 (socket says 172.21.15.155:43820) 2024-02-10T09:03:58.427 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:58.425+0000 7fa335a20700 1 -- 172.21.15.155:0/3919662345 learned_addr learned my addr 172.21.15.155:0/3919662345 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:03:58.427 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:58.425+0000 7fa335a20700 1 -- 172.21.15.155:0/3919662345 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa320005040 con 0x7fa33006b960 2024-02-10T09:03:58.427 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:58.426+0000 7fa335a20700 1 --2- 172.21.15.155:0/3919662345 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa33006b960 0x7fa3301089a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7fa3200089f0 tx=0x7fa32000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:03:58.427 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:58.426+0000 7fa3267fc700 1 -- 172.21.15.155:0/3919662345 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa320009320 con 0x7fa33006b960 2024-02-10T09:03:58.427 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:58.426+0000 7fa337c84700 1 -- 172.21.15.155:0/3919662345 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa330109090 con 0x7fa33006b960 2024-02-10T09:03:58.428 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:58.426+0000 7fa3267fc700 1 -- 172.21.15.155:0/3919662345 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa320013380 con 0x7fa33006b960 2024-02-10T09:03:58.428 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:58.426+0000 7fa3267fc700 1 -- 172.21.15.155:0/3919662345 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa320014e20 con 0x7fa33006b960 2024-02-10T09:03:58.428 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:58.426+0000 7fa337c84700 1 -- 172.21.15.155:0/3919662345 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa330109520 con 0x7fa33006b960 2024-02-10T09:03:58.429 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:58.427+0000 7fa3267fc700 1 -- 172.21.15.155:0/3919662345 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa320014420 con 0x7fa33006b960 2024-02-10T09:03:58.429 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:58.427+0000 7fa337c84700 1 -- 172.21.15.155:0/3919662345 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa330045ad0 con 0x7fa33006b960 2024-02-10T09:03:58.429 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:58.427+0000 7fa3267fc700 1 --2- 172.21.15.155:0/3919662345 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fa31c02ff10 0x7fa31c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:03:58.429 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:58.427+0000 7fa3267fc700 1 -- 172.21.15.155:0/3919662345 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa320040f50 con 0x7fa33006b960 2024-02-10T09:03:58.429 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:58.427+0000 7fa33521f700 1 --2- 172.21.15.155:0/3919662345 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fa31c02ff10 0x7fa31c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:03:58.429 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:58.427+0000 7fa33521f700 1 --2- 172.21.15.155:0/3919662345 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fa31c02ff10 0x7fa31c0323d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fa32c0096a0 tx=0x7fa32c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:03:58.431 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:58.431+0000 7fa3267fc700 1 -- 172.21.15.155:0/3919662345 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa32006d2b0 con 0x7fa33006b960 2024-02-10T09:03:58.638 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:58.638+0000 7fa337c84700 1 -- 172.21.15.155:0/3919662345 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa33010a3d0 con 0x7fa33006b960 2024-02-10T09:03:58.639 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:58.638+0000 7fa3267fc700 1 -- 172.21.15.155:0/3919662345 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa3200095f0 con 0x7fa33006b960 2024-02-10T09:03:58.639 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:03:58.639 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:03:58.640 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:58.640+0000 7fa337c84700 1 -- 172.21.15.155:0/3919662345 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fa31c02ff10 msgr2=0x7fa31c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:03:58.641 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:58.640+0000 7fa337c84700 1 --2- 172.21.15.155:0/3919662345 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fa31c02ff10 0x7fa31c0323d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fa32c0096a0 tx=0x7fa32c006b40).stop 2024-02-10T09:03:58.641 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:58.640+0000 7fa337c84700 1 -- 172.21.15.155:0/3919662345 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa33006b960 msgr2=0x7fa3301089a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:03:58.641 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:58.640+0000 7fa337c84700 1 --2- 172.21.15.155:0/3919662345 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa33006b960 0x7fa3301089a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7fa3200089f0 tx=0x7fa32000b040).stop 2024-02-10T09:03:58.641 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:58.640+0000 7fa337c84700 1 -- 172.21.15.155:0/3919662345 shutdown_connections 2024-02-10T09:03:58.641 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:58.640+0000 7fa337c84700 1 --2- 172.21.15.155:0/3919662345 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fa31c02ff10 0x7fa31c0323d0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:03:58.641 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:58.640+0000 7fa337c84700 1 --2- 172.21.15.155:0/3919662345 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa33006b960 0x7fa3301089a0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:03:58.641 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:58.640+0000 7fa337c84700 1 -- 172.21.15.155:0/3919662345 >> 172.21.15.155:0/3919662345 conn(0x7fa3300f1c70 msgr2=0x7fa3300f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:03:58.641 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:58.640+0000 7fa337c84700 1 -- 172.21.15.155:0/3919662345 shutdown_connections 2024-02-10T09:03:58.641 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:03:58.640+0000 7fa337c84700 1 -- 172.21.15.155:0/3919662345 wait complete. 2024-02-10T09:03:58.641 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:03:58.899 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:58 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:03:58.514048+0000 mon.smithi077 (mon.0) 196 : audit [INF] from='mgr.14162 172.21.15.77:0/2029106299' entity='mgr.smithi077.dxntvw' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi077", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-10T09:03:58.899 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:58 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:03:58.516585+0000 mon.smithi077 (mon.0) 197 : audit [INF] from='mgr.14162 172.21.15.77:0/2029106299' entity='mgr.smithi077.dxntvw' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi077", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-02-10T09:03:58.899 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:58 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:03:58.517371+0000 mon.smithi077 (mon.0) 198 : audit [DBG] from='mgr.14162 172.21.15.77:0/2029106299' entity='mgr.smithi077.dxntvw' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T09:03:58.899 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:58 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:03:58.639009+0000 mon.smithi077 (mon.0) 199 : audit [DBG] from='client.? 172.21.15.155:0/3919662345' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:03:59.900 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cephadm 2024-02-10T09:03:58.512752+0000 mgr.smithi077.dxntvw (mgr.14162) 157 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi077 ... 2024-02-10T09:03:59.900 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2024-02-10T09:03:59.900 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-10T09:03:59.900 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: stat: stderr time="2024-02-10T08:59:36Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: copying system image from manifest list: determining manifest MIME type for docker://prom/alertmanager:v0.20.0: reading manifest sha256:b9323917a2eda265bec69e59a457f001c529facbbc8166df277f4850cdac61a0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:03:59.900 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: stat: stderr time="2024-02-10T09:00:14Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:03:59.900 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: stat: stderr time="2024-02-10T09:00:51Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:03:59.900 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-10T09:03:59.901 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2024-02-10T09:03:59.901 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-10T09:03:59.901 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: stat: stderr time="2024-02-10T09:02:06Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:03:59.901 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: stat: stderr time="2024-02-10T09:02:43Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:03:59.901 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: stat: stderr time="2024-02-10T09:03:21Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:03:59.901 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-10T09:03:59.901 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: Traceback (most recent call last): 2024-02-10T09:03:59.901 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: File "/var/lib/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-10T09:03:59.901 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: main() 2024-02-10T09:03:59.901 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: File "/var/lib/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-10T09:03:59.901 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: r = ctx.func(ctx) 2024-02-10T09:03:59.901 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: File "/var/lib/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-10T09:03:59.901 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: return func(ctx) 2024-02-10T09:03:59.902 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: File "/var/lib/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-02-10T09:03:59.902 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-10T09:03:59.902 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: File "/var/lib/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2024-02-10T09:03:59.902 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2024-02-10T09:03:59.902 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: File "/var/lib/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-02-10T09:03:59.902 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: raise RuntimeError('uid/gid not found') 2024-02-10T09:03:59.902 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: RuntimeError: uid/gid not found 2024-02-10T09:03:59.902 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: Traceback (most recent call last): 2024-02-10T09:03:59.902 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-02-10T09:03:59.902 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: yield (conn, connr) 2024-02-10T09:03:59.902 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-02-10T09:03:59.902 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: code, '\n'.join(err))) 2024-02-10T09:03:59.902 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi077 ... 2024-02-10T09:03:59.903 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2024-02-10T09:03:59.903 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-10T09:03:59.903 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: stat: stderr time="2024-02-10T08:59:36Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: copying system image from manifest list: determining manifest MIME type for docker://prom/alertmanager:v0.20.0: reading manifest sha256:b9323917a2eda265bec69e59a457f001c529facbbc8166df277f4850cdac61a0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:03:59.903 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: stat: stderr time="2024-02-10T09:00:14Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:03:59.903 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: stat: stderr time="2024-02-10T09:00:51Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:03:59.903 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-10T09:03:59.903 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2024-02-10T09:03:59.903 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-10T09:03:59.903 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: stat: stderr time="2024-02-10T09:02:06Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:03:59.903 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: stat: stderr time="2024-02-10T09:02:43Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:03:59.903 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: stat: stderr time="2024-02-10T09:03:21Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:03:59.903 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-10T09:03:59.903 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: Traceback (most recent call last): 2024-02-10T09:03:59.904 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: File "/var/lib/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-10T09:03:59.904 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: main() 2024-02-10T09:03:59.904 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: File "/var/lib/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-10T09:03:59.904 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: r = ctx.func(ctx) 2024-02-10T09:03:59.904 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: File "/var/lib/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-10T09:03:59.904 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: return func(ctx) 2024-02-10T09:03:59.904 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: File "/var/lib/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-02-10T09:03:59.904 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-10T09:03:59.904 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: File "/var/lib/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2024-02-10T09:03:59.904 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2024-02-10T09:03:59.904 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: File "/var/lib/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-02-10T09:03:59.904 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: raise RuntimeError('uid/gid not found') 2024-02-10T09:03:59.904 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: RuntimeError: uid/gid not found 2024-02-10T09:03:59.905 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cephadm 2024-02-10T09:03:58.517935+0000 mgr.smithi077.dxntvw (mgr.14162) 158 : cephadm [INF] Deploying daemon crash.smithi077 on smithi077 2024-02-10T09:03:59.905 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:03:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:03:59.003370+0000 mgr.smithi077.dxntvw (mgr.14162) 159 : cluster [DBG] pgmap v150: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:04:00.448 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:04:00.449 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:04:02.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:04:01 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:04:00.858280+0000 mon.smithi077 (mon.0) 200 : audit [INF] from='mgr.14162 172.21.15.77:0/2029106299' entity='mgr.smithi077.dxntvw' 2024-02-10T09:04:02.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:04:01 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:04:01.003626+0000 mgr.smithi077.dxntvw (mgr.14162) 160 : cluster [DBG] pgmap v151: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:04:02.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:04:01 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:04:01.038260+0000 mon.smithi077 (mon.0) 201 : audit [INF] from='mgr.14162 172.21.15.77:0/2029106299' entity='mgr.smithi077.dxntvw' 2024-02-10T09:04:02.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:04:01 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:04:01.040364+0000 mon.smithi077 (mon.0) 202 : audit [INF] from='mgr.14162 172.21.15.77:0/2029106299' entity='mgr.smithi077.dxntvw' 2024-02-10T09:04:02.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:04:01 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:04:01.040742+0000 mon.smithi077 (mon.0) 203 : audit [INF] from='mgr.14162 172.21.15.77:0/2029106299' entity='mgr.smithi077.dxntvw' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T09:04:02.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:04:01 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:04:01.043493+0000 mon.smithi077 (mon.0) 204 : audit [INF] from='mgr.14162 172.21.15.77:0/2029106299' entity='mgr.smithi077.dxntvw' 2024-02-10T09:04:02.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:02.356+0000 7f1cc339c700 1 -- 172.21.15.155:0/1137758000 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1cbc060280 msgr2=0x7f1cbc0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:04:02.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:02.356+0000 7f1cc339c700 1 --2- 172.21.15.155:0/1137758000 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1cbc060280 0x7f1cbc0f90f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f1cac009fb0 tx=0x7f1cac009b30).stop 2024-02-10T09:04:02.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:02.357+0000 7f1cc339c700 1 -- 172.21.15.155:0/1137758000 shutdown_connections 2024-02-10T09:04:02.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:02.357+0000 7f1cc339c700 1 --2- 172.21.15.155:0/1137758000 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1cbc060280 0x7f1cbc0f90f0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:04:02.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:02.357+0000 7f1cc339c700 1 -- 172.21.15.155:0/1137758000 >> 172.21.15.155:0/1137758000 conn(0x7f1cbc0f0b70 msgr2=0x7f1cbc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:04:02.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:02.357+0000 7f1cc339c700 1 -- 172.21.15.155:0/1137758000 shutdown_connections 2024-02-10T09:04:02.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:02.357+0000 7f1cc339c700 1 -- 172.21.15.155:0/1137758000 wait complete. 2024-02-10T09:04:02.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:02.357+0000 7f1cc339c700 1 Processor -- start 2024-02-10T09:04:02.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:02.357+0000 7f1cc339c700 1 -- start start 2024-02-10T09:04:02.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:02.357+0000 7f1cc339c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1cbc060280 0x7f1cbc106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:04:02.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:02.357+0000 7f1cc339c700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cbc106c70 con 0x7f1cbc060280 2024-02-10T09:04:02.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:02.358+0000 7f1cc1138700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1cbc060280 0x7f1cbc106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:04:02.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:02.358+0000 7f1cc1138700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1cbc060280 0x7f1cbc106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:50240/0 (socket says 172.21.15.155:50240) 2024-02-10T09:04:02.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:02.358+0000 7f1cc1138700 1 -- 172.21.15.155:0/3562172806 learned_addr learned my addr 172.21.15.155:0/3562172806 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:04:02.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:02.358+0000 7f1cc1138700 1 -- 172.21.15.155:0/3562172806 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cac005040 con 0x7f1cbc060280 2024-02-10T09:04:02.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:02.358+0000 7f1cc1138700 1 --2- 172.21.15.155:0/3562172806 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1cbc060280 0x7f1cbc106780 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f1cac009830 tx=0x7f1cac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:04:02.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:02.358+0000 7f1cb1ffb700 1 -- 172.21.15.155:0/3562172806 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1cac009320 con 0x7f1cbc060280 2024-02-10T09:04:02.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:02.358+0000 7f1cc339c700 1 -- 172.21.15.155:0/3562172806 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cbc106e70 con 0x7f1cbc060280 2024-02-10T09:04:02.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:02.358+0000 7f1cb1ffb700 1 -- 172.21.15.155:0/3562172806 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1cac013380 con 0x7f1cbc060280 2024-02-10T09:04:02.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:02.358+0000 7f1cb1ffb700 1 -- 172.21.15.155:0/3562172806 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1cac014420 con 0x7f1cbc060280 2024-02-10T09:04:02.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:02.358+0000 7f1cc339c700 1 -- 172.21.15.155:0/3562172806 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cbc107300 con 0x7f1cbc060280 2024-02-10T09:04:02.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:02.359+0000 7f1cc339c700 1 -- 172.21.15.155:0/3562172806 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cbc045ad0 con 0x7f1cbc060280 2024-02-10T09:04:02.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:02.359+0000 7f1cb1ffb700 1 -- 172.21.15.155:0/3562172806 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1cac01e420 con 0x7f1cbc060280 2024-02-10T09:04:02.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:02.359+0000 7f1cb1ffb700 1 --2- 172.21.15.155:0/3562172806 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f1ca802ff60 0x7f1ca8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:04:02.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:02.359+0000 7f1cb1ffb700 1 -- 172.21.15.155:0/3562172806 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1cac040ee0 con 0x7f1cbc060280 2024-02-10T09:04:02.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:02.360+0000 7f1cc0937700 1 --2- 172.21.15.155:0/3562172806 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f1ca802ff60 0x7f1ca8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:04:02.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:02.360+0000 7f1cc0937700 1 --2- 172.21.15.155:0/3562172806 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f1ca802ff60 0x7f1ca8032420 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f1cb80096a0 tx=0x7f1cb8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:04:02.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:02.364+0000 7f1cb1ffb700 1 -- 172.21.15.155:0/3562172806 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1cac01c930 con 0x7f1cbc060280 2024-02-10T09:04:02.567 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:02.567+0000 7f1cc339c700 1 -- 172.21.15.155:0/3562172806 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1cbc0fc090 con 0x7f1cbc060280 2024-02-10T09:04:02.568 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:02.568+0000 7f1cb1ffb700 1 -- 172.21.15.155:0/3562172806 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1cac02c2d0 con 0x7f1cbc060280 2024-02-10T09:04:02.568 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:04:02.568 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:04:02.570 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:02.569+0000 7f1cc339c700 1 -- 172.21.15.155:0/3562172806 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f1ca802ff60 msgr2=0x7f1ca8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:04:02.570 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:02.569+0000 7f1cc339c700 1 --2- 172.21.15.155:0/3562172806 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f1ca802ff60 0x7f1ca8032420 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f1cb80096a0 tx=0x7f1cb8006b40).stop 2024-02-10T09:04:02.570 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:02.569+0000 7f1cc339c700 1 -- 172.21.15.155:0/3562172806 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1cbc060280 msgr2=0x7f1cbc106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:04:02.570 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:02.569+0000 7f1cc339c700 1 --2- 172.21.15.155:0/3562172806 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1cbc060280 0x7f1cbc106780 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f1cac009830 tx=0x7f1cac00b040).stop 2024-02-10T09:04:02.570 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:02.569+0000 7f1cc339c700 1 -- 172.21.15.155:0/3562172806 shutdown_connections 2024-02-10T09:04:02.570 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:02.569+0000 7f1cc339c700 1 --2- 172.21.15.155:0/3562172806 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f1ca802ff60 0x7f1ca8032420 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:04:02.570 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:02.569+0000 7f1cc339c700 1 --2- 172.21.15.155:0/3562172806 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1cbc060280 0x7f1cbc106780 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:04:02.570 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:02.569+0000 7f1cc339c700 1 -- 172.21.15.155:0/3562172806 >> 172.21.15.155:0/3562172806 conn(0x7f1cbc0f0b70 msgr2=0x7f1cbc0f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:04:02.570 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:02.569+0000 7f1cc339c700 1 -- 172.21.15.155:0/3562172806 shutdown_connections 2024-02-10T09:04:02.570 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:02.570+0000 7f1cc339c700 1 -- 172.21.15.155:0/3562172806 wait complete. 2024-02-10T09:04:02.571 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:04:03.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:04:02 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:04:01.041142+0000 mgr.smithi077.dxntvw (mgr.14162) 161 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T09:04:03.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:04:02 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cephadm 2024-02-10T09:04:01.046661+0000 mgr.smithi077.dxntvw (mgr.14162) 162 : cephadm [INF] Deploying daemon grafana.smithi077 on smithi077 2024-02-10T09:04:03.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:04:02 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:04:02.568464+0000 mon.smithi077 (mon.0) 205 : audit [DBG] from='client.? 172.21.15.155:0/3562172806' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:04:04.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:04:03 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:04:03.003979+0000 mgr.smithi077.dxntvw (mgr.14162) 163 : cluster [DBG] pgmap v152: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:04:04.206 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:04:04.207 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:04:05.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:04:05 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:04:05.004443+0000 mgr.smithi077.dxntvw (mgr.14162) 164 : cluster [DBG] pgmap v153: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:04:06.308 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:06.291+0000 7f7025c81700 1 -- 172.21.15.155:0/601434011 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70200f5410 msgr2=0x7f70200f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:04:06.309 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:06.291+0000 7f7025c81700 1 --2- 172.21.15.155:0/601434011 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70200f5410 0x7f70200f5830 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f7008009fb0 tx=0x7f7008009b30).stop 2024-02-10T09:04:06.309 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:06.291+0000 7f7025c81700 1 -- 172.21.15.155:0/601434011 shutdown_connections 2024-02-10T09:04:06.309 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:06.291+0000 7f7025c81700 1 --2- 172.21.15.155:0/601434011 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70200f5410 0x7f70200f5830 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:04:06.309 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:06.291+0000 7f7025c81700 1 -- 172.21.15.155:0/601434011 >> 172.21.15.155:0/601434011 conn(0x7f70200f0b70 msgr2=0x7f70200f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:04:06.309 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:06.291+0000 7f7025c81700 1 -- 172.21.15.155:0/601434011 shutdown_connections 2024-02-10T09:04:06.309 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:06.291+0000 7f7025c81700 1 -- 172.21.15.155:0/601434011 wait complete. 2024-02-10T09:04:06.309 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:06.292+0000 7f7025c81700 1 Processor -- start 2024-02-10T09:04:06.309 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:06.292+0000 7f7025c81700 1 -- start start 2024-02-10T09:04:06.309 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:06.292+0000 7f7025c81700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70200f5410 0x7f7020105710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:04:06.309 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:06.292+0000 7f7025c81700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7020105c00 con 0x7f70200f5410 2024-02-10T09:04:06.309 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:06.293+0000 7f701f7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70200f5410 0x7f7020105710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:04:06.310 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:06.293+0000 7f701f7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70200f5410 0x7f7020105710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:50264/0 (socket says 172.21.15.155:50264) 2024-02-10T09:04:06.310 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:06.293+0000 7f701f7fe700 1 -- 172.21.15.155:0/1895539907 learned_addr learned my addr 172.21.15.155:0/1895539907 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:04:06.310 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:06.293+0000 7f701f7fe700 1 -- 172.21.15.155:0/1895539907 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7008005040 con 0x7f70200f5410 2024-02-10T09:04:06.310 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:06.293+0000 7f701f7fe700 1 --2- 172.21.15.155:0/1895539907 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70200f5410 0x7f7020105710 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f7008005010 tx=0x7f700800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:04:06.310 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:06.293+0000 7f7017fff700 1 -- 172.21.15.155:0/1895539907 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7008009320 con 0x7f70200f5410 2024-02-10T09:04:06.310 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:06.293+0000 7f7025c81700 1 -- 172.21.15.155:0/1895539907 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7020105e00 con 0x7f70200f5410 2024-02-10T09:04:06.310 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:06.293+0000 7f7017fff700 1 -- 172.21.15.155:0/1895539907 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7008013380 con 0x7f70200f5410 2024-02-10T09:04:06.310 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:06.293+0000 7f7017fff700 1 -- 172.21.15.155:0/1895539907 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7008014e50 con 0x7f70200f5410 2024-02-10T09:04:06.310 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:06.293+0000 7f7025c81700 1 -- 172.21.15.155:0/1895539907 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7020106290 con 0x7f70200f5410 2024-02-10T09:04:06.310 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:06.294+0000 7f7017fff700 1 -- 172.21.15.155:0/1895539907 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7008014420 con 0x7f70200f5410 2024-02-10T09:04:06.310 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:06.294+0000 7f7025c81700 1 -- 172.21.15.155:0/1895539907 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7020045ad0 con 0x7f70200f5410 2024-02-10T09:04:06.310 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:06.294+0000 7f7017fff700 1 --2- 172.21.15.155:0/1895539907 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f700c02ff10 0x7f700c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:04:06.310 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:06.294+0000 7f7017fff700 1 -- 172.21.15.155:0/1895539907 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7008040e30 con 0x7f70200f5410 2024-02-10T09:04:06.310 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:06.295+0000 7f701effd700 1 --2- 172.21.15.155:0/1895539907 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f700c02ff10 0x7f700c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:04:06.311 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:06.295+0000 7f701effd700 1 --2- 172.21.15.155:0/1895539907 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f700c02ff10 0x7f700c0323d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f70100096a0 tx=0x7f7010006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:04:06.311 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:06.299+0000 7f7017fff700 1 -- 172.21.15.155:0/1895539907 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f70080095f0 con 0x7f70200f5410 2024-02-10T09:04:06.507 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:06.507+0000 7f7025c81700 1 -- 172.21.15.155:0/1895539907 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7020107140 con 0x7f70200f5410 2024-02-10T09:04:06.508 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:06.508+0000 7f7017fff700 1 -- 172.21.15.155:0/1895539907 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f700801db70 con 0x7f70200f5410 2024-02-10T09:04:06.508 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:04:06.508 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:04:06.510 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:06.509+0000 7f7025c81700 1 -- 172.21.15.155:0/1895539907 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f700c02ff10 msgr2=0x7f700c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:04:06.510 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:06.509+0000 7f7025c81700 1 --2- 172.21.15.155:0/1895539907 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f700c02ff10 0x7f700c0323d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f70100096a0 tx=0x7f7010006b40).stop 2024-02-10T09:04:06.510 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:06.509+0000 7f7025c81700 1 -- 172.21.15.155:0/1895539907 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70200f5410 msgr2=0x7f7020105710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:04:06.510 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:06.509+0000 7f7025c81700 1 --2- 172.21.15.155:0/1895539907 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70200f5410 0x7f7020105710 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f7008005010 tx=0x7f700800b040).stop 2024-02-10T09:04:06.510 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:06.509+0000 7f7025c81700 1 -- 172.21.15.155:0/1895539907 shutdown_connections 2024-02-10T09:04:06.510 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:06.509+0000 7f7025c81700 1 --2- 172.21.15.155:0/1895539907 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f700c02ff10 0x7f700c0323d0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:04:06.510 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:06.509+0000 7f7025c81700 1 --2- 172.21.15.155:0/1895539907 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70200f5410 0x7f7020105710 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:04:06.510 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:06.509+0000 7f7025c81700 1 -- 172.21.15.155:0/1895539907 >> 172.21.15.155:0/1895539907 conn(0x7f70200f0b70 msgr2=0x7f70200f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:04:06.510 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:06.509+0000 7f7025c81700 1 -- 172.21.15.155:0/1895539907 shutdown_connections 2024-02-10T09:04:06.510 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:06.510+0000 7f7025c81700 1 -- 172.21.15.155:0/1895539907 wait complete. 2024-02-10T09:04:06.511 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:04:06.899 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:04:06 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:04:06.508254+0000 mon.smithi077 (mon.0) 206 : audit [DBG] from='client.? 172.21.15.155:0/1895539907' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:04:07.899 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:04:07 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:04:07.004771+0000 mgr.smithi077.dxntvw (mgr.14162) 165 : cluster [DBG] pgmap v154: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:04:08.313 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:04:08.314 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:04:09.398 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:04:09 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:04:09.005007+0000 mgr.smithi077.dxntvw (mgr.14162) 166 : cluster [DBG] pgmap v155: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:04:10.280 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:10.275+0000 7f23dbfff700 1 -- 172.21.15.155:0/1741498331 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23dc0f51a0 msgr2=0x7f23dc0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:04:10.280 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:10.275+0000 7f23dbfff700 1 --2- 172.21.15.155:0/1741498331 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23dc0f51a0 0x7f23dc0f55c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f23d0009fb0 tx=0x7f23d0009b30).stop 2024-02-10T09:04:10.281 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:10.275+0000 7f23dbfff700 1 -- 172.21.15.155:0/1741498331 shutdown_connections 2024-02-10T09:04:10.281 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:10.275+0000 7f23dbfff700 1 --2- 172.21.15.155:0/1741498331 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23dc0f51a0 0x7f23dc0f55c0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:04:10.281 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:10.275+0000 7f23dbfff700 1 -- 172.21.15.155:0/1741498331 >> 172.21.15.155:0/1741498331 conn(0x7f23dc0f09a0 msgr2=0x7f23dc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:04:10.281 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:10.275+0000 7f23dbfff700 1 -- 172.21.15.155:0/1741498331 shutdown_connections 2024-02-10T09:04:10.281 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:10.275+0000 7f23dbfff700 1 -- 172.21.15.155:0/1741498331 wait complete. 2024-02-10T09:04:10.281 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:10.275+0000 7f23dbfff700 1 Processor -- start 2024-02-10T09:04:10.281 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:10.275+0000 7f23dbfff700 1 -- start start 2024-02-10T09:04:10.281 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:10.276+0000 7f23dbfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23dc0f51a0 0x7f23dc1065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:04:10.281 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:10.276+0000 7f23dbfff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23dc106ac0 con 0x7f23dc0f51a0 2024-02-10T09:04:10.281 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:10.276+0000 7f23daffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23dc0f51a0 0x7f23dc1065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:04:10.281 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:10.276+0000 7f23daffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23dc0f51a0 0x7f23dc1065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:50286/0 (socket says 172.21.15.155:50286) 2024-02-10T09:04:10.281 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:10.276+0000 7f23daffd700 1 -- 172.21.15.155:0/2069606642 learned_addr learned my addr 172.21.15.155:0/2069606642 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:04:10.281 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:10.276+0000 7f23daffd700 1 -- 172.21.15.155:0/2069606642 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23d0005040 con 0x7f23dc0f51a0 2024-02-10T09:04:10.282 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:10.276+0000 7f23daffd700 1 --2- 172.21.15.155:0/2069606642 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23dc0f51a0 0x7f23dc1065d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f23d0009830 tx=0x7f23d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:04:10.282 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:10.276+0000 7f23c37fe700 1 -- 172.21.15.155:0/2069606642 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23d0009320 con 0x7f23dc0f51a0 2024-02-10T09:04:10.282 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:10.276+0000 7f23c37fe700 1 -- 172.21.15.155:0/2069606642 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f23d0013380 con 0x7f23dc0f51a0 2024-02-10T09:04:10.282 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:10.276+0000 7f23dbfff700 1 -- 172.21.15.155:0/2069606642 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23dc106cc0 con 0x7f23dc0f51a0 2024-02-10T09:04:10.282 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:10.276+0000 7f23c37fe700 1 -- 172.21.15.155:0/2069606642 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23d0014420 con 0x7f23dc0f51a0 2024-02-10T09:04:10.282 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:10.276+0000 7f23dbfff700 1 -- 172.21.15.155:0/2069606642 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23dc107150 con 0x7f23dc0f51a0 2024-02-10T09:04:10.282 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:10.277+0000 7f23c37fe700 1 -- 172.21.15.155:0/2069606642 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f23d001e420 con 0x7f23dc0f51a0 2024-02-10T09:04:10.282 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:10.277+0000 7f23dbfff700 1 -- 172.21.15.155:0/2069606642 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23dc045ad0 con 0x7f23dc0f51a0 2024-02-10T09:04:10.282 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:10.277+0000 7f23c37fe700 1 --2- 172.21.15.155:0/2069606642 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f23c402ff60 0x7f23c4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:04:10.282 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:10.277+0000 7f23c37fe700 1 -- 172.21.15.155:0/2069606642 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f23d0040e20 con 0x7f23dc0f51a0 2024-02-10T09:04:10.282 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:10.278+0000 7f23da7fc700 1 --2- 172.21.15.155:0/2069606642 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f23c402ff60 0x7f23c4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:04:10.282 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:10.278+0000 7f23da7fc700 1 --2- 172.21.15.155:0/2069606642 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f23c402ff60 0x7f23c4032420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f23cc0096a0 tx=0x7f23cc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:04:10.283 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:10.282+0000 7f23c37fe700 1 -- 172.21.15.155:0/2069606642 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f23d001c970 con 0x7f23dc0f51a0 2024-02-10T09:04:10.489 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:10.489+0000 7f23dbfff700 1 -- 172.21.15.155:0/2069606642 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f23dc0fad30 con 0x7f23dc0f51a0 2024-02-10T09:04:10.489 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:10.489+0000 7f23c37fe700 1 -- 172.21.15.155:0/2069606642 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f23d002c2d0 con 0x7f23dc0f51a0 2024-02-10T09:04:10.490 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:04:10.490 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:04:10.491 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:10.491+0000 7f23dbfff700 1 -- 172.21.15.155:0/2069606642 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f23c402ff60 msgr2=0x7f23c4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:04:10.491 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:10.491+0000 7f23dbfff700 1 --2- 172.21.15.155:0/2069606642 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f23c402ff60 0x7f23c4032420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f23cc0096a0 tx=0x7f23cc006b40).stop 2024-02-10T09:04:10.491 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:10.491+0000 7f23dbfff700 1 -- 172.21.15.155:0/2069606642 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23dc0f51a0 msgr2=0x7f23dc1065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:04:10.492 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:10.491+0000 7f23dbfff700 1 --2- 172.21.15.155:0/2069606642 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23dc0f51a0 0x7f23dc1065d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f23d0009830 tx=0x7f23d000b040).stop 2024-02-10T09:04:10.492 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:10.491+0000 7f23dbfff700 1 -- 172.21.15.155:0/2069606642 shutdown_connections 2024-02-10T09:04:10.492 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:10.491+0000 7f23dbfff700 1 --2- 172.21.15.155:0/2069606642 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f23c402ff60 0x7f23c4032420 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:04:10.492 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:10.491+0000 7f23dbfff700 1 --2- 172.21.15.155:0/2069606642 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23dc0f51a0 0x7f23dc1065d0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:04:10.492 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:10.491+0000 7f23dbfff700 1 -- 172.21.15.155:0/2069606642 >> 172.21.15.155:0/2069606642 conn(0x7f23dc0f09a0 msgr2=0x7f23dc05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:04:10.492 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:10.491+0000 7f23dbfff700 1 -- 172.21.15.155:0/2069606642 shutdown_connections 2024-02-10T09:04:10.492 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:10.491+0000 7f23dbfff700 1 -- 172.21.15.155:0/2069606642 wait complete. 2024-02-10T09:04:10.492 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:04:10.898 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:04:10 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:04:10.489995+0000 mon.smithi077 (mon.0) 207 : audit [DBG] from='client.? 172.21.15.155:0/2069606642' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:04:11.899 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:04:11 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:04:11.005225+0000 mgr.smithi077.dxntvw (mgr.14162) 167 : cluster [DBG] pgmap v156: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:04:12.139 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:04:12.140 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:04:14.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:04:13 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:04:13.005405+0000 mgr.smithi077.dxntvw (mgr.14162) 168 : cluster [DBG] pgmap v157: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:04:14.184 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:14.181+0000 7f1a2759e700 1 -- 172.21.15.155:0/3292745951 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a280f51a0 msgr2=0x7f1a280f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:04:14.184 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:14.181+0000 7f1a2759e700 1 --2- 172.21.15.155:0/3292745951 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a280f51a0 0x7f1a280f55c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f1a18009fb0 tx=0x7f1a18009b30).stop 2024-02-10T09:04:14.185 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:14.182+0000 7f1a2759e700 1 -- 172.21.15.155:0/3292745951 shutdown_connections 2024-02-10T09:04:14.185 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:14.182+0000 7f1a2759e700 1 --2- 172.21.15.155:0/3292745951 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a280f51a0 0x7f1a280f55c0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:04:14.185 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:14.182+0000 7f1a2759e700 1 -- 172.21.15.155:0/3292745951 >> 172.21.15.155:0/3292745951 conn(0x7f1a280f09a0 msgr2=0x7f1a280f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:04:14.185 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:14.182+0000 7f1a2759e700 1 -- 172.21.15.155:0/3292745951 shutdown_connections 2024-02-10T09:04:14.185 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:14.182+0000 7f1a2759e700 1 -- 172.21.15.155:0/3292745951 wait complete. 2024-02-10T09:04:14.185 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:14.182+0000 7f1a2759e700 1 Processor -- start 2024-02-10T09:04:14.185 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:14.182+0000 7f1a2759e700 1 -- start start 2024-02-10T09:04:14.185 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:14.183+0000 7f1a2759e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a280f51a0 0x7f1a28109290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:04:14.185 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:14.183+0000 7f1a2759e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a28060280 con 0x7f1a280f51a0 2024-02-10T09:04:14.185 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:14.183+0000 7f1a2659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a280f51a0 0x7f1a28109290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:04:14.185 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:14.183+0000 7f1a2659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a280f51a0 0x7f1a28109290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:37822/0 (socket says 172.21.15.155:37822) 2024-02-10T09:04:14.185 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:14.183+0000 7f1a2659c700 1 -- 172.21.15.155:0/1932923297 learned_addr learned my addr 172.21.15.155:0/1932923297 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:04:14.185 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:14.183+0000 7f1a2659c700 1 -- 172.21.15.155:0/1932923297 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a18005040 con 0x7f1a280f51a0 2024-02-10T09:04:14.186 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:14.183+0000 7f1a2659c700 1 --2- 172.21.15.155:0/1932923297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a280f51a0 0x7f1a28109290 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f1a180089f0 tx=0x7f1a1800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:04:14.186 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:14.183+0000 7f1a16ffd700 1 -- 172.21.15.155:0/1932923297 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a18009320 con 0x7f1a280f51a0 2024-02-10T09:04:14.186 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:14.183+0000 7f1a16ffd700 1 -- 172.21.15.155:0/1932923297 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1a18013380 con 0x7f1a280f51a0 2024-02-10T09:04:14.186 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:14.183+0000 7f1a2759e700 1 -- 172.21.15.155:0/1932923297 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a28060480 con 0x7f1a280f51a0 2024-02-10T09:04:14.186 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:14.183+0000 7f1a16ffd700 1 -- 172.21.15.155:0/1932923297 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a18014e20 con 0x7f1a280f51a0 2024-02-10T09:04:14.186 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:14.183+0000 7f1a2759e700 1 -- 172.21.15.155:0/1932923297 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a281099c0 con 0x7f1a280f51a0 2024-02-10T09:04:14.187 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:14.184+0000 7f1a16ffd700 1 -- 172.21.15.155:0/1932923297 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1a180263a0 con 0x7f1a280f51a0 2024-02-10T09:04:14.187 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:14.184+0000 7f1a16ffd700 1 --2- 172.21.15.155:0/1932923297 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f1a1002ff60 0x7f1a10032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:04:14.187 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:14.184+0000 7f1a16ffd700 1 -- 172.21.15.155:0/1932923297 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1a180411f0 con 0x7f1a280f51a0 2024-02-10T09:04:14.187 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:14.184+0000 7f1a2759e700 1 -- 172.21.15.155:0/1932923297 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a28045ad0 con 0x7f1a280f51a0 2024-02-10T09:04:14.187 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:14.184+0000 7f1a25d9b700 1 --2- 172.21.15.155:0/1932923297 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f1a1002ff60 0x7f1a10032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:04:14.187 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:14.185+0000 7f1a25d9b700 1 --2- 172.21.15.155:0/1932923297 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f1a1002ff60 0x7f1a10032420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f1a1c0096a0 tx=0x7f1a1c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:04:14.190 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:14.190+0000 7f1a16ffd700 1 -- 172.21.15.155:0/1932923297 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1a18012920 con 0x7f1a280f51a0 2024-02-10T09:04:14.398 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:14.397+0000 7f1a2759e700 1 -- 172.21.15.155:0/1932923297 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1a2810a0a0 con 0x7f1a280f51a0 2024-02-10T09:04:14.398 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:14.397+0000 7f1a16ffd700 1 -- 172.21.15.155:0/1932923297 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1a18012330 con 0x7f1a280f51a0 2024-02-10T09:04:14.398 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:04:14.399 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:04:14.400 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:14.399+0000 7f1a2759e700 1 -- 172.21.15.155:0/1932923297 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f1a1002ff60 msgr2=0x7f1a10032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:04:14.400 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:14.399+0000 7f1a2759e700 1 --2- 172.21.15.155:0/1932923297 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f1a1002ff60 0x7f1a10032420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f1a1c0096a0 tx=0x7f1a1c006b40).stop 2024-02-10T09:04:14.400 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:14.399+0000 7f1a2759e700 1 -- 172.21.15.155:0/1932923297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a280f51a0 msgr2=0x7f1a28109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:04:14.400 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:14.399+0000 7f1a2759e700 1 --2- 172.21.15.155:0/1932923297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a280f51a0 0x7f1a28109290 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f1a180089f0 tx=0x7f1a1800b040).stop 2024-02-10T09:04:14.401 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:14.399+0000 7f1a2759e700 1 -- 172.21.15.155:0/1932923297 shutdown_connections 2024-02-10T09:04:14.401 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:14.399+0000 7f1a2759e700 1 --2- 172.21.15.155:0/1932923297 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f1a1002ff60 0x7f1a10032420 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:04:14.401 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:14.399+0000 7f1a2759e700 1 --2- 172.21.15.155:0/1932923297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1a280f51a0 0x7f1a28109290 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:04:14.401 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:14.399+0000 7f1a2759e700 1 -- 172.21.15.155:0/1932923297 >> 172.21.15.155:0/1932923297 conn(0x7f1a280f09a0 msgr2=0x7f1a280f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:04:14.401 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:14.399+0000 7f1a2759e700 1 -- 172.21.15.155:0/1932923297 shutdown_connections 2024-02-10T09:04:14.401 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:14.399+0000 7f1a2759e700 1 -- 172.21.15.155:0/1932923297 wait complete. 2024-02-10T09:04:14.401 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:04:15.148 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:04:14 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:04:14.398746+0000 mon.smithi077 (mon.0) 208 : audit [DBG] from='client.? 172.21.15.155:0/1932923297' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:04:16.004 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:04:15 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:04:15.005630+0000 mgr.smithi077.dxntvw (mgr.14162) 169 : cluster [DBG] pgmap v158: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:04:16.014 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:04:16.015 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:04:17.343 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:04:17 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:04:17.005882+0000 mgr.smithi077.dxntvw (mgr.14162) 170 : cluster [DBG] pgmap v159: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:04:18.015 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:18.012+0000 7f45409e8700 1 -- 172.21.15.155:0/1589604474 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f453c0601e0 msgr2=0x7f453c0f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:04:18.016 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:18.012+0000 7f45409e8700 1 --2- 172.21.15.155:0/1589604474 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f453c0601e0 0x7f453c0f8f20 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f452c009fb0 tx=0x7f452c009b30).stop 2024-02-10T09:04:18.016 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:18.012+0000 7f45409e8700 1 -- 172.21.15.155:0/1589604474 shutdown_connections 2024-02-10T09:04:18.016 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:18.012+0000 7f45409e8700 1 --2- 172.21.15.155:0/1589604474 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f453c0601e0 0x7f453c0f8f20 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:04:18.016 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:18.012+0000 7f45409e8700 1 -- 172.21.15.155:0/1589604474 >> 172.21.15.155:0/1589604474 conn(0x7f453c0f09a0 msgr2=0x7f453c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:04:18.016 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:18.012+0000 7f45409e8700 1 -- 172.21.15.155:0/1589604474 shutdown_connections 2024-02-10T09:04:18.016 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:18.012+0000 7f45409e8700 1 -- 172.21.15.155:0/1589604474 wait complete. 2024-02-10T09:04:18.016 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:18.013+0000 7f45409e8700 1 Processor -- start 2024-02-10T09:04:18.016 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:18.013+0000 7f45409e8700 1 -- start start 2024-02-10T09:04:18.016 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:18.013+0000 7f45409e8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f453c0601e0 0x7f453c1087d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:04:18.017 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:18.013+0000 7f45409e8700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f453c108cc0 con 0x7f453c0601e0 2024-02-10T09:04:18.017 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:18.014+0000 7f453b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f453c0601e0 0x7f453c1087d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:04:18.017 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:18.014+0000 7f453b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f453c0601e0 0x7f453c1087d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:37836/0 (socket says 172.21.15.155:37836) 2024-02-10T09:04:18.017 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:18.014+0000 7f453b7fe700 1 -- 172.21.15.155:0/1572522935 learned_addr learned my addr 172.21.15.155:0/1572522935 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:04:18.017 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:18.014+0000 7f453b7fe700 1 -- 172.21.15.155:0/1572522935 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f452c005040 con 0x7f453c0601e0 2024-02-10T09:04:18.017 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:18.014+0000 7f453b7fe700 1 --2- 172.21.15.155:0/1572522935 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f453c0601e0 0x7f453c1087d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f452c005010 tx=0x7f452c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:04:18.017 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:18.014+0000 7f4523fff700 1 -- 172.21.15.155:0/1572522935 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f452c009320 con 0x7f453c0601e0 2024-02-10T09:04:18.017 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:18.014+0000 7f45409e8700 1 -- 172.21.15.155:0/1572522935 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f453c108ec0 con 0x7f453c0601e0 2024-02-10T09:04:18.017 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:18.014+0000 7f4523fff700 1 -- 172.21.15.155:0/1572522935 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f452c013380 con 0x7f453c0601e0 2024-02-10T09:04:18.017 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:18.014+0000 7f4523fff700 1 -- 172.21.15.155:0/1572522935 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f452c014e50 con 0x7f453c0601e0 2024-02-10T09:04:18.017 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:18.014+0000 7f45409e8700 1 -- 172.21.15.155:0/1572522935 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f453c109350 con 0x7f453c0601e0 2024-02-10T09:04:18.019 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:18.015+0000 7f4523fff700 1 -- 172.21.15.155:0/1572522935 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f452c0263a0 con 0x7f453c0601e0 2024-02-10T09:04:18.019 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:18.015+0000 7f4523fff700 1 --2- 172.21.15.155:0/1572522935 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f452402ff10 0x7f45240323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:04:18.019 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:18.015+0000 7f4523fff700 1 -- 172.21.15.155:0/1572522935 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f452c040e80 con 0x7f453c0601e0 2024-02-10T09:04:18.019 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:18.015+0000 7f45409e8700 1 -- 172.21.15.155:0/1572522935 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f453c045ad0 con 0x7f453c0601e0 2024-02-10T09:04:18.019 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:18.015+0000 7f453affd700 1 --2- 172.21.15.155:0/1572522935 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f452402ff10 0x7f45240323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:04:18.019 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:18.016+0000 7f453affd700 1 --2- 172.21.15.155:0/1572522935 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f452402ff10 0x7f45240323d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f45300096a0 tx=0x7f4530006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:04:18.021 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:18.020+0000 7f4523fff700 1 -- 172.21.15.155:0/1572522935 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f452c012920 con 0x7f453c0601e0 2024-02-10T09:04:18.228 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:18.227+0000 7f45409e8700 1 -- 172.21.15.155:0/1572522935 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f453c10a0a0 con 0x7f453c0601e0 2024-02-10T09:04:18.229 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:18.228+0000 7f4523fff700 1 -- 172.21.15.155:0/1572522935 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f452c02b020 con 0x7f453c0601e0 2024-02-10T09:04:18.229 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:04:18.229 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:04:18.231 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:18.230+0000 7f45409e8700 1 -- 172.21.15.155:0/1572522935 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f452402ff10 msgr2=0x7f45240323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:04:18.231 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:18.230+0000 7f45409e8700 1 --2- 172.21.15.155:0/1572522935 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f452402ff10 0x7f45240323d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f45300096a0 tx=0x7f4530006b40).stop 2024-02-10T09:04:18.232 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:18.230+0000 7f45409e8700 1 -- 172.21.15.155:0/1572522935 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f453c0601e0 msgr2=0x7f453c1087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:04:18.232 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:18.230+0000 7f45409e8700 1 --2- 172.21.15.155:0/1572522935 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f453c0601e0 0x7f453c1087d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f452c005010 tx=0x7f452c00b040).stop 2024-02-10T09:04:18.232 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:18.230+0000 7f45409e8700 1 -- 172.21.15.155:0/1572522935 shutdown_connections 2024-02-10T09:04:18.232 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:18.230+0000 7f45409e8700 1 --2- 172.21.15.155:0/1572522935 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f452402ff10 0x7f45240323d0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:04:18.232 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:18.230+0000 7f45409e8700 1 --2- 172.21.15.155:0/1572522935 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f453c0601e0 0x7f453c1087d0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:04:18.232 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:18.230+0000 7f45409e8700 1 -- 172.21.15.155:0/1572522935 >> 172.21.15.155:0/1572522935 conn(0x7f453c0f09a0 msgr2=0x7f453c0f1390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:04:18.232 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:18.230+0000 7f45409e8700 1 -- 172.21.15.155:0/1572522935 shutdown_connections 2024-02-10T09:04:18.232 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:18.230+0000 7f45409e8700 1 -- 172.21.15.155:0/1572522935 wait complete. 2024-02-10T09:04:18.233 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:04:19.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:04:18 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:04:18.229681+0000 mon.smithi077 (mon.0) 209 : audit [DBG] from='client.? 172.21.15.155:0/1572522935' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:04:19.752 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:04:19 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:04:19.006211+0000 mgr.smithi077.dxntvw (mgr.14162) 171 : cluster [DBG] pgmap v160: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:04:19.752 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:04:19 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:04:19.645831+0000 mon.smithi077 (mon.0) 210 : audit [INF] from='mgr.14162 172.21.15.77:0/2029106299' entity='mgr.smithi077.dxntvw' 2024-02-10T09:04:19.855 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:04:19.856 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:04:21.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:04:20 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cephadm 2024-02-10T09:04:19.647097+0000 mgr.smithi077.dxntvw (mgr.14162) 172 : cephadm [INF] Deploying daemon node-exporter.smithi077 on smithi077 2024-02-10T09:04:21.959 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:21.957+0000 7fa276a0b700 1 -- 172.21.15.155:0/305347535 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2700f6510 msgr2=0x7fa2700f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:04:21.959 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:21.957+0000 7fa276a0b700 1 --2- 172.21.15.155:0/305347535 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2700f6510 0x7fa2700f6930 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7fa260009fb0 tx=0x7fa260009b30).stop 2024-02-10T09:04:21.959 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:21.957+0000 7fa276a0b700 1 -- 172.21.15.155:0/305347535 shutdown_connections 2024-02-10T09:04:21.959 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:21.957+0000 7fa276a0b700 1 --2- 172.21.15.155:0/305347535 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2700f6510 0x7fa2700f6930 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:04:21.959 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:21.957+0000 7fa276a0b700 1 -- 172.21.15.155:0/305347535 >> 172.21.15.155:0/305347535 conn(0x7fa2700f1c70 msgr2=0x7fa2700f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:04:21.959 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:21.958+0000 7fa276a0b700 1 -- 172.21.15.155:0/305347535 shutdown_connections 2024-02-10T09:04:21.959 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:21.958+0000 7fa276a0b700 1 -- 172.21.15.155:0/305347535 wait complete. 2024-02-10T09:04:21.959 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:21.958+0000 7fa276a0b700 1 Processor -- start 2024-02-10T09:04:21.961 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:21.958+0000 7fa276a0b700 1 -- start start 2024-02-10T09:04:21.961 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:21.959+0000 7fa276a0b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2700f6510 0x7fa27006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:04:21.961 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:21.959+0000 7fa276a0b700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa270069e20 con 0x7fa2700f6510 2024-02-10T09:04:21.961 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:21.959+0000 7fa26ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2700f6510 0x7fa27006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:04:21.962 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:21.959+0000 7fa26ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2700f6510 0x7fa27006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:37858/0 (socket says 172.21.15.155:37858) 2024-02-10T09:04:21.962 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:21.959+0000 7fa26ffff700 1 -- 172.21.15.155:0/2166984881 learned_addr learned my addr 172.21.15.155:0/2166984881 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:04:21.962 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:21.959+0000 7fa26ffff700 1 -- 172.21.15.155:0/2166984881 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa260005040 con 0x7fa2700f6510 2024-02-10T09:04:21.962 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:21.959+0000 7fa26ffff700 1 --2- 172.21.15.155:0/2166984881 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2700f6510 0x7fa27006b9a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7fa260005010 tx=0x7fa26000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:04:21.962 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:21.959+0000 7fa26cff9700 1 -- 172.21.15.155:0/2166984881 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa260009320 con 0x7fa2700f6510 2024-02-10T09:04:21.962 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:21.959+0000 7fa276a0b700 1 -- 172.21.15.155:0/2166984881 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa270069fc0 con 0x7fa2700f6510 2024-02-10T09:04:21.962 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:21.959+0000 7fa26cff9700 1 -- 172.21.15.155:0/2166984881 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa260013380 con 0x7fa2700f6510 2024-02-10T09:04:21.962 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:21.959+0000 7fa26cff9700 1 -- 172.21.15.155:0/2166984881 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa260014e50 con 0x7fa2700f6510 2024-02-10T09:04:21.962 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:21.959+0000 7fa276a0b700 1 -- 172.21.15.155:0/2166984881 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa27006a450 con 0x7fa2700f6510 2024-02-10T09:04:21.962 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:21.960+0000 7fa26cff9700 1 -- 172.21.15.155:0/2166984881 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa2600095f0 con 0x7fa2700f6510 2024-02-10T09:04:21.962 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:21.960+0000 7fa26cff9700 1 --2- 172.21.15.155:0/2166984881 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fa25802ff10 0x7fa2580323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:04:21.962 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:21.960+0000 7fa276a0b700 1 -- 172.21.15.155:0/2166984881 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa270059100 con 0x7fa2700f6510 2024-02-10T09:04:21.962 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:21.960+0000 7fa26cff9700 1 -- 172.21.15.155:0/2166984881 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa260040e90 con 0x7fa2700f6510 2024-02-10T09:04:21.962 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:21.961+0000 7fa26f7fe700 1 --2- 172.21.15.155:0/2166984881 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fa25802ff10 0x7fa2580323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:04:21.963 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:21.961+0000 7fa26f7fe700 1 --2- 172.21.15.155:0/2166984881 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fa25802ff10 0x7fa2580323d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fa2640096a0 tx=0x7fa264006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:04:21.966 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:21.965+0000 7fa26cff9700 1 -- 172.21.15.155:0/2166984881 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa26001d990 con 0x7fa2700f6510 2024-02-10T09:04:22.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:04:21 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:04:21.006652+0000 mgr.smithi077.dxntvw (mgr.14162) 173 : cluster [DBG] pgmap v161: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:04:22.174 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:22.173+0000 7fa276a0b700 1 -- 172.21.15.155:0/2166984881 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa2700fc0a0 con 0x7fa2700f6510 2024-02-10T09:04:22.174 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:22.173+0000 7fa26cff9700 1 -- 172.21.15.155:0/2166984881 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa260010230 con 0x7fa2700f6510 2024-02-10T09:04:22.175 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:04:22.175 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:04:22.176 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:22.175+0000 7fa276a0b700 1 -- 172.21.15.155:0/2166984881 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fa25802ff10 msgr2=0x7fa2580323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:04:22.176 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:22.175+0000 7fa276a0b700 1 --2- 172.21.15.155:0/2166984881 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fa25802ff10 0x7fa2580323d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fa2640096a0 tx=0x7fa264006b40).stop 2024-02-10T09:04:22.177 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:22.175+0000 7fa276a0b700 1 -- 172.21.15.155:0/2166984881 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2700f6510 msgr2=0x7fa27006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:04:22.177 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:22.175+0000 7fa276a0b700 1 --2- 172.21.15.155:0/2166984881 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2700f6510 0x7fa27006b9a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7fa260005010 tx=0x7fa26000b040).stop 2024-02-10T09:04:22.177 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:22.175+0000 7fa276a0b700 1 -- 172.21.15.155:0/2166984881 shutdown_connections 2024-02-10T09:04:22.177 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:22.175+0000 7fa276a0b700 1 --2- 172.21.15.155:0/2166984881 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fa25802ff10 0x7fa2580323d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:04:22.177 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:22.175+0000 7fa276a0b700 1 --2- 172.21.15.155:0/2166984881 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa2700f6510 0x7fa27006b9a0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:04:22.177 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:22.175+0000 7fa276a0b700 1 -- 172.21.15.155:0/2166984881 >> 172.21.15.155:0/2166984881 conn(0x7fa2700f1c70 msgr2=0x7fa2700f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:04:22.177 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:22.175+0000 7fa276a0b700 1 -- 172.21.15.155:0/2166984881 shutdown_connections 2024-02-10T09:04:22.177 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:22.175+0000 7fa276a0b700 1 -- 172.21.15.155:0/2166984881 wait complete. 2024-02-10T09:04:22.177 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:04:23.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:04:22 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:04:22.174901+0000 mon.smithi077 (mon.0) 211 : audit [DBG] from='client.? 172.21.15.155:0/2166984881' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:04:23.866 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:04:23.867 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:04:24.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:04:23 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:04:23.007030+0000 mgr.smithi077.dxntvw (mgr.14162) 174 : cluster [DBG] pgmap v162: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:04:25.398 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:04:25 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:04:25.007307+0000 mgr.smithi077.dxntvw (mgr.14162) 175 : cluster [DBG] pgmap v163: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:04:26.065 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:26.062+0000 7fbb5ee81700 1 -- 172.21.15.155:0/1740068834 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb580f5410 msgr2=0x7fbb580f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:04:26.065 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:26.062+0000 7fbb5ee81700 1 --2- 172.21.15.155:0/1740068834 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb580f5410 0x7fbb580f5830 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7fbb4c009fb0 tx=0x7fbb4c009b30).stop 2024-02-10T09:04:26.066 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:26.062+0000 7fbb5ee81700 1 -- 172.21.15.155:0/1740068834 shutdown_connections 2024-02-10T09:04:26.066 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:26.062+0000 7fbb5ee81700 1 --2- 172.21.15.155:0/1740068834 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb580f5410 0x7fbb580f5830 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:04:26.066 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:26.062+0000 7fbb5ee81700 1 -- 172.21.15.155:0/1740068834 >> 172.21.15.155:0/1740068834 conn(0x7fbb580f0b70 msgr2=0x7fbb580f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:04:26.066 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:26.063+0000 7fbb5ee81700 1 -- 172.21.15.155:0/1740068834 shutdown_connections 2024-02-10T09:04:26.066 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:26.063+0000 7fbb5ee81700 1 -- 172.21.15.155:0/1740068834 wait complete. 2024-02-10T09:04:26.066 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:26.063+0000 7fbb5ee81700 1 Processor -- start 2024-02-10T09:04:26.066 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:26.063+0000 7fbb5ee81700 1 -- start start 2024-02-10T09:04:26.066 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:26.063+0000 7fbb5ee81700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb580f5410 0x7fbb581056c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:04:26.066 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:26.063+0000 7fbb5ee81700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb58105bb0 con 0x7fbb580f5410 2024-02-10T09:04:26.067 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:26.064+0000 7fbb5cc1d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb580f5410 0x7fbb581056c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:04:26.067 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:26.064+0000 7fbb5cc1d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb580f5410 0x7fbb581056c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:51056/0 (socket says 172.21.15.155:51056) 2024-02-10T09:04:26.067 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:26.064+0000 7fbb5cc1d700 1 -- 172.21.15.155:0/1141731166 learned_addr learned my addr 172.21.15.155:0/1141731166 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:04:26.067 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:26.064+0000 7fbb5cc1d700 1 -- 172.21.15.155:0/1141731166 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb4c005040 con 0x7fbb580f5410 2024-02-10T09:04:26.067 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:26.064+0000 7fbb5cc1d700 1 --2- 172.21.15.155:0/1141731166 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb580f5410 0x7fbb581056c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7fbb4c005010 tx=0x7fbb4c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:04:26.067 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:26.064+0000 7fbb557fa700 1 -- 172.21.15.155:0/1141731166 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb4c009320 con 0x7fbb580f5410 2024-02-10T09:04:26.067 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:26.064+0000 7fbb5ee81700 1 -- 172.21.15.155:0/1141731166 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb58105db0 con 0x7fbb580f5410 2024-02-10T09:04:26.067 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:26.064+0000 7fbb557fa700 1 -- 172.21.15.155:0/1141731166 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbb4c013380 con 0x7fbb580f5410 2024-02-10T09:04:26.067 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:26.064+0000 7fbb557fa700 1 -- 172.21.15.155:0/1141731166 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb4c014e50 con 0x7fbb580f5410 2024-02-10T09:04:26.067 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:26.064+0000 7fbb5ee81700 1 -- 172.21.15.155:0/1141731166 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb58106240 con 0x7fbb580f5410 2024-02-10T09:04:26.068 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:26.065+0000 7fbb557fa700 1 -- 172.21.15.155:0/1141731166 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbb4c014420 con 0x7fbb580f5410 2024-02-10T09:04:26.068 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:26.065+0000 7fbb5ee81700 1 -- 172.21.15.155:0/1141731166 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb58045ad0 con 0x7fbb580f5410 2024-02-10T09:04:26.068 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:26.065+0000 7fbb557fa700 1 --2- 172.21.15.155:0/1141731166 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fbb4802ff10 0x7fbb480323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:04:26.068 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:26.065+0000 7fbb557fa700 1 -- 172.21.15.155:0/1141731166 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbb4c040e30 con 0x7fbb580f5410 2024-02-10T09:04:26.068 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:26.066+0000 7fbb57fff700 1 --2- 172.21.15.155:0/1141731166 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fbb4802ff10 0x7fbb480323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:04:26.069 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:26.066+0000 7fbb57fff700 1 --2- 172.21.15.155:0/1141731166 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fbb4802ff10 0x7fbb480323d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fbb440096a0 tx=0x7fbb44006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:04:26.071 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:26.070+0000 7fbb557fa700 1 -- 172.21.15.155:0/1141731166 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbb4c0095f0 con 0x7fbb580f5410 2024-02-10T09:04:26.276 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:26.275+0000 7fbb5ee81700 1 -- 172.21.15.155:0/1141731166 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbb581070f0 con 0x7fbb580f5410 2024-02-10T09:04:26.277 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:26.276+0000 7fbb557fa700 1 -- 172.21.15.155:0/1141731166 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbb4c01db70 con 0x7fbb580f5410 2024-02-10T09:04:26.277 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:04:26.277 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:04:26.278 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:26.277+0000 7fbb5ee81700 1 -- 172.21.15.155:0/1141731166 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fbb4802ff10 msgr2=0x7fbb480323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:04:26.278 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:26.277+0000 7fbb5ee81700 1 --2- 172.21.15.155:0/1141731166 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fbb4802ff10 0x7fbb480323d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fbb440096a0 tx=0x7fbb44006b40).stop 2024-02-10T09:04:26.279 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:26.277+0000 7fbb5ee81700 1 -- 172.21.15.155:0/1141731166 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb580f5410 msgr2=0x7fbb581056c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:04:26.279 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:26.277+0000 7fbb5ee81700 1 --2- 172.21.15.155:0/1141731166 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb580f5410 0x7fbb581056c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7fbb4c005010 tx=0x7fbb4c00b040).stop 2024-02-10T09:04:26.279 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:26.277+0000 7fbb5ee81700 1 -- 172.21.15.155:0/1141731166 shutdown_connections 2024-02-10T09:04:26.279 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:26.277+0000 7fbb5ee81700 1 --2- 172.21.15.155:0/1141731166 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fbb4802ff10 0x7fbb480323d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:04:26.279 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:26.277+0000 7fbb5ee81700 1 --2- 172.21.15.155:0/1141731166 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb580f5410 0x7fbb581056c0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:04:26.279 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:26.277+0000 7fbb5ee81700 1 -- 172.21.15.155:0/1141731166 >> 172.21.15.155:0/1141731166 conn(0x7fbb580f0b70 msgr2=0x7fbb580f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:04:26.279 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:26.277+0000 7fbb5ee81700 1 -- 172.21.15.155:0/1141731166 shutdown_connections 2024-02-10T09:04:26.279 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:26.277+0000 7fbb5ee81700 1 -- 172.21.15.155:0/1141731166 wait complete. 2024-02-10T09:04:26.279 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:04:26.649 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:04:26 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:04:26.277192+0000 mon.smithi077 (mon.0) 212 : audit [DBG] from='client.? 172.21.15.155:0/1141731166' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:04:27.648 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:04:27 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:04:27.007558+0000 mgr.smithi077.dxntvw (mgr.14162) 176 : cluster [DBG] pgmap v164: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:04:28.071 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:04:28.071 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:04:29.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:04:29 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:04:29.007895+0000 mgr.smithi077.dxntvw (mgr.14162) 177 : cluster [DBG] pgmap v165: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:04:30.265 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:30.262+0000 7f8a8496b700 1 -- 172.21.15.155:0/2935514852 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a800f5410 msgr2=0x7f8a800f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:04:30.265 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:30.262+0000 7f8a8496b700 1 --2- 172.21.15.155:0/2935514852 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a800f5410 0x7f8a800f5830 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f8a68009fb0 tx=0x7f8a68009b30).stop 2024-02-10T09:04:30.266 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:30.262+0000 7f8a8496b700 1 -- 172.21.15.155:0/2935514852 shutdown_connections 2024-02-10T09:04:30.266 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:30.262+0000 7f8a8496b700 1 --2- 172.21.15.155:0/2935514852 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a800f5410 0x7f8a800f5830 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:04:30.266 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:30.262+0000 7f8a8496b700 1 -- 172.21.15.155:0/2935514852 >> 172.21.15.155:0/2935514852 conn(0x7f8a800f0b70 msgr2=0x7f8a800f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:04:30.266 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:30.262+0000 7f8a8496b700 1 -- 172.21.15.155:0/2935514852 shutdown_connections 2024-02-10T09:04:30.266 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:30.262+0000 7f8a8496b700 1 -- 172.21.15.155:0/2935514852 wait complete. 2024-02-10T09:04:30.266 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:30.263+0000 7f8a8496b700 1 Processor -- start 2024-02-10T09:04:30.266 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:30.263+0000 7f8a8496b700 1 -- start start 2024-02-10T09:04:30.266 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:30.263+0000 7f8a8496b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a800f5410 0x7f8a800fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:04:30.266 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:30.263+0000 7f8a8496b700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a800fb960 con 0x7f8a800f5410 2024-02-10T09:04:30.266 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:30.263+0000 7f8a7e59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a800f5410 0x7f8a800fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:04:30.266 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:30.263+0000 7f8a7e59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a800f5410 0x7f8a800fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:51076/0 (socket says 172.21.15.155:51076) 2024-02-10T09:04:30.267 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:30.263+0000 7f8a7e59c700 1 -- 172.21.15.155:0/2619254556 learned_addr learned my addr 172.21.15.155:0/2619254556 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:04:30.267 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:30.263+0000 7f8a7e59c700 1 -- 172.21.15.155:0/2619254556 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a68005040 con 0x7f8a800f5410 2024-02-10T09:04:30.267 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:30.264+0000 7f8a7e59c700 1 --2- 172.21.15.155:0/2619254556 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a800f5410 0x7f8a800fd4e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f8a680089f0 tx=0x7f8a6800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:04:30.267 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:30.264+0000 7f8a76ffd700 1 -- 172.21.15.155:0/2619254556 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a68009320 con 0x7f8a800f5410 2024-02-10T09:04:30.267 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:30.264+0000 7f8a8496b700 1 -- 172.21.15.155:0/2619254556 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a800fbb00 con 0x7f8a800f5410 2024-02-10T09:04:30.267 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:30.264+0000 7f8a76ffd700 1 -- 172.21.15.155:0/2619254556 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8a68013380 con 0x7f8a800f5410 2024-02-10T09:04:30.267 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:30.264+0000 7f8a76ffd700 1 -- 172.21.15.155:0/2619254556 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a6801ed80 con 0x7f8a800f5410 2024-02-10T09:04:30.267 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:30.264+0000 7f8a8496b700 1 -- 172.21.15.155:0/2619254556 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a800fbf90 con 0x7f8a800f5410 2024-02-10T09:04:30.268 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:30.265+0000 7f8a8496b700 1 -- 172.21.15.155:0/2619254556 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a80059100 con 0x7f8a800f5410 2024-02-10T09:04:30.268 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:30.265+0000 7f8a76ffd700 1 -- 172.21.15.155:0/2619254556 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8a6801e420 con 0x7f8a800f5410 2024-02-10T09:04:30.268 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:30.265+0000 7f8a76ffd700 1 --2- 172.21.15.155:0/2619254556 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f8a6c02ff60 0x7f8a6c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:04:30.268 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:30.265+0000 7f8a76ffd700 1 -- 172.21.15.155:0/2619254556 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8a68041090 con 0x7f8a800f5410 2024-02-10T09:04:30.268 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:30.265+0000 7f8a7dd9b700 1 --2- 172.21.15.155:0/2619254556 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f8a6c02ff60 0x7f8a6c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:04:30.268 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:30.266+0000 7f8a7dd9b700 1 --2- 172.21.15.155:0/2619254556 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f8a6c02ff60 0x7f8a6c032420 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f8a700096a0 tx=0x7f8a70006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:04:30.271 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:30.270+0000 7f8a76ffd700 1 -- 172.21.15.155:0/2619254556 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8a680095f0 con 0x7f8a800f5410 2024-02-10T09:04:30.479 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:30.478+0000 7f8a8496b700 1 -- 172.21.15.155:0/2619254556 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8a800fafa0 con 0x7f8a800f5410 2024-02-10T09:04:30.480 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:30.479+0000 7f8a76ffd700 1 -- 172.21.15.155:0/2619254556 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8a68004490 con 0x7f8a800f5410 2024-02-10T09:04:30.480 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:04:30.480 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:04:30.482 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:30.481+0000 7f8a8496b700 1 -- 172.21.15.155:0/2619254556 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f8a6c02ff60 msgr2=0x7f8a6c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:04:30.482 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:30.481+0000 7f8a8496b700 1 --2- 172.21.15.155:0/2619254556 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f8a6c02ff60 0x7f8a6c032420 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f8a700096a0 tx=0x7f8a70006b40).stop 2024-02-10T09:04:30.482 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:30.481+0000 7f8a8496b700 1 -- 172.21.15.155:0/2619254556 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a800f5410 msgr2=0x7f8a800fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:04:30.482 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:30.481+0000 7f8a8496b700 1 --2- 172.21.15.155:0/2619254556 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a800f5410 0x7f8a800fd4e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f8a680089f0 tx=0x7f8a6800b040).stop 2024-02-10T09:04:30.482 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:30.481+0000 7f8a8496b700 1 -- 172.21.15.155:0/2619254556 shutdown_connections 2024-02-10T09:04:30.482 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:30.481+0000 7f8a8496b700 1 --2- 172.21.15.155:0/2619254556 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f8a6c02ff60 0x7f8a6c032420 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:04:30.482 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:30.481+0000 7f8a8496b700 1 --2- 172.21.15.155:0/2619254556 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a800f5410 0x7f8a800fd4e0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:04:30.482 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:30.481+0000 7f8a8496b700 1 -- 172.21.15.155:0/2619254556 >> 172.21.15.155:0/2619254556 conn(0x7f8a800f0b70 msgr2=0x7f8a8005fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:04:30.482 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:30.481+0000 7f8a8496b700 1 -- 172.21.15.155:0/2619254556 shutdown_connections 2024-02-10T09:04:30.482 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:30.481+0000 7f8a8496b700 1 -- 172.21.15.155:0/2619254556 wait complete. 2024-02-10T09:04:30.483 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:04:30.899 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:04:30 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:04:30.480429+0000 mon.smithi077 (mon.0) 213 : audit [DBG] from='client.? 172.21.15.155:0/2619254556' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:04:31.899 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:04:31 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:04:31.008255+0000 mgr.smithi077.dxntvw (mgr.14162) 178 : cluster [DBG] pgmap v166: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:04:32.361 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:04:32.361 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:04:34.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:04:33 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:04:33.008645+0000 mgr.smithi077.dxntvw (mgr.14162) 179 : cluster [DBG] pgmap v167: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:04:34.426 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:34.424+0000 7f1690e4a700 1 -- 172.21.15.155:0/2062120255 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f168c0f5410 msgr2=0x7f168c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:04:34.437 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:34.424+0000 7f1690e4a700 1 --2- 172.21.15.155:0/2062120255 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f168c0f5410 0x7f168c0f5830 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f1680009fb0 tx=0x7f1680009b30).stop 2024-02-10T09:04:34.437 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:34.424+0000 7f1690e4a700 1 -- 172.21.15.155:0/2062120255 shutdown_connections 2024-02-10T09:04:34.437 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:34.424+0000 7f1690e4a700 1 --2- 172.21.15.155:0/2062120255 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f168c0f5410 0x7f168c0f5830 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:04:34.437 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:34.424+0000 7f1690e4a700 1 -- 172.21.15.155:0/2062120255 >> 172.21.15.155:0/2062120255 conn(0x7f168c0f0b70 msgr2=0x7f168c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:04:34.437 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:34.424+0000 7f1690e4a700 1 -- 172.21.15.155:0/2062120255 shutdown_connections 2024-02-10T09:04:34.437 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:34.424+0000 7f1690e4a700 1 -- 172.21.15.155:0/2062120255 wait complete. 2024-02-10T09:04:34.437 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:34.425+0000 7f1690e4a700 1 Processor -- start 2024-02-10T09:04:34.437 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:34.425+0000 7f1690e4a700 1 -- start start 2024-02-10T09:04:34.437 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:34.425+0000 7f1690e4a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f168c0f5410 0x7f168c1067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:04:34.438 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:34.425+0000 7f1690e4a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f168c106cc0 con 0x7f168c0f5410 2024-02-10T09:04:34.439 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:34.425+0000 7f168a59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f168c0f5410 0x7f168c1067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:04:34.440 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:34.425+0000 7f168a59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f168c0f5410 0x7f168c1067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:48958/0 (socket says 172.21.15.155:48958) 2024-02-10T09:04:34.440 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:34.425+0000 7f168a59c700 1 -- 172.21.15.155:0/226998839 learned_addr learned my addr 172.21.15.155:0/226998839 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:04:34.440 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:34.425+0000 7f168a59c700 1 -- 172.21.15.155:0/226998839 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1680005040 con 0x7f168c0f5410 2024-02-10T09:04:34.440 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:34.426+0000 7f168a59c700 1 --2- 172.21.15.155:0/226998839 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f168c0f5410 0x7f168c1067d0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f16800089f0 tx=0x7f168000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:04:34.440 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:34.426+0000 7f167affd700 1 -- 172.21.15.155:0/226998839 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1680009320 con 0x7f168c0f5410 2024-02-10T09:04:34.440 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:34.426+0000 7f1690e4a700 1 -- 172.21.15.155:0/226998839 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f168c106ec0 con 0x7f168c0f5410 2024-02-10T09:04:34.440 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:34.426+0000 7f167affd700 1 -- 172.21.15.155:0/226998839 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1680013380 con 0x7f168c0f5410 2024-02-10T09:04:34.440 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:34.426+0000 7f167affd700 1 -- 172.21.15.155:0/226998839 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f168001ed80 con 0x7f168c0f5410 2024-02-10T09:04:34.440 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:34.426+0000 7f1690e4a700 1 -- 172.21.15.155:0/226998839 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f168c107350 con 0x7f168c0f5410 2024-02-10T09:04:34.440 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:34.427+0000 7f167affd700 1 -- 172.21.15.155:0/226998839 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f168001e420 con 0x7f168c0f5410 2024-02-10T09:04:34.440 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:34.427+0000 7f1690e4a700 1 -- 172.21.15.155:0/226998839 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f168c045ad0 con 0x7f168c0f5410 2024-02-10T09:04:34.440 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:34.427+0000 7f167affd700 1 --2- 172.21.15.155:0/226998839 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f167402ffb0 0x7f1674032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:04:34.440 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:34.427+0000 7f167affd700 1 -- 172.21.15.155:0/226998839 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1680040ff0 con 0x7f168c0f5410 2024-02-10T09:04:34.441 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:34.427+0000 7f1689d9b700 1 --2- 172.21.15.155:0/226998839 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f167402ffb0 0x7f1674032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:04:34.441 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:34.428+0000 7f1689d9b700 1 --2- 172.21.15.155:0/226998839 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f167402ffb0 0x7f1674032470 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f167c0096a0 tx=0x7f167c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:04:34.441 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:34.432+0000 7f167affd700 1 -- 172.21.15.155:0/226998839 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f16800095f0 con 0x7f168c0f5410 2024-02-10T09:04:34.640 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:34.638+0000 7f1690e4a700 1 -- 172.21.15.155:0/226998839 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f168c108200 con 0x7f168c0f5410 2024-02-10T09:04:34.640 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:34.639+0000 7f167affd700 1 -- 172.21.15.155:0/226998839 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f168001cb10 con 0x7f168c0f5410 2024-02-10T09:04:34.640 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:04:34.641 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:04:34.642 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:34.641+0000 7f1690e4a700 1 -- 172.21.15.155:0/226998839 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f167402ffb0 msgr2=0x7f1674032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:04:34.642 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:34.641+0000 7f1690e4a700 1 --2- 172.21.15.155:0/226998839 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f167402ffb0 0x7f1674032470 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f167c0096a0 tx=0x7f167c006b40).stop 2024-02-10T09:04:34.642 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:34.641+0000 7f1690e4a700 1 -- 172.21.15.155:0/226998839 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f168c0f5410 msgr2=0x7f168c1067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:04:34.642 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:34.641+0000 7f1690e4a700 1 --2- 172.21.15.155:0/226998839 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f168c0f5410 0x7f168c1067d0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f16800089f0 tx=0x7f168000b040).stop 2024-02-10T09:04:34.642 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:34.641+0000 7f1690e4a700 1 -- 172.21.15.155:0/226998839 shutdown_connections 2024-02-10T09:04:34.642 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:34.641+0000 7f1690e4a700 1 --2- 172.21.15.155:0/226998839 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f167402ffb0 0x7f1674032470 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:04:34.642 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:34.641+0000 7f1690e4a700 1 --2- 172.21.15.155:0/226998839 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f168c0f5410 0x7f168c1067d0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:04:34.642 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:34.641+0000 7f1690e4a700 1 -- 172.21.15.155:0/226998839 >> 172.21.15.155:0/226998839 conn(0x7f168c0f0b70 msgr2=0x7f168c05fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:04:34.642 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:34.641+0000 7f1690e4a700 1 -- 172.21.15.155:0/226998839 shutdown_connections 2024-02-10T09:04:34.643 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:34.641+0000 7f1690e4a700 1 -- 172.21.15.155:0/226998839 wait complete. 2024-02-10T09:04:34.643 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:04:35.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:04:34 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:04:34.640562+0000 mon.smithi077 (mon.0) 214 : audit [DBG] from='client.? 172.21.15.155:0/226998839' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:04:36.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:04:35 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:04:35.008991+0000 mgr.smithi077.dxntvw (mgr.14162) 180 : cluster [DBG] pgmap v168: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:04:36.407 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:04:36.408 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:04:37.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:04:37 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:04:37.009354+0000 mgr.smithi077.dxntvw (mgr.14162) 181 : cluster [DBG] pgmap v169: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:04:38.643 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:38.642+0000 7f6cd9b5d700 1 -- 172.21.15.155:0/967445511 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6cd40f51a0 msgr2=0x7f6cd40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:04:38.644 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:38.642+0000 7f6cd9b5d700 1 --2- 172.21.15.155:0/967445511 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6cd40f51a0 0x7f6cd40f55c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f6cc8009fb0 tx=0x7f6cc8009b30).stop 2024-02-10T09:04:38.644 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:38.642+0000 7f6cd9b5d700 1 -- 172.21.15.155:0/967445511 shutdown_connections 2024-02-10T09:04:38.644 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:38.642+0000 7f6cd9b5d700 1 --2- 172.21.15.155:0/967445511 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6cd40f51a0 0x7f6cd40f55c0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:04:38.644 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:38.642+0000 7f6cd9b5d700 1 -- 172.21.15.155:0/967445511 >> 172.21.15.155:0/967445511 conn(0x7f6cd40f09a0 msgr2=0x7f6cd40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:04:38.644 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:38.642+0000 7f6cd9b5d700 1 -- 172.21.15.155:0/967445511 shutdown_connections 2024-02-10T09:04:38.644 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:38.642+0000 7f6cd9b5d700 1 -- 172.21.15.155:0/967445511 wait complete. 2024-02-10T09:04:38.644 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:38.642+0000 7f6cd9b5d700 1 Processor -- start 2024-02-10T09:04:38.644 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:38.642+0000 7f6cd9b5d700 1 -- start start 2024-02-10T09:04:38.644 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:38.643+0000 7f6cd9b5d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6cd40f51a0 0x7f6cd4106650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:04:38.644 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:38.643+0000 7f6cd9b5d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd4106b40 con 0x7f6cd40f51a0 2024-02-10T09:04:38.646 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:38.643+0000 7f6cd8b5b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6cd40f51a0 0x7f6cd4106650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:04:38.646 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:38.643+0000 7f6cd8b5b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6cd40f51a0 0x7f6cd4106650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:48984/0 (socket says 172.21.15.155:48984) 2024-02-10T09:04:38.646 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:38.643+0000 7f6cd8b5b700 1 -- 172.21.15.155:0/1682624403 learned_addr learned my addr 172.21.15.155:0/1682624403 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:04:38.646 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:38.643+0000 7f6cd8b5b700 1 -- 172.21.15.155:0/1682624403 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cc8005040 con 0x7f6cd40f51a0 2024-02-10T09:04:38.646 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:38.643+0000 7f6cd8b5b700 1 --2- 172.21.15.155:0/1682624403 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6cd40f51a0 0x7f6cd4106650 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f6cc8009f80 tx=0x7f6cc800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:04:38.646 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:38.644+0000 7f6cd17fa700 1 -- 172.21.15.155:0/1682624403 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6cc80095f0 con 0x7f6cd40f51a0 2024-02-10T09:04:38.647 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:38.644+0000 7f6cd9b5d700 1 -- 172.21.15.155:0/1682624403 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cd4106d40 con 0x7f6cd40f51a0 2024-02-10T09:04:38.647 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:38.644+0000 7f6cd17fa700 1 -- 172.21.15.155:0/1682624403 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6cc8007e90 con 0x7f6cd40f51a0 2024-02-10T09:04:38.647 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:38.644+0000 7f6cd17fa700 1 -- 172.21.15.155:0/1682624403 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6cc8014d80 con 0x7f6cd40f51a0 2024-02-10T09:04:38.647 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:38.644+0000 7f6cd9b5d700 1 -- 172.21.15.155:0/1682624403 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cd41071d0 con 0x7f6cd40f51a0 2024-02-10T09:04:38.647 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:38.644+0000 7f6cd17fa700 1 -- 172.21.15.155:0/1682624403 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6cc8025070 con 0x7f6cd40f51a0 2024-02-10T09:04:38.647 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:38.644+0000 7f6cd9b5d700 1 -- 172.21.15.155:0/1682624403 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cd4045ad0 con 0x7f6cd40f51a0 2024-02-10T09:04:38.647 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:38.645+0000 7f6cd17fa700 1 --2- 172.21.15.155:0/1682624403 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f6cc402ff60 0x7f6cc4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:04:38.647 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:38.645+0000 7f6cd17fa700 1 -- 172.21.15.155:0/1682624403 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6cc80408f0 con 0x7f6cd40f51a0 2024-02-10T09:04:38.647 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:38.645+0000 7f6cd3fff700 1 --2- 172.21.15.155:0/1682624403 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f6cc402ff60 0x7f6cc4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:04:38.647 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:38.645+0000 7f6cd3fff700 1 --2- 172.21.15.155:0/1682624403 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f6cc402ff60 0x7f6cc4032420 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f6cc00096a0 tx=0x7f6cc0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:04:38.650 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:38.649+0000 7f6cd17fa700 1 -- 172.21.15.155:0/1682624403 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6cc800f4e0 con 0x7f6cd40f51a0 2024-02-10T09:04:38.854 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:38.853+0000 7f6cd9b5d700 1 -- 172.21.15.155:0/1682624403 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6cd4108080 con 0x7f6cd40f51a0 2024-02-10T09:04:38.854 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:38.853+0000 7f6cd17fa700 1 -- 172.21.15.155:0/1682624403 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6cc800f300 con 0x7f6cd40f51a0 2024-02-10T09:04:38.854 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:04:38.855 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:04:38.856 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:38.855+0000 7f6cd9b5d700 1 -- 172.21.15.155:0/1682624403 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f6cc402ff60 msgr2=0x7f6cc4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:04:38.856 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:38.855+0000 7f6cd9b5d700 1 --2- 172.21.15.155:0/1682624403 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f6cc402ff60 0x7f6cc4032420 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f6cc00096a0 tx=0x7f6cc0006b40).stop 2024-02-10T09:04:38.856 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:38.855+0000 7f6cd9b5d700 1 -- 172.21.15.155:0/1682624403 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6cd40f51a0 msgr2=0x7f6cd4106650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:04:38.856 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:38.855+0000 7f6cd9b5d700 1 --2- 172.21.15.155:0/1682624403 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6cd40f51a0 0x7f6cd4106650 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f6cc8009f80 tx=0x7f6cc800b040).stop 2024-02-10T09:04:38.856 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:38.855+0000 7f6cd9b5d700 1 -- 172.21.15.155:0/1682624403 shutdown_connections 2024-02-10T09:04:38.856 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:38.855+0000 7f6cd9b5d700 1 --2- 172.21.15.155:0/1682624403 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f6cc402ff60 0x7f6cc4032420 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:04:38.857 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:38.855+0000 7f6cd9b5d700 1 --2- 172.21.15.155:0/1682624403 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6cd40f51a0 0x7f6cd4106650 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:04:38.857 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:38.855+0000 7f6cd9b5d700 1 -- 172.21.15.155:0/1682624403 >> 172.21.15.155:0/1682624403 conn(0x7f6cd40f09a0 msgr2=0x7f6cd405fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:04:38.857 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:38.855+0000 7f6cd9b5d700 1 -- 172.21.15.155:0/1682624403 shutdown_connections 2024-02-10T09:04:38.857 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:38.855+0000 7f6cd9b5d700 1 -- 172.21.15.155:0/1682624403 wait complete. 2024-02-10T09:04:38.857 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:04:39.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:04:38 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:04:38.854990+0000 mon.smithi077 (mon.0) 215 : audit [DBG] from='client.? 172.21.15.155:0/1682624403' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:04:40.400 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:04:39 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:04:39.009695+0000 mgr.smithi077.dxntvw (mgr.14162) 182 : cluster [DBG] pgmap v170: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:04:40.697 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:04:40.697 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:04:41.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:04:41 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:04:41.010055+0000 mgr.smithi077.dxntvw (mgr.14162) 183 : cluster [DBG] pgmap v171: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:04:42.745 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:42.743+0000 7ffb1c8ec700 1 -- 172.21.15.155:0/572938489 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb180f5410 msgr2=0x7ffb180f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:04:42.745 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:42.743+0000 7ffb1c8ec700 1 --2- 172.21.15.155:0/572938489 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb180f5410 0x7ffb180f5830 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7ffb00009fb0 tx=0x7ffb00009b30).stop 2024-02-10T09:04:42.745 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:42.744+0000 7ffb1c8ec700 1 -- 172.21.15.155:0/572938489 shutdown_connections 2024-02-10T09:04:42.745 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:42.744+0000 7ffb1c8ec700 1 --2- 172.21.15.155:0/572938489 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb180f5410 0x7ffb180f5830 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:04:42.745 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:42.744+0000 7ffb1c8ec700 1 -- 172.21.15.155:0/572938489 >> 172.21.15.155:0/572938489 conn(0x7ffb180f0b70 msgr2=0x7ffb180f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:04:42.745 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:42.744+0000 7ffb1c8ec700 1 -- 172.21.15.155:0/572938489 shutdown_connections 2024-02-10T09:04:42.746 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:42.744+0000 7ffb1c8ec700 1 -- 172.21.15.155:0/572938489 wait complete. 2024-02-10T09:04:42.748 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:42.744+0000 7ffb1c8ec700 1 Processor -- start 2024-02-10T09:04:42.748 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:42.744+0000 7ffb1c8ec700 1 -- start start 2024-02-10T09:04:42.748 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:42.744+0000 7ffb1c8ec700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb180f5410 0x7ffb181094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:04:42.748 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:42.745+0000 7ffb1c8ec700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb18060240 con 0x7ffb180f5410 2024-02-10T09:04:42.748 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:42.745+0000 7ffb1659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb180f5410 0x7ffb181094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:04:42.748 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:42.745+0000 7ffb1659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb180f5410 0x7ffb181094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:50558/0 (socket says 172.21.15.155:50558) 2024-02-10T09:04:42.748 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:42.745+0000 7ffb1659c700 1 -- 172.21.15.155:0/1719605897 learned_addr learned my addr 172.21.15.155:0/1719605897 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:04:42.748 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:42.745+0000 7ffb1659c700 1 -- 172.21.15.155:0/1719605897 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb00005040 con 0x7ffb180f5410 2024-02-10T09:04:42.749 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:42.745+0000 7ffb1659c700 1 --2- 172.21.15.155:0/1719605897 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb180f5410 0x7ffb181094e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7ffb00005010 tx=0x7ffb0000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:04:42.749 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:42.745+0000 7ffb0effd700 1 -- 172.21.15.155:0/1719605897 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb00009320 con 0x7ffb180f5410 2024-02-10T09:04:42.749 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:42.745+0000 7ffb0effd700 1 -- 172.21.15.155:0/1719605897 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffb00013380 con 0x7ffb180f5410 2024-02-10T09:04:42.749 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:42.745+0000 7ffb0effd700 1 -- 172.21.15.155:0/1719605897 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb00014e50 con 0x7ffb180f5410 2024-02-10T09:04:42.749 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:42.745+0000 7ffb1c8ec700 1 -- 172.21.15.155:0/1719605897 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb18060440 con 0x7ffb180f5410 2024-02-10T09:04:42.749 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:42.745+0000 7ffb1c8ec700 1 -- 172.21.15.155:0/1719605897 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb18109b90 con 0x7ffb180f5410 2024-02-10T09:04:42.749 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:42.746+0000 7ffb0effd700 1 -- 172.21.15.155:0/1719605897 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ffb000095f0 con 0x7ffb180f5410 2024-02-10T09:04:42.750 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:42.746+0000 7ffb1c8ec700 1 -- 172.21.15.155:0/1719605897 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb18045ad0 con 0x7ffb180f5410 2024-02-10T09:04:42.750 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:42.746+0000 7ffb0effd700 1 --2- 172.21.15.155:0/1719605897 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7ffb0402ff10 0x7ffb040323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:04:42.750 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:42.746+0000 7ffb0effd700 1 -- 172.21.15.155:0/1719605897 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ffb00040f80 con 0x7ffb180f5410 2024-02-10T09:04:42.750 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:42.746+0000 7ffb15d9b700 1 --2- 172.21.15.155:0/1719605897 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7ffb0402ff10 0x7ffb040323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:04:42.750 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:42.747+0000 7ffb15d9b700 1 --2- 172.21.15.155:0/1719605897 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7ffb0402ff10 0x7ffb040323d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7ffb080096a0 tx=0x7ffb08006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:04:42.752 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:42.752+0000 7ffb0effd700 1 -- 172.21.15.155:0/1719605897 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ffb00014420 con 0x7ffb180f5410 2024-02-10T09:04:42.962 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:42.961+0000 7ffb1c8ec700 1 -- 172.21.15.155:0/1719605897 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ffb1810a3b0 con 0x7ffb180f5410 2024-02-10T09:04:42.963 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:42.962+0000 7ffb0effd700 1 -- 172.21.15.155:0/1719605897 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ffb00010230 con 0x7ffb180f5410 2024-02-10T09:04:42.963 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:04:42.963 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:04:42.964 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:42.963+0000 7ffb1c8ec700 1 -- 172.21.15.155:0/1719605897 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7ffb0402ff10 msgr2=0x7ffb040323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:04:42.965 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:42.963+0000 7ffb1c8ec700 1 --2- 172.21.15.155:0/1719605897 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7ffb0402ff10 0x7ffb040323d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7ffb080096a0 tx=0x7ffb08006b40).stop 2024-02-10T09:04:42.965 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:42.963+0000 7ffb1c8ec700 1 -- 172.21.15.155:0/1719605897 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb180f5410 msgr2=0x7ffb181094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:04:42.965 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:42.963+0000 7ffb1c8ec700 1 --2- 172.21.15.155:0/1719605897 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb180f5410 0x7ffb181094e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7ffb00005010 tx=0x7ffb0000b040).stop 2024-02-10T09:04:42.965 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:42.963+0000 7ffb1c8ec700 1 -- 172.21.15.155:0/1719605897 shutdown_connections 2024-02-10T09:04:42.965 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:42.963+0000 7ffb1c8ec700 1 --2- 172.21.15.155:0/1719605897 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7ffb0402ff10 0x7ffb040323d0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:04:42.965 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:42.963+0000 7ffb1c8ec700 1 --2- 172.21.15.155:0/1719605897 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb180f5410 0x7ffb181094e0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:04:42.965 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:42.963+0000 7ffb1c8ec700 1 -- 172.21.15.155:0/1719605897 >> 172.21.15.155:0/1719605897 conn(0x7ffb180f0b70 msgr2=0x7ffb180f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:04:42.965 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:42.963+0000 7ffb1c8ec700 1 -- 172.21.15.155:0/1719605897 shutdown_connections 2024-02-10T09:04:42.965 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:42.963+0000 7ffb1c8ec700 1 -- 172.21.15.155:0/1719605897 wait complete. 2024-02-10T09:04:42.965 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:04:44.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:04:43 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:04:42.963096+0000 mon.smithi077 (mon.0) 216 : audit [DBG] from='client.? 172.21.15.155:0/1719605897' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:04:44.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:04:43 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:04:43.010398+0000 mgr.smithi077.dxntvw (mgr.14162) 184 : cluster [DBG] pgmap v172: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:04:44.756 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:04:44.756 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:04:45.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:04:45 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:04:45.010712+0000 mgr.smithi077.dxntvw (mgr.14162) 185 : cluster [DBG] pgmap v173: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:04:46.898 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:46.897+0000 7f1ad4813700 1 -- 172.21.15.155:0/3133694236 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ad00f6510 msgr2=0x7f1ad00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:04:46.899 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:46.897+0000 7f1ad4813700 1 --2- 172.21.15.155:0/3133694236 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ad00f6510 0x7f1ad00f6930 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f1ac4009fb0 tx=0x7f1ac4009b30).stop 2024-02-10T09:04:46.899 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:46.897+0000 7f1ad4813700 1 -- 172.21.15.155:0/3133694236 shutdown_connections 2024-02-10T09:04:46.899 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:46.897+0000 7f1ad4813700 1 --2- 172.21.15.155:0/3133694236 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ad00f6510 0x7f1ad00f6930 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:04:46.899 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:46.897+0000 7f1ad4813700 1 -- 172.21.15.155:0/3133694236 >> 172.21.15.155:0/3133694236 conn(0x7f1ad00f1c70 msgr2=0x7f1ad00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:04:46.899 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:46.897+0000 7f1ad4813700 1 -- 172.21.15.155:0/3133694236 shutdown_connections 2024-02-10T09:04:46.899 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:46.897+0000 7f1ad4813700 1 -- 172.21.15.155:0/3133694236 wait complete. 2024-02-10T09:04:46.899 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:46.898+0000 7f1ad4813700 1 Processor -- start 2024-02-10T09:04:46.899 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:46.898+0000 7f1ad4813700 1 -- start start 2024-02-10T09:04:46.899 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:46.898+0000 7f1ad4813700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ad00f6510 0x7f1ad0106810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:04:46.899 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:46.898+0000 7f1ad4813700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad0106d00 con 0x7f1ad00f6510 2024-02-10T09:04:46.899 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:46.899+0000 7f1ace59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ad00f6510 0x7f1ad0106810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:04:46.899 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:46.899+0000 7f1ace59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ad00f6510 0x7f1ad0106810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:50582/0 (socket says 172.21.15.155:50582) 2024-02-10T09:04:46.900 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:46.899+0000 7f1ace59c700 1 -- 172.21.15.155:0/3359225432 learned_addr learned my addr 172.21.15.155:0/3359225432 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:04:46.901 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:46.899+0000 7f1ace59c700 1 -- 172.21.15.155:0/3359225432 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ac4005040 con 0x7f1ad00f6510 2024-02-10T09:04:46.901 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:46.899+0000 7f1ace59c700 1 --2- 172.21.15.155:0/3359225432 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ad00f6510 0x7f1ad0106810 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f1ac4005010 tx=0x7f1ac400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:04:46.901 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:46.899+0000 7f1abeffd700 1 -- 172.21.15.155:0/3359225432 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ac4009320 con 0x7f1ad00f6510 2024-02-10T09:04:46.901 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:46.899+0000 7f1ad4813700 1 -- 172.21.15.155:0/3359225432 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ad0106f00 con 0x7f1ad00f6510 2024-02-10T09:04:46.902 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:46.899+0000 7f1abeffd700 1 -- 172.21.15.155:0/3359225432 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1ac4013380 con 0x7f1ad00f6510 2024-02-10T09:04:46.902 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:46.899+0000 7f1abeffd700 1 -- 172.21.15.155:0/3359225432 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ac4014e50 con 0x7f1ad00f6510 2024-02-10T09:04:46.902 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:46.899+0000 7f1ad4813700 1 -- 172.21.15.155:0/3359225432 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ad0107390 con 0x7f1ad00f6510 2024-02-10T09:04:46.902 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:46.900+0000 7f1abeffd700 1 -- 172.21.15.155:0/3359225432 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1ac40095f0 con 0x7f1ad00f6510 2024-02-10T09:04:46.902 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:46.900+0000 7f1ad4813700 1 -- 172.21.15.155:0/3359225432 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ad0045ad0 con 0x7f1ad00f6510 2024-02-10T09:04:46.902 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:46.900+0000 7f1abeffd700 1 --2- 172.21.15.155:0/3359225432 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f1ab802ff10 0x7f1ab80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:04:46.902 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:46.900+0000 7f1abeffd700 1 -- 172.21.15.155:0/3359225432 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1ac4040fa0 con 0x7f1ad00f6510 2024-02-10T09:04:46.902 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:46.900+0000 7f1acdd9b700 1 --2- 172.21.15.155:0/3359225432 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f1ab802ff10 0x7f1ab80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:04:46.902 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:46.901+0000 7f1acdd9b700 1 --2- 172.21.15.155:0/3359225432 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f1ab802ff10 0x7f1ab80323d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f1ac00096a0 tx=0x7f1ac0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:04:46.904 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:46.905+0000 7f1abeffd700 1 -- 172.21.15.155:0/3359225432 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1ac4014420 con 0x7f1ad00f6510 2024-02-10T09:04:47.110 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:47.110+0000 7f1ad4813700 1 -- 172.21.15.155:0/3359225432 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1ad0108240 con 0x7f1ad00f6510 2024-02-10T09:04:47.111 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:47.111+0000 7f1abeffd700 1 -- 172.21.15.155:0/3359225432 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1ac4010230 con 0x7f1ad00f6510 2024-02-10T09:04:47.111 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:04:47.112 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:04:47.113 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:47.113+0000 7f1ad4813700 1 -- 172.21.15.155:0/3359225432 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f1ab802ff10 msgr2=0x7f1ab80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:04:47.113 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:47.113+0000 7f1ad4813700 1 --2- 172.21.15.155:0/3359225432 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f1ab802ff10 0x7f1ab80323d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f1ac00096a0 tx=0x7f1ac0006b40).stop 2024-02-10T09:04:47.113 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:47.113+0000 7f1ad4813700 1 -- 172.21.15.155:0/3359225432 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ad00f6510 msgr2=0x7f1ad0106810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:04:47.113 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:47.113+0000 7f1ad4813700 1 --2- 172.21.15.155:0/3359225432 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ad00f6510 0x7f1ad0106810 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f1ac4005010 tx=0x7f1ac400b040).stop 2024-02-10T09:04:47.113 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:47.113+0000 7f1ad4813700 1 -- 172.21.15.155:0/3359225432 shutdown_connections 2024-02-10T09:04:47.113 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:47.113+0000 7f1ad4813700 1 --2- 172.21.15.155:0/3359225432 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f1ab802ff10 0x7f1ab80323d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:04:47.113 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:47.113+0000 7f1ad4813700 1 --2- 172.21.15.155:0/3359225432 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ad00f6510 0x7f1ad0106810 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:04:47.113 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:47.113+0000 7f1ad4813700 1 -- 172.21.15.155:0/3359225432 >> 172.21.15.155:0/3359225432 conn(0x7f1ad00f1c70 msgr2=0x7f1ad00f26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:04:47.113 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:47.113+0000 7f1ad4813700 1 -- 172.21.15.155:0/3359225432 shutdown_connections 2024-02-10T09:04:47.114 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:47.113+0000 7f1ad4813700 1 -- 172.21.15.155:0/3359225432 wait complete. 2024-02-10T09:04:47.114 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:04:47.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:04:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:04:47.011014+0000 mgr.smithi077.dxntvw (mgr.14162) 186 : cluster [DBG] pgmap v174: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:04:48.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:04:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:04:47.093487+0000 mon.smithi077 (mon.0) 217 : audit [INF] from='mgr.14162 172.21.15.77:0/2029106299' entity='mgr.smithi077.dxntvw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi077.dxntvw/mirror_snapshot_schedule"}]: dispatch 2024-02-10T09:04:48.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:04:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:04:47.100399+0000 mon.smithi077 (mon.0) 218 : audit [INF] from='mgr.14162 172.21.15.77:0/2029106299' entity='mgr.smithi077.dxntvw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi077.dxntvw/trash_purge_schedule"}]: dispatch 2024-02-10T09:04:48.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:04:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:04:47.111580+0000 mon.smithi077 (mon.0) 219 : audit [DBG] from='client.? 172.21.15.155:0/3359225432' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:04:48.823 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:04:48.823 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:04:49.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:04:49 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:04:49.011258+0000 mgr.smithi077.dxntvw (mgr.14162) 187 : cluster [DBG] pgmap v175: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:04:50.808 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:50.807+0000 7fd004b53700 1 -- 172.21.15.155:0/167164838 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0000f5410 msgr2=0x7fd0000f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:04:50.809 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:50.807+0000 7fd004b53700 1 --2- 172.21.15.155:0/167164838 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0000f5410 0x7fd0000f5830 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7fcfe8009fb0 tx=0x7fcfe8009b30).stop 2024-02-10T09:04:50.809 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:50.807+0000 7fd004b53700 1 -- 172.21.15.155:0/167164838 shutdown_connections 2024-02-10T09:04:50.809 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:50.807+0000 7fd004b53700 1 --2- 172.21.15.155:0/167164838 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0000f5410 0x7fd0000f5830 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:04:50.809 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:50.807+0000 7fd004b53700 1 -- 172.21.15.155:0/167164838 >> 172.21.15.155:0/167164838 conn(0x7fd0000f0b70 msgr2=0x7fd0000f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:04:50.809 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:50.807+0000 7fd004b53700 1 -- 172.21.15.155:0/167164838 shutdown_connections 2024-02-10T09:04:50.809 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:50.808+0000 7fd004b53700 1 -- 172.21.15.155:0/167164838 wait complete. 2024-02-10T09:04:50.809 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:50.808+0000 7fd004b53700 1 Processor -- start 2024-02-10T09:04:50.809 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:50.808+0000 7fd004b53700 1 -- start start 2024-02-10T09:04:50.809 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:50.808+0000 7fd004b53700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0000f5410 0x7fd0001094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:04:50.809 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:50.808+0000 7fd004b53700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd000060240 con 0x7fd0000f5410 2024-02-10T09:04:50.809 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:50.808+0000 7fcffe59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0000f5410 0x7fd0001094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:04:50.809 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:50.808+0000 7fcffe59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0000f5410 0x7fd0001094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:50610/0 (socket says 172.21.15.155:50610) 2024-02-10T09:04:50.810 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:50.808+0000 7fcffe59c700 1 -- 172.21.15.155:0/3266462670 learned_addr learned my addr 172.21.15.155:0/3266462670 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:04:50.810 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:50.809+0000 7fcffe59c700 1 -- 172.21.15.155:0/3266462670 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfe8005040 con 0x7fd0000f5410 2024-02-10T09:04:50.810 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:50.809+0000 7fcffe59c700 1 --2- 172.21.15.155:0/3266462670 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0000f5410 0x7fd0001094e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7fcfe8005010 tx=0x7fcfe800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:04:50.811 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:50.809+0000 7fcff6ffd700 1 -- 172.21.15.155:0/3266462670 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcfe8009320 con 0x7fd0000f5410 2024-02-10T09:04:50.811 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:50.809+0000 7fd004b53700 1 -- 172.21.15.155:0/3266462670 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd000060440 con 0x7fd0000f5410 2024-02-10T09:04:50.811 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:50.809+0000 7fcff6ffd700 1 -- 172.21.15.155:0/3266462670 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe8013380 con 0x7fd0000f5410 2024-02-10T09:04:50.811 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:50.809+0000 7fcff6ffd700 1 -- 172.21.15.155:0/3266462670 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcfe8014e50 con 0x7fd0000f5410 2024-02-10T09:04:50.811 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:50.809+0000 7fd004b53700 1 -- 172.21.15.155:0/3266462670 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd000109b90 con 0x7fd0000f5410 2024-02-10T09:04:50.811 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:50.810+0000 7fcff6ffd700 1 -- 172.21.15.155:0/3266462670 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcfe80095f0 con 0x7fd0000f5410 2024-02-10T09:04:50.811 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:50.810+0000 7fd004b53700 1 -- 172.21.15.155:0/3266462670 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd000045ad0 con 0x7fd0000f5410 2024-02-10T09:04:50.811 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:50.810+0000 7fcff6ffd700 1 --2- 172.21.15.155:0/3266462670 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fcfec02ff10 0x7fcfec0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:04:50.812 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:50.810+0000 7fcff6ffd700 1 -- 172.21.15.155:0/3266462670 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcfe8040e60 con 0x7fd0000f5410 2024-02-10T09:04:50.812 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:50.810+0000 7fcffdd9b700 1 --2- 172.21.15.155:0/3266462670 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fcfec02ff10 0x7fcfec0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:04:50.812 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:50.811+0000 7fcffdd9b700 1 --2- 172.21.15.155:0/3266462670 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fcfec02ff10 0x7fcfec0323d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fcff00096a0 tx=0x7fcff0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:04:50.815 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:50.815+0000 7fcff6ffd700 1 -- 172.21.15.155:0/3266462670 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcfe8014420 con 0x7fd0000f5410 2024-02-10T09:04:51.019 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:51.019+0000 7fd004b53700 1 -- 172.21.15.155:0/3266462670 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd00010a3b0 con 0x7fd0000f5410 2024-02-10T09:04:51.020 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:51.020+0000 7fcff6ffd700 1 -- 172.21.15.155:0/3266462670 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcfe801d980 con 0x7fd0000f5410 2024-02-10T09:04:51.020 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:04:51.020 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:04:51.021 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:51.021+0000 7fd004b53700 1 -- 172.21.15.155:0/3266462670 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fcfec02ff10 msgr2=0x7fcfec0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:04:51.022 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:51.021+0000 7fd004b53700 1 --2- 172.21.15.155:0/3266462670 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fcfec02ff10 0x7fcfec0323d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fcff00096a0 tx=0x7fcff0006b40).stop 2024-02-10T09:04:51.022 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:51.021+0000 7fd004b53700 1 -- 172.21.15.155:0/3266462670 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0000f5410 msgr2=0x7fd0001094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:04:51.022 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:51.021+0000 7fd004b53700 1 --2- 172.21.15.155:0/3266462670 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0000f5410 0x7fd0001094e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7fcfe8005010 tx=0x7fcfe800b040).stop 2024-02-10T09:04:51.022 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:51.021+0000 7fd004b53700 1 -- 172.21.15.155:0/3266462670 shutdown_connections 2024-02-10T09:04:51.022 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:51.021+0000 7fd004b53700 1 --2- 172.21.15.155:0/3266462670 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fcfec02ff10 0x7fcfec0323d0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:04:51.022 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:51.021+0000 7fd004b53700 1 --2- 172.21.15.155:0/3266462670 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd0000f5410 0x7fd0001094e0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:04:51.022 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:51.021+0000 7fd004b53700 1 -- 172.21.15.155:0/3266462670 >> 172.21.15.155:0/3266462670 conn(0x7fd0000f0b70 msgr2=0x7fd0000f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:04:51.022 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:51.021+0000 7fd004b53700 1 -- 172.21.15.155:0/3266462670 shutdown_connections 2024-02-10T09:04:51.022 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:51.021+0000 7fd004b53700 1 -- 172.21.15.155:0/3266462670 wait complete. 2024-02-10T09:04:51.022 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:04:51.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:04:51 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:04:51.011575+0000 mgr.smithi077.dxntvw (mgr.14162) 188 : cluster [DBG] pgmap v176: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:04:51.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:04:51 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:04:51.020303+0000 mon.smithi077 (mon.0) 220 : audit [DBG] from='client.? 172.21.15.155:0/3266462670' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:04:52.844 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:04:52.844 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:04:54.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:04:53 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:04:53.011936+0000 mgr.smithi077.dxntvw (mgr.14162) 189 : cluster [DBG] pgmap v177: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:04:54.830 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:54.811+0000 7fdcfc2d7700 1 -- 172.21.15.155:0/1563772067 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdcf40f53f0 msgr2=0x7fdcf40f5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:04:54.830 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:54.811+0000 7fdcfc2d7700 1 --2- 172.21.15.155:0/1563772067 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdcf40f53f0 0x7fdcf40f5810 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7fdce4009fb0 tx=0x7fdce4009b30).stop 2024-02-10T09:04:54.830 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:54.811+0000 7fdcfc2d7700 1 -- 172.21.15.155:0/1563772067 shutdown_connections 2024-02-10T09:04:54.830 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:54.811+0000 7fdcfc2d7700 1 --2- 172.21.15.155:0/1563772067 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdcf40f53f0 0x7fdcf40f5810 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:04:54.831 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:54.811+0000 7fdcfc2d7700 1 -- 172.21.15.155:0/1563772067 >> 172.21.15.155:0/1563772067 conn(0x7fdcf40f0b70 msgr2=0x7fdcf40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:04:54.831 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:54.811+0000 7fdcfc2d7700 1 -- 172.21.15.155:0/1563772067 shutdown_connections 2024-02-10T09:04:54.831 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:54.811+0000 7fdcfc2d7700 1 -- 172.21.15.155:0/1563772067 wait complete. 2024-02-10T09:04:54.831 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:54.811+0000 7fdcfc2d7700 1 Processor -- start 2024-02-10T09:04:54.831 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:54.811+0000 7fdcfc2d7700 1 -- start start 2024-02-10T09:04:54.831 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:54.812+0000 7fdcfc2d7700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdcf40f53f0 0x7fdcf41094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:04:54.831 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:54.812+0000 7fdcfc2d7700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcf4060220 con 0x7fdcf40f53f0 2024-02-10T09:04:54.831 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:54.812+0000 7fdcfa073700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdcf40f53f0 0x7fdcf41094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:04:54.831 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:54.812+0000 7fdcfa073700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdcf40f53f0 0x7fdcf41094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:42214/0 (socket says 172.21.15.155:42214) 2024-02-10T09:04:54.831 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:54.812+0000 7fdcfa073700 1 -- 172.21.15.155:0/3751871864 learned_addr learned my addr 172.21.15.155:0/3751871864 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:04:54.831 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:54.812+0000 7fdcfa073700 1 -- 172.21.15.155:0/3751871864 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdce4005040 con 0x7fdcf40f53f0 2024-02-10T09:04:54.831 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:54.812+0000 7fdcfa073700 1 --2- 172.21.15.155:0/3751871864 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdcf40f53f0 0x7fdcf41094e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7fdce4005010 tx=0x7fdce400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:04:54.831 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:54.812+0000 7fdceaffd700 1 -- 172.21.15.155:0/3751871864 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdce4009320 con 0x7fdcf40f53f0 2024-02-10T09:04:54.831 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:54.812+0000 7fdcfc2d7700 1 -- 172.21.15.155:0/3751871864 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcf4060420 con 0x7fdcf40f53f0 2024-02-10T09:04:54.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:54.813+0000 7fdceaffd700 1 -- 172.21.15.155:0/3751871864 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdce4013380 con 0x7fdcf40f53f0 2024-02-10T09:04:54.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:54.813+0000 7fdcfc2d7700 1 -- 172.21.15.155:0/3751871864 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcf4109b90 con 0x7fdcf40f53f0 2024-02-10T09:04:54.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:54.813+0000 7fdceaffd700 1 -- 172.21.15.155:0/3751871864 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdce4014e50 con 0x7fdcf40f53f0 2024-02-10T09:04:54.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:54.813+0000 7fdcfc2d7700 1 -- 172.21.15.155:0/3751871864 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcf4059100 con 0x7fdcf40f53f0 2024-02-10T09:04:54.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:54.813+0000 7fdceaffd700 1 -- 172.21.15.155:0/3751871864 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdce40095f0 con 0x7fdcf40f53f0 2024-02-10T09:04:54.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:54.814+0000 7fdceaffd700 1 --2- 172.21.15.155:0/3751871864 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fdce002ff10 0x7fdce00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:04:54.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:54.814+0000 7fdceaffd700 1 -- 172.21.15.155:0/3751871864 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdce4040ed0 con 0x7fdcf40f53f0 2024-02-10T09:04:54.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:54.814+0000 7fdcf9872700 1 --2- 172.21.15.155:0/3751871864 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fdce002ff10 0x7fdce00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:04:54.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:54.814+0000 7fdcf9872700 1 --2- 172.21.15.155:0/3751871864 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fdce002ff10 0x7fdce00323d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fdcf00096a0 tx=0x7fdcf0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:04:54.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:54.818+0000 7fdceaffd700 1 -- 172.21.15.155:0/3751871864 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdce4014420 con 0x7fdcf40f53f0 2024-02-10T09:04:55.025 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:55.025+0000 7fdcfc2d7700 1 -- 172.21.15.155:0/3751871864 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdcf40faf80 con 0x7fdcf40f53f0 2024-02-10T09:04:55.025 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:55.025+0000 7fdceaffd700 1 -- 172.21.15.155:0/3751871864 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdce4010230 con 0x7fdcf40f53f0 2024-02-10T09:04:55.026 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:04:55.026 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:04:55.027 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:55.027+0000 7fdcfc2d7700 1 -- 172.21.15.155:0/3751871864 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fdce002ff10 msgr2=0x7fdce00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:04:55.027 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:55.027+0000 7fdcfc2d7700 1 --2- 172.21.15.155:0/3751871864 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fdce002ff10 0x7fdce00323d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fdcf00096a0 tx=0x7fdcf0006b40).stop 2024-02-10T09:04:55.028 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:55.027+0000 7fdcfc2d7700 1 -- 172.21.15.155:0/3751871864 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdcf40f53f0 msgr2=0x7fdcf41094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:04:55.028 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:55.027+0000 7fdcfc2d7700 1 --2- 172.21.15.155:0/3751871864 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdcf40f53f0 0x7fdcf41094e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7fdce4005010 tx=0x7fdce400b040).stop 2024-02-10T09:04:55.028 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:55.027+0000 7fdcfc2d7700 1 -- 172.21.15.155:0/3751871864 shutdown_connections 2024-02-10T09:04:55.028 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:55.027+0000 7fdcfc2d7700 1 --2- 172.21.15.155:0/3751871864 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fdce002ff10 0x7fdce00323d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:04:55.028 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:55.027+0000 7fdcfc2d7700 1 --2- 172.21.15.155:0/3751871864 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdcf40f53f0 0x7fdcf41094e0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:04:55.028 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:55.027+0000 7fdcfc2d7700 1 -- 172.21.15.155:0/3751871864 >> 172.21.15.155:0/3751871864 conn(0x7fdcf40f0b70 msgr2=0x7fdcf40f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:04:55.028 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:55.027+0000 7fdcfc2d7700 1 -- 172.21.15.155:0/3751871864 shutdown_connections 2024-02-10T09:04:55.028 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:55.027+0000 7fdcfc2d7700 1 -- 172.21.15.155:0/3751871864 wait complete. 2024-02-10T09:04:55.029 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:04:55.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:04:55 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:04:55.012244+0000 mgr.smithi077.dxntvw (mgr.14162) 190 : cluster [DBG] pgmap v178: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:04:55.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:04:55 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:04:55.025895+0000 mon.smithi077 (mon.0) 221 : audit [DBG] from='client.? 172.21.15.155:0/3751871864' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:04:56.820 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:04:56.820 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:04:57.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:04:57 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:04:57.012624+0000 mgr.smithi077.dxntvw (mgr.14162) 191 : cluster [DBG] pgmap v179: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:04:59.004 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:59.003+0000 7f6b03fff700 1 -- 172.21.15.155:0/771798056 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b040f62a0 msgr2=0x7f6b040f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:04:59.005 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:59.003+0000 7f6b03fff700 1 --2- 172.21.15.155:0/771798056 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b040f62a0 0x7f6b040f66c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f6af8009fb0 tx=0x7f6af8009b30).stop 2024-02-10T09:04:59.005 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:59.003+0000 7f6b03fff700 1 -- 172.21.15.155:0/771798056 shutdown_connections 2024-02-10T09:04:59.005 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:59.003+0000 7f6b03fff700 1 --2- 172.21.15.155:0/771798056 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b040f62a0 0x7f6b040f66c0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:04:59.005 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:59.003+0000 7f6b03fff700 1 -- 172.21.15.155:0/771798056 >> 172.21.15.155:0/771798056 conn(0x7f6b040f1aa0 msgr2=0x7f6b040f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:04:59.005 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:59.003+0000 7f6b03fff700 1 -- 172.21.15.155:0/771798056 shutdown_connections 2024-02-10T09:04:59.005 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:59.003+0000 7f6b03fff700 1 -- 172.21.15.155:0/771798056 wait complete. 2024-02-10T09:04:59.005 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:59.004+0000 7f6b03fff700 1 Processor -- start 2024-02-10T09:04:59.005 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:59.004+0000 7f6b03fff700 1 -- start start 2024-02-10T09:04:59.005 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:59.004+0000 7f6b03fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b040f62a0 0x7f6b041065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:04:59.005 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:59.004+0000 7f6b03fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b04106ac0 con 0x7f6b040f62a0 2024-02-10T09:04:59.005 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:59.004+0000 7f6b02ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b040f62a0 0x7f6b041065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:04:59.005 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:59.004+0000 7f6b02ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b040f62a0 0x7f6b041065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:42230/0 (socket says 172.21.15.155:42230) 2024-02-10T09:04:59.005 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:59.004+0000 7f6b02ffd700 1 -- 172.21.15.155:0/3441203913 learned_addr learned my addr 172.21.15.155:0/3441203913 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:04:59.005 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:59.005+0000 7f6b02ffd700 1 -- 172.21.15.155:0/3441203913 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6af8005040 con 0x7f6b040f62a0 2024-02-10T09:04:59.007 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:59.005+0000 7f6b02ffd700 1 --2- 172.21.15.155:0/3441203913 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b040f62a0 0x7f6b041065d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f6af80089f0 tx=0x7f6af800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:04:59.007 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:59.005+0000 7f6aeb7fe700 1 -- 172.21.15.155:0/3441203913 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6af8009320 con 0x7f6b040f62a0 2024-02-10T09:04:59.007 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:59.005+0000 7f6b03fff700 1 -- 172.21.15.155:0/3441203913 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b04106cc0 con 0x7f6b040f62a0 2024-02-10T09:04:59.007 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:59.005+0000 7f6aeb7fe700 1 -- 172.21.15.155:0/3441203913 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6af8013380 con 0x7f6b040f62a0 2024-02-10T09:04:59.007 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:59.005+0000 7f6aeb7fe700 1 -- 172.21.15.155:0/3441203913 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6af8014e20 con 0x7f6b040f62a0 2024-02-10T09:04:59.007 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:59.005+0000 7f6b03fff700 1 -- 172.21.15.155:0/3441203913 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b04107150 con 0x7f6b040f62a0 2024-02-10T09:04:59.007 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:59.006+0000 7f6aeb7fe700 1 -- 172.21.15.155:0/3441203913 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6af8014420 con 0x7f6b040f62a0 2024-02-10T09:04:59.007 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:59.006+0000 7f6b03fff700 1 -- 172.21.15.155:0/3441203913 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b04045ad0 con 0x7f6b040f62a0 2024-02-10T09:04:59.008 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:59.006+0000 7f6aeb7fe700 1 --2- 172.21.15.155:0/3441203913 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f6aec02ff60 0x7f6aec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:04:59.008 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:59.006+0000 7f6aeb7fe700 1 -- 172.21.15.155:0/3441203913 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6af8040ea0 con 0x7f6b040f62a0 2024-02-10T09:04:59.008 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:59.006+0000 7f6b027fc700 1 --2- 172.21.15.155:0/3441203913 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f6aec02ff60 0x7f6aec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:04:59.008 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:59.007+0000 7f6b027fc700 1 --2- 172.21.15.155:0/3441203913 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f6aec02ff60 0x7f6aec032420 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f6af40096a0 tx=0x7f6af4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:04:59.011 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:59.011+0000 7f6aeb7fe700 1 -- 172.21.15.155:0/3441203913 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6af8014790 con 0x7f6b040f62a0 2024-02-10T09:04:59.219 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:59.219+0000 7f6b03fff700 1 -- 172.21.15.155:0/3441203913 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6b04108000 con 0x7f6b040f62a0 2024-02-10T09:04:59.220 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:59.220+0000 7f6aeb7fe700 1 -- 172.21.15.155:0/3441203913 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6af8010230 con 0x7f6b040f62a0 2024-02-10T09:04:59.220 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:04:59.220 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:04:59.222 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:59.221+0000 7f6b03fff700 1 -- 172.21.15.155:0/3441203913 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f6aec02ff60 msgr2=0x7f6aec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:04:59.222 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:59.221+0000 7f6b03fff700 1 --2- 172.21.15.155:0/3441203913 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f6aec02ff60 0x7f6aec032420 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f6af40096a0 tx=0x7f6af4006b40).stop 2024-02-10T09:04:59.222 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:59.221+0000 7f6b03fff700 1 -- 172.21.15.155:0/3441203913 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b040f62a0 msgr2=0x7f6b041065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:04:59.222 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:59.221+0000 7f6b03fff700 1 --2- 172.21.15.155:0/3441203913 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b040f62a0 0x7f6b041065d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f6af80089f0 tx=0x7f6af800b040).stop 2024-02-10T09:04:59.222 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:59.222+0000 7f6b03fff700 1 -- 172.21.15.155:0/3441203913 shutdown_connections 2024-02-10T09:04:59.222 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:59.222+0000 7f6b03fff700 1 --2- 172.21.15.155:0/3441203913 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f6aec02ff60 0x7f6aec032420 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:04:59.222 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:59.222+0000 7f6b03fff700 1 --2- 172.21.15.155:0/3441203913 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b040f62a0 0x7f6b041065d0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:04:59.222 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:59.222+0000 7f6b03fff700 1 -- 172.21.15.155:0/3441203913 >> 172.21.15.155:0/3441203913 conn(0x7f6b040f1aa0 msgr2=0x7f6b040f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:04:59.222 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:59.222+0000 7f6b03fff700 1 -- 172.21.15.155:0/3441203913 shutdown_connections 2024-02-10T09:04:59.222 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:04:59.222+0000 7f6b03fff700 1 -- 172.21.15.155:0/3441203913 wait complete. 2024-02-10T09:04:59.223 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:04:59.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:04:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:04:59.013026+0000 mgr.smithi077.dxntvw (mgr.14162) 192 : cluster [DBG] pgmap v180: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:05:00.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:05:00 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:04:59.220511+0000 mon.smithi077 (mon.0) 222 : audit [DBG] from='client.? 172.21.15.155:0/3441203913' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:05:01.068 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:05:01.068 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:05:01.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:05:01 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:05:01.013397+0000 mgr.smithi077.dxntvw (mgr.14162) 193 : cluster [DBG] pgmap v181: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:05:03.238 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:03.235+0000 7fdd6e1ea700 1 -- 172.21.15.155:0/2511331733 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd680f51a0 msgr2=0x7fdd680f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:05:03.238 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:03.235+0000 7fdd6e1ea700 1 --2- 172.21.15.155:0/2511331733 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd680f51a0 0x7fdd680f55c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7fdd64009fb0 tx=0x7fdd64009b30).stop 2024-02-10T09:05:03.238 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:03.235+0000 7fdd6e1ea700 1 -- 172.21.15.155:0/2511331733 shutdown_connections 2024-02-10T09:05:03.238 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:03.235+0000 7fdd6e1ea700 1 --2- 172.21.15.155:0/2511331733 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd680f51a0 0x7fdd680f55c0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:05:03.239 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:03.235+0000 7fdd6e1ea700 1 -- 172.21.15.155:0/2511331733 >> 172.21.15.155:0/2511331733 conn(0x7fdd680f09a0 msgr2=0x7fdd680f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:05:03.239 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:03.235+0000 7fdd6e1ea700 1 -- 172.21.15.155:0/2511331733 shutdown_connections 2024-02-10T09:05:03.239 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:03.235+0000 7fdd6e1ea700 1 -- 172.21.15.155:0/2511331733 wait complete. 2024-02-10T09:05:03.239 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:03.235+0000 7fdd6e1ea700 1 Processor -- start 2024-02-10T09:05:03.239 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:03.235+0000 7fdd6e1ea700 1 -- start start 2024-02-10T09:05:03.239 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:03.236+0000 7fdd6e1ea700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd680f51a0 0x7fdd680f2f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:05:03.239 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:03.236+0000 7fdd6e1ea700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd680f13a0 con 0x7fdd680f51a0 2024-02-10T09:05:03.239 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:03.236+0000 7fdd6d1e8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd680f51a0 0x7fdd680f2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:05:03.239 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:03.236+0000 7fdd6d1e8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd680f51a0 0x7fdd680f2f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:51646/0 (socket says 172.21.15.155:51646) 2024-02-10T09:05:03.239 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:03.236+0000 7fdd6d1e8700 1 -- 172.21.15.155:0/2115830157 learned_addr learned my addr 172.21.15.155:0/2115830157 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:05:03.239 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:03.236+0000 7fdd6d1e8700 1 -- 172.21.15.155:0/2115830157 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd64005040 con 0x7fdd680f51a0 2024-02-10T09:05:03.239 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:03.236+0000 7fdd6d1e8700 1 --2- 172.21.15.155:0/2115830157 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd680f51a0 0x7fdd680f2f20 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7fdd640089f0 tx=0x7fdd6400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:05:03.240 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:03.237+0000 7fdd5dffb700 1 -- 172.21.15.155:0/2115830157 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd64009320 con 0x7fdd680f51a0 2024-02-10T09:05:03.240 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:03.237+0000 7fdd5dffb700 1 -- 172.21.15.155:0/2115830157 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdd64013380 con 0x7fdd680f51a0 2024-02-10T09:05:03.240 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:03.237+0000 7fdd6e1ea700 1 -- 172.21.15.155:0/2115830157 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd680f1540 con 0x7fdd680f51a0 2024-02-10T09:05:03.240 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:03.237+0000 7fdd5dffb700 1 -- 172.21.15.155:0/2115830157 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd64014e20 con 0x7fdd680f51a0 2024-02-10T09:05:03.240 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:03.237+0000 7fdd6e1ea700 1 -- 172.21.15.155:0/2115830157 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd680f19d0 con 0x7fdd680f51a0 2024-02-10T09:05:03.241 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:03.238+0000 7fdd5dffb700 1 -- 172.21.15.155:0/2115830157 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdd64014420 con 0x7fdd680f51a0 2024-02-10T09:05:03.241 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:03.238+0000 7fdd6e1ea700 1 -- 172.21.15.155:0/2115830157 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd68059100 con 0x7fdd680f51a0 2024-02-10T09:05:03.241 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:03.238+0000 7fdd5dffb700 1 --2- 172.21.15.155:0/2115830157 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fdd5402ff10 0x7fdd540323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:05:03.241 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:03.238+0000 7fdd5dffb700 1 -- 172.21.15.155:0/2115830157 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdd64040f50 con 0x7fdd680f51a0 2024-02-10T09:05:03.241 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:03.238+0000 7fdd6c9e7700 1 --2- 172.21.15.155:0/2115830157 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fdd5402ff10 0x7fdd540323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:05:03.241 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:03.239+0000 7fdd6c9e7700 1 --2- 172.21.15.155:0/2115830157 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fdd5402ff10 0x7fdd540323d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fdd580096a0 tx=0x7fdd58006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:05:03.244 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:03.243+0000 7fdd5dffb700 1 -- 172.21.15.155:0/2115830157 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdd64014790 con 0x7fdd680f51a0 2024-02-10T09:05:03.451 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:03.450+0000 7fdd6e1ea700 1 -- 172.21.15.155:0/2115830157 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdd68045ad0 con 0x7fdd680f51a0 2024-02-10T09:05:03.452 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:03.451+0000 7fdd5dffb700 1 -- 172.21.15.155:0/2115830157 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdd6401d7f0 con 0x7fdd680f51a0 2024-02-10T09:05:03.452 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:05:03.452 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:05:03.454 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:03.453+0000 7fdd6e1ea700 1 -- 172.21.15.155:0/2115830157 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fdd5402ff10 msgr2=0x7fdd540323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:05:03.454 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:03.453+0000 7fdd6e1ea700 1 --2- 172.21.15.155:0/2115830157 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fdd5402ff10 0x7fdd540323d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fdd580096a0 tx=0x7fdd58006b40).stop 2024-02-10T09:05:03.454 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:03.453+0000 7fdd6e1ea700 1 -- 172.21.15.155:0/2115830157 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd680f51a0 msgr2=0x7fdd680f2f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:05:03.454 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:03.453+0000 7fdd6e1ea700 1 --2- 172.21.15.155:0/2115830157 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd680f51a0 0x7fdd680f2f20 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7fdd640089f0 tx=0x7fdd6400b040).stop 2024-02-10T09:05:03.455 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:03.453+0000 7fdd6e1ea700 1 -- 172.21.15.155:0/2115830157 shutdown_connections 2024-02-10T09:05:03.455 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:03.453+0000 7fdd6e1ea700 1 --2- 172.21.15.155:0/2115830157 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fdd5402ff10 0x7fdd540323d0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:05:03.455 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:03.453+0000 7fdd6e1ea700 1 --2- 172.21.15.155:0/2115830157 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd680f51a0 0x7fdd680f2f20 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:05:03.455 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:03.453+0000 7fdd6e1ea700 1 -- 172.21.15.155:0/2115830157 >> 172.21.15.155:0/2115830157 conn(0x7fdd680f09a0 msgr2=0x7fdd6805fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:05:03.455 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:03.453+0000 7fdd6e1ea700 1 -- 172.21.15.155:0/2115830157 shutdown_connections 2024-02-10T09:05:03.455 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:03.453+0000 7fdd6e1ea700 1 -- 172.21.15.155:0/2115830157 wait complete. 2024-02-10T09:05:03.455 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:05:04.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:05:03 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:05:03.013716+0000 mgr.smithi077.dxntvw (mgr.14162) 194 : cluster [DBG] pgmap v182: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:05:04.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:05:03 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:05:03.452613+0000 mon.smithi077 (mon.0) 223 : audit [DBG] from='client.? 172.21.15.155:0/2115830157' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:05:05.342 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:05:05.343 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:05:05.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:05:05 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:05:05.014087+0000 mgr.smithi077.dxntvw (mgr.14162) 195 : cluster [DBG] pgmap v183: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:05:07.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:05:07 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:05:07.014438+0000 mgr.smithi077.dxntvw (mgr.14162) 196 : cluster [DBG] pgmap v184: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:05:07.662 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:07.659+0000 7f708ce42700 1 -- 172.21.15.155:0/2613687280 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70880f5410 msgr2=0x7f70880f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:05:07.662 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:07.659+0000 7f708ce42700 1 --2- 172.21.15.155:0/2613687280 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70880f5410 0x7f70880f5830 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f707c009fb0 tx=0x7f707c009b30).stop 2024-02-10T09:05:07.662 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:07.659+0000 7f708ce42700 1 -- 172.21.15.155:0/2613687280 shutdown_connections 2024-02-10T09:05:07.662 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:07.659+0000 7f708ce42700 1 --2- 172.21.15.155:0/2613687280 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70880f5410 0x7f70880f5830 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:05:07.663 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:07.659+0000 7f708ce42700 1 -- 172.21.15.155:0/2613687280 >> 172.21.15.155:0/2613687280 conn(0x7f70880f0b70 msgr2=0x7f70880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:05:07.663 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:07.659+0000 7f708ce42700 1 -- 172.21.15.155:0/2613687280 shutdown_connections 2024-02-10T09:05:07.663 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:07.659+0000 7f708ce42700 1 -- 172.21.15.155:0/2613687280 wait complete. 2024-02-10T09:05:07.663 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:07.660+0000 7f708ce42700 1 Processor -- start 2024-02-10T09:05:07.663 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:07.660+0000 7f708ce42700 1 -- start start 2024-02-10T09:05:07.663 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:07.660+0000 7f708ce42700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70880f5410 0x7f70881094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:05:07.663 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:07.660+0000 7f708ce42700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7088060240 con 0x7f70880f5410 2024-02-10T09:05:07.663 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:07.660+0000 7f708659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70880f5410 0x7f70881094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:05:07.663 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:07.660+0000 7f708659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70880f5410 0x7f70881094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:51658/0 (socket says 172.21.15.155:51658) 2024-02-10T09:05:07.663 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:07.660+0000 7f708659c700 1 -- 172.21.15.155:0/4267866470 learned_addr learned my addr 172.21.15.155:0/4267866470 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:05:07.663 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:07.661+0000 7f708659c700 1 -- 172.21.15.155:0/4267866470 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f707c005040 con 0x7f70880f5410 2024-02-10T09:05:07.663 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:07.661+0000 7f708659c700 1 --2- 172.21.15.155:0/4267866470 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70880f5410 0x7f70881094e0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f707c0089f0 tx=0x7f707c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:05:07.663 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:07.661+0000 7f7076ffd700 1 -- 172.21.15.155:0/4267866470 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f707c009320 con 0x7f70880f5410 2024-02-10T09:05:07.663 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:07.661+0000 7f708ce42700 1 -- 172.21.15.155:0/4267866470 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7088060440 con 0x7f70880f5410 2024-02-10T09:05:07.664 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:07.661+0000 7f708ce42700 1 -- 172.21.15.155:0/4267866470 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7088109b90 con 0x7f70880f5410 2024-02-10T09:05:07.664 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:07.661+0000 7f7076ffd700 1 -- 172.21.15.155:0/4267866470 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f707c013380 con 0x7f70880f5410 2024-02-10T09:05:07.664 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:07.661+0000 7f7076ffd700 1 -- 172.21.15.155:0/4267866470 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f707c01ed80 con 0x7f70880f5410 2024-02-10T09:05:07.665 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:07.662+0000 7f7076ffd700 1 -- 172.21.15.155:0/4267866470 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f707c014420 con 0x7f70880f5410 2024-02-10T09:05:07.665 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:07.662+0000 7f708ce42700 1 -- 172.21.15.155:0/4267866470 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f706c004fa0 con 0x7f70880f5410 2024-02-10T09:05:07.665 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:07.662+0000 7f7076ffd700 1 --2- 172.21.15.155:0/4267866470 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f707002ff60 0x7f7070032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:05:07.665 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:07.662+0000 7f7076ffd700 1 -- 172.21.15.155:0/4267866470 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f707c041040 con 0x7f70880f5410 2024-02-10T09:05:07.665 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:07.662+0000 7f7085d9b700 1 --2- 172.21.15.155:0/4267866470 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f707002ff60 0x7f7070032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:05:07.665 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:07.663+0000 7f7085d9b700 1 --2- 172.21.15.155:0/4267866470 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f707002ff60 0x7f7070032420 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f70780096a0 tx=0x7f7078006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:05:07.668 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:07.667+0000 7f7076ffd700 1 -- 172.21.15.155:0/4267866470 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f707c01c500 con 0x7f70880f5410 2024-02-10T09:05:07.874 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:07.873+0000 7f708ce42700 1 -- 172.21.15.155:0/4267866470 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f706c005670 con 0x7f70880f5410 2024-02-10T09:05:07.875 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:07.874+0000 7f7076ffd700 1 -- 172.21.15.155:0/4267866470 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f707c02c2d0 con 0x7f70880f5410 2024-02-10T09:05:07.875 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:05:07.875 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:05:07.877 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:07.875+0000 7f708ce42700 1 -- 172.21.15.155:0/4267866470 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f707002ff60 msgr2=0x7f7070032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:05:07.877 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:07.875+0000 7f708ce42700 1 --2- 172.21.15.155:0/4267866470 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f707002ff60 0x7f7070032420 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f70780096a0 tx=0x7f7078006b40).stop 2024-02-10T09:05:07.877 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:07.875+0000 7f708ce42700 1 -- 172.21.15.155:0/4267866470 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70880f5410 msgr2=0x7f70881094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:05:07.877 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:07.875+0000 7f708ce42700 1 --2- 172.21.15.155:0/4267866470 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70880f5410 0x7f70881094e0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f707c0089f0 tx=0x7f707c00b040).stop 2024-02-10T09:05:07.877 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:07.876+0000 7f708ce42700 1 -- 172.21.15.155:0/4267866470 shutdown_connections 2024-02-10T09:05:07.877 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:07.876+0000 7f708ce42700 1 --2- 172.21.15.155:0/4267866470 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f707002ff60 0x7f7070032420 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:05:07.877 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:07.876+0000 7f708ce42700 1 --2- 172.21.15.155:0/4267866470 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f70880f5410 0x7f70881094e0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:05:07.877 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:07.876+0000 7f708ce42700 1 -- 172.21.15.155:0/4267866470 >> 172.21.15.155:0/4267866470 conn(0x7f70880f0b70 msgr2=0x7f70880f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:05:07.877 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:07.876+0000 7f708ce42700 1 -- 172.21.15.155:0/4267866470 shutdown_connections 2024-02-10T09:05:07.877 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:07.876+0000 7f708ce42700 1 -- 172.21.15.155:0/4267866470 wait complete. 2024-02-10T09:05:07.878 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:05:09.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:05:08 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:05:07.875542+0000 mon.smithi077 (mon.0) 224 : audit [DBG] from='client.? 172.21.15.155:0/4267866470' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:05:09.733 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:05:09.733 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:05:10.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:05:09 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:05:09.014752+0000 mgr.smithi077.dxntvw (mgr.14162) 197 : cluster [DBG] pgmap v185: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:05:11.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:05:11 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:05:11.015004+0000 mgr.smithi077.dxntvw (mgr.14162) 198 : cluster [DBG] pgmap v186: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:05:11.804 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:11.802+0000 7fb8eefc5700 1 -- 172.21.15.155:0/3841173915 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8e80f51a0 msgr2=0x7fb8e80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:05:11.804 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:11.802+0000 7fb8eefc5700 1 --2- 172.21.15.155:0/3841173915 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8e80f51a0 0x7fb8e80f55c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7fb8e4009fb0 tx=0x7fb8e4009b30).stop 2024-02-10T09:05:11.804 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:11.802+0000 7fb8eefc5700 1 -- 172.21.15.155:0/3841173915 shutdown_connections 2024-02-10T09:05:11.804 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:11.802+0000 7fb8eefc5700 1 --2- 172.21.15.155:0/3841173915 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8e80f51a0 0x7fb8e80f55c0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:05:11.804 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:11.802+0000 7fb8eefc5700 1 -- 172.21.15.155:0/3841173915 >> 172.21.15.155:0/3841173915 conn(0x7fb8e80f09a0 msgr2=0x7fb8e80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:05:11.804 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:11.802+0000 7fb8eefc5700 1 -- 172.21.15.155:0/3841173915 shutdown_connections 2024-02-10T09:05:11.804 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:11.802+0000 7fb8eefc5700 1 -- 172.21.15.155:0/3841173915 wait complete. 2024-02-10T09:05:11.804 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:11.803+0000 7fb8eefc5700 1 Processor -- start 2024-02-10T09:05:11.805 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:11.803+0000 7fb8eefc5700 1 -- start start 2024-02-10T09:05:11.805 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:11.803+0000 7fb8eefc5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8e80f51a0 0x7fb8e81054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:05:11.805 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:11.803+0000 7fb8eefc5700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e81059c0 con 0x7fb8e80f51a0 2024-02-10T09:05:11.807 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:11.803+0000 7fb8edfc3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8e80f51a0 0x7fb8e81054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:05:11.807 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:11.803+0000 7fb8edfc3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8e80f51a0 0x7fb8e81054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:51680/0 (socket says 172.21.15.155:51680) 2024-02-10T09:05:11.807 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:11.803+0000 7fb8edfc3700 1 -- 172.21.15.155:0/1301138004 learned_addr learned my addr 172.21.15.155:0/1301138004 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:05:11.807 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:11.804+0000 7fb8edfc3700 1 -- 172.21.15.155:0/1301138004 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8e4005040 con 0x7fb8e80f51a0 2024-02-10T09:05:11.807 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:11.804+0000 7fb8edfc3700 1 --2- 172.21.15.155:0/1301138004 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8e80f51a0 0x7fb8e81054d0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7fb8e4009830 tx=0x7fb8e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:05:11.807 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:11.804+0000 7fb8de7fc700 1 -- 172.21.15.155:0/1301138004 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8e4009320 con 0x7fb8e80f51a0 2024-02-10T09:05:11.807 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:11.804+0000 7fb8eefc5700 1 -- 172.21.15.155:0/1301138004 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8e8105bc0 con 0x7fb8e80f51a0 2024-02-10T09:05:11.807 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:11.804+0000 7fb8de7fc700 1 -- 172.21.15.155:0/1301138004 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb8e4013380 con 0x7fb8e80f51a0 2024-02-10T09:05:11.807 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:11.804+0000 7fb8de7fc700 1 -- 172.21.15.155:0/1301138004 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8e4014420 con 0x7fb8e80f51a0 2024-02-10T09:05:11.807 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:11.804+0000 7fb8eefc5700 1 -- 172.21.15.155:0/1301138004 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8e8106050 con 0x7fb8e80f51a0 2024-02-10T09:05:11.807 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:11.805+0000 7fb8eefc5700 1 -- 172.21.15.155:0/1301138004 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8e8045ad0 con 0x7fb8e80f51a0 2024-02-10T09:05:11.807 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:11.805+0000 7fb8de7fc700 1 -- 172.21.15.155:0/1301138004 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb8e4020420 con 0x7fb8e80f51a0 2024-02-10T09:05:11.808 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:11.805+0000 7fb8de7fc700 1 --2- 172.21.15.155:0/1301138004 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb8d402ff60 0x7fb8d4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:05:11.808 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:11.805+0000 7fb8de7fc700 1 -- 172.21.15.155:0/1301138004 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb8e4042ba0 con 0x7fb8e80f51a0 2024-02-10T09:05:11.808 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:11.805+0000 7fb8ed7c2700 1 --2- 172.21.15.155:0/1301138004 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb8d402ff60 0x7fb8d4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:05:11.808 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:11.806+0000 7fb8ed7c2700 1 --2- 172.21.15.155:0/1301138004 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb8d402ff60 0x7fb8d4032420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fb8d80096a0 tx=0x7fb8d8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:05:11.811 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:11.810+0000 7fb8de7fc700 1 -- 172.21.15.155:0/1301138004 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb8e40095f0 con 0x7fb8e80f51a0 2024-02-10T09:05:12.018 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:12.017+0000 7fb8eefc5700 1 -- 172.21.15.155:0/1301138004 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb8e80fad30 con 0x7fb8e80f51a0 2024-02-10T09:05:12.019 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:12.018+0000 7fb8de7fc700 1 -- 172.21.15.155:0/1301138004 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb8e4011290 con 0x7fb8e80f51a0 2024-02-10T09:05:12.019 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:05:12.019 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:05:12.021 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:12.020+0000 7fb8eefc5700 1 -- 172.21.15.155:0/1301138004 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb8d402ff60 msgr2=0x7fb8d4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:05:12.021 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:12.020+0000 7fb8eefc5700 1 --2- 172.21.15.155:0/1301138004 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb8d402ff60 0x7fb8d4032420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fb8d80096a0 tx=0x7fb8d8006b40).stop 2024-02-10T09:05:12.021 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:12.020+0000 7fb8eefc5700 1 -- 172.21.15.155:0/1301138004 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8e80f51a0 msgr2=0x7fb8e81054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:05:12.021 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:12.020+0000 7fb8eefc5700 1 --2- 172.21.15.155:0/1301138004 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8e80f51a0 0x7fb8e81054d0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7fb8e4009830 tx=0x7fb8e400b040).stop 2024-02-10T09:05:12.021 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:12.020+0000 7fb8eefc5700 1 -- 172.21.15.155:0/1301138004 shutdown_connections 2024-02-10T09:05:12.021 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:12.020+0000 7fb8eefc5700 1 --2- 172.21.15.155:0/1301138004 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb8d402ff60 0x7fb8d4032420 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:05:12.022 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:12.020+0000 7fb8eefc5700 1 --2- 172.21.15.155:0/1301138004 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8e80f51a0 0x7fb8e81054d0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:05:12.022 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:12.020+0000 7fb8eefc5700 1 -- 172.21.15.155:0/1301138004 >> 172.21.15.155:0/1301138004 conn(0x7fb8e80f09a0 msgr2=0x7fb8e80f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:05:12.022 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:12.020+0000 7fb8eefc5700 1 -- 172.21.15.155:0/1301138004 shutdown_connections 2024-02-10T09:05:12.022 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:12.020+0000 7fb8eefc5700 1 -- 172.21.15.155:0/1301138004 wait complete. 2024-02-10T09:05:12.022 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:05:12.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:05:12 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:05:12.019572+0000 mon.smithi077 (mon.0) 225 : audit [DBG] from='client.? 172.21.15.155:0/1301138004' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:05:13.867 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:05:13.867 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:05:14.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:05:13 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:05:13.015350+0000 mgr.smithi077.dxntvw (mgr.14162) 199 : cluster [DBG] pgmap v187: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:05:15.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:05:15 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:05:15.015697+0000 mgr.smithi077.dxntvw (mgr.14162) 200 : cluster [DBG] pgmap v188: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:05:15.936 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:15.933+0000 7fe4492a0700 1 -- 172.21.15.155:0/201344187 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4440f5410 msgr2=0x7fe4440f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:05:15.937 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:15.933+0000 7fe4492a0700 1 --2- 172.21.15.155:0/201344187 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4440f5410 0x7fe4440f5830 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7fe438009fb0 tx=0x7fe438009b30).stop 2024-02-10T09:05:15.937 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:15.934+0000 7fe4492a0700 1 -- 172.21.15.155:0/201344187 shutdown_connections 2024-02-10T09:05:15.937 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:15.934+0000 7fe4492a0700 1 --2- 172.21.15.155:0/201344187 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4440f5410 0x7fe4440f5830 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:05:15.937 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:15.934+0000 7fe4492a0700 1 -- 172.21.15.155:0/201344187 >> 172.21.15.155:0/201344187 conn(0x7fe4440f0b70 msgr2=0x7fe4440f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:05:15.937 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:15.934+0000 7fe4492a0700 1 -- 172.21.15.155:0/201344187 shutdown_connections 2024-02-10T09:05:15.937 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:15.934+0000 7fe4492a0700 1 -- 172.21.15.155:0/201344187 wait complete. 2024-02-10T09:05:15.938 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:15.934+0000 7fe4492a0700 1 Processor -- start 2024-02-10T09:05:15.938 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:15.934+0000 7fe4492a0700 1 -- start start 2024-02-10T09:05:15.939 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:15.934+0000 7fe4492a0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4440f5410 0x7fe4440fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:05:15.939 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:15.934+0000 7fe4492a0700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4440fb960 con 0x7fe4440f5410 2024-02-10T09:05:15.939 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:15.935+0000 7fe442ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4440f5410 0x7fe4440fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:05:15.939 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:15.935+0000 7fe442ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4440f5410 0x7fe4440fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:36934/0 (socket says 172.21.15.155:36934) 2024-02-10T09:05:15.939 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:15.935+0000 7fe442ffd700 1 -- 172.21.15.155:0/1576636979 learned_addr learned my addr 172.21.15.155:0/1576636979 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:05:15.939 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:15.935+0000 7fe442ffd700 1 -- 172.21.15.155:0/1576636979 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe438005040 con 0x7fe4440f5410 2024-02-10T09:05:15.939 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:15.935+0000 7fe442ffd700 1 --2- 172.21.15.155:0/1576636979 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4440f5410 0x7fe4440fd4e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7fe438005010 tx=0x7fe43800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:05:15.939 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:15.935+0000 7fe42b7fe700 1 -- 172.21.15.155:0/1576636979 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe438009320 con 0x7fe4440f5410 2024-02-10T09:05:15.939 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:15.935+0000 7fe4492a0700 1 -- 172.21.15.155:0/1576636979 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4440fbb00 con 0x7fe4440f5410 2024-02-10T09:05:15.939 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:15.935+0000 7fe42b7fe700 1 -- 172.21.15.155:0/1576636979 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe438013380 con 0x7fe4440f5410 2024-02-10T09:05:15.939 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:15.935+0000 7fe42b7fe700 1 -- 172.21.15.155:0/1576636979 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe438014e50 con 0x7fe4440f5410 2024-02-10T09:05:15.939 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:15.935+0000 7fe4492a0700 1 -- 172.21.15.155:0/1576636979 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4440fbf90 con 0x7fe4440f5410 2024-02-10T09:05:15.940 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:15.936+0000 7fe42b7fe700 1 -- 172.21.15.155:0/1576636979 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe438014420 con 0x7fe4440f5410 2024-02-10T09:05:15.940 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:15.936+0000 7fe4492a0700 1 -- 172.21.15.155:0/1576636979 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe444059100 con 0x7fe4440f5410 2024-02-10T09:05:15.940 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:15.936+0000 7fe42b7fe700 1 --2- 172.21.15.155:0/1576636979 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fe42c02ff10 0x7fe42c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:05:15.941 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:15.936+0000 7fe42b7fe700 1 -- 172.21.15.155:0/1576636979 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe438040e30 con 0x7fe4440f5410 2024-02-10T09:05:15.941 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:15.936+0000 7fe4427fc700 1 --2- 172.21.15.155:0/1576636979 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fe42c02ff10 0x7fe42c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:05:15.941 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:15.937+0000 7fe4427fc700 1 --2- 172.21.15.155:0/1576636979 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fe42c02ff10 0x7fe42c0323d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fe4340096a0 tx=0x7fe434006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:05:15.942 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:15.941+0000 7fe42b7fe700 1 -- 172.21.15.155:0/1576636979 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe43801dd50 con 0x7fe4440f5410 2024-02-10T09:05:16.147 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:16.146+0000 7fe4492a0700 1 -- 172.21.15.155:0/1576636979 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe4440fafa0 con 0x7fe4440f5410 2024-02-10T09:05:16.148 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:16.146+0000 7fe42b7fe700 1 -- 172.21.15.155:0/1576636979 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe43801db70 con 0x7fe4440f5410 2024-02-10T09:05:16.148 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:05:16.148 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:05:16.149 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:16.148+0000 7fe4492a0700 1 -- 172.21.15.155:0/1576636979 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fe42c02ff10 msgr2=0x7fe42c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:05:16.149 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:16.148+0000 7fe4492a0700 1 --2- 172.21.15.155:0/1576636979 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fe42c02ff10 0x7fe42c0323d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fe4340096a0 tx=0x7fe434006b40).stop 2024-02-10T09:05:16.149 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:16.148+0000 7fe4492a0700 1 -- 172.21.15.155:0/1576636979 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4440f5410 msgr2=0x7fe4440fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:05:16.149 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:16.148+0000 7fe4492a0700 1 --2- 172.21.15.155:0/1576636979 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4440f5410 0x7fe4440fd4e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7fe438005010 tx=0x7fe43800b040).stop 2024-02-10T09:05:16.149 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:16.148+0000 7fe4492a0700 1 -- 172.21.15.155:0/1576636979 shutdown_connections 2024-02-10T09:05:16.149 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:16.148+0000 7fe4492a0700 1 --2- 172.21.15.155:0/1576636979 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fe42c02ff10 0x7fe42c0323d0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:05:16.149 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:16.148+0000 7fe4492a0700 1 --2- 172.21.15.155:0/1576636979 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4440f5410 0x7fe4440fd4e0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:05:16.150 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:16.148+0000 7fe4492a0700 1 -- 172.21.15.155:0/1576636979 >> 172.21.15.155:0/1576636979 conn(0x7fe4440f0b70 msgr2=0x7fe44405fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:05:16.150 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:16.148+0000 7fe4492a0700 1 -- 172.21.15.155:0/1576636979 shutdown_connections 2024-02-10T09:05:16.150 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:16.148+0000 7fe4492a0700 1 -- 172.21.15.155:0/1576636979 wait complete. 2024-02-10T09:05:16.150 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:05:16.649 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:05:16 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:05:16.147836+0000 mon.smithi077 (mon.0) 226 : audit [DBG] from='client.? 172.21.15.155:0/1576636979' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:05:17.649 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:05:17 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:05:17.016041+0000 mgr.smithi077.dxntvw (mgr.14162) 201 : cluster [DBG] pgmap v189: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:05:17.882 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:05:17.882 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:05:19.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:05:19 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:05:19.016435+0000 mgr.smithi077.dxntvw (mgr.14162) 202 : cluster [DBG] pgmap v190: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:05:20.026 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:20.024+0000 7fe3c3fff700 1 -- 172.21.15.155:0/1739169757 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3c40601e0 msgr2=0x7fe3c40f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:05:20.026 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:20.024+0000 7fe3c3fff700 1 --2- 172.21.15.155:0/1739169757 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3c40601e0 0x7fe3c40f8f20 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7fe3b8009fb0 tx=0x7fe3b8009b30).stop 2024-02-10T09:05:20.026 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:20.024+0000 7fe3c3fff700 1 -- 172.21.15.155:0/1739169757 shutdown_connections 2024-02-10T09:05:20.026 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:20.024+0000 7fe3c3fff700 1 --2- 172.21.15.155:0/1739169757 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3c40601e0 0x7fe3c40f8f20 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:05:20.027 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:20.024+0000 7fe3c3fff700 1 -- 172.21.15.155:0/1739169757 >> 172.21.15.155:0/1739169757 conn(0x7fe3c40f09a0 msgr2=0x7fe3c40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:05:20.027 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:20.024+0000 7fe3c3fff700 1 -- 172.21.15.155:0/1739169757 shutdown_connections 2024-02-10T09:05:20.027 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:20.024+0000 7fe3c3fff700 1 -- 172.21.15.155:0/1739169757 wait complete. 2024-02-10T09:05:20.027 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:20.025+0000 7fe3c3fff700 1 Processor -- start 2024-02-10T09:05:20.027 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:20.025+0000 7fe3c3fff700 1 -- start start 2024-02-10T09:05:20.027 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:20.025+0000 7fe3c3fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3c40601e0 0x7fe3c40f69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:05:20.027 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:20.025+0000 7fe3c3fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c40f4e20 con 0x7fe3c40601e0 2024-02-10T09:05:20.027 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:20.025+0000 7fe3c2ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3c40601e0 0x7fe3c40f69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:05:20.027 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:20.025+0000 7fe3c2ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3c40601e0 0x7fe3c40f69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:36956/0 (socket says 172.21.15.155:36956) 2024-02-10T09:05:20.027 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:20.025+0000 7fe3c2ffd700 1 -- 172.21.15.155:0/693694513 learned_addr learned my addr 172.21.15.155:0/693694513 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:05:20.029 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:20.026+0000 7fe3c2ffd700 1 -- 172.21.15.155:0/693694513 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3b8005040 con 0x7fe3c40601e0 2024-02-10T09:05:20.029 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:20.026+0000 7fe3c2ffd700 1 --2- 172.21.15.155:0/693694513 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3c40601e0 0x7fe3c40f69a0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7fe3b8005010 tx=0x7fe3b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:05:20.029 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:20.026+0000 7fe3ab7fe700 1 -- 172.21.15.155:0/693694513 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe3b8009320 con 0x7fe3c40601e0 2024-02-10T09:05:20.029 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:20.026+0000 7fe3c3fff700 1 -- 172.21.15.155:0/693694513 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3c40f4fc0 con 0x7fe3c40601e0 2024-02-10T09:05:20.029 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:20.026+0000 7fe3ab7fe700 1 -- 172.21.15.155:0/693694513 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe3b8013380 con 0x7fe3c40601e0 2024-02-10T09:05:20.029 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:20.026+0000 7fe3ab7fe700 1 -- 172.21.15.155:0/693694513 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe3b8014e50 con 0x7fe3c40601e0 2024-02-10T09:05:20.029 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:20.026+0000 7fe3c3fff700 1 -- 172.21.15.155:0/693694513 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3c40f5450 con 0x7fe3c40601e0 2024-02-10T09:05:20.029 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:20.027+0000 7fe3ab7fe700 1 -- 172.21.15.155:0/693694513 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe3b80095f0 con 0x7fe3c40601e0 2024-02-10T09:05:20.029 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:20.027+0000 7fe3c3fff700 1 -- 172.21.15.155:0/693694513 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3c4059100 con 0x7fe3c40601e0 2024-02-10T09:05:20.029 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:20.027+0000 7fe3ab7fe700 1 --2- 172.21.15.155:0/693694513 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fe3ac02ff10 0x7fe3ac0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:05:20.029 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:20.027+0000 7fe3ab7fe700 1 -- 172.21.15.155:0/693694513 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe3b8040e90 con 0x7fe3c40601e0 2024-02-10T09:05:20.029 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:20.027+0000 7fe3c27fc700 1 --2- 172.21.15.155:0/693694513 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fe3ac02ff10 0x7fe3ac0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:05:20.030 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:20.027+0000 7fe3c27fc700 1 --2- 172.21.15.155:0/693694513 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fe3ac02ff10 0x7fe3ac0323d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fe3b40096a0 tx=0x7fe3b4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:05:20.032 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:20.032+0000 7fe3ab7fe700 1 -- 172.21.15.155:0/693694513 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe3b8014420 con 0x7fe3c40601e0 2024-02-10T09:05:20.242 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:20.241+0000 7fe3c3fff700 1 -- 172.21.15.155:0/693694513 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe3c40fbe10 con 0x7fe3c40601e0 2024-02-10T09:05:20.242 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:20.241+0000 7fe3ab7fe700 1 -- 172.21.15.155:0/693694513 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe3b8010230 con 0x7fe3c40601e0 2024-02-10T09:05:20.242 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:05:20.243 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:05:20.244 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:20.243+0000 7fe3c3fff700 1 -- 172.21.15.155:0/693694513 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fe3ac02ff10 msgr2=0x7fe3ac0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:05:20.244 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:20.243+0000 7fe3c3fff700 1 --2- 172.21.15.155:0/693694513 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fe3ac02ff10 0x7fe3ac0323d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fe3b40096a0 tx=0x7fe3b4006b40).stop 2024-02-10T09:05:20.244 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:20.243+0000 7fe3c3fff700 1 -- 172.21.15.155:0/693694513 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3c40601e0 msgr2=0x7fe3c40f69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:05:20.244 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:20.243+0000 7fe3c3fff700 1 --2- 172.21.15.155:0/693694513 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3c40601e0 0x7fe3c40f69a0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7fe3b8005010 tx=0x7fe3b800b040).stop 2024-02-10T09:05:20.244 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:20.243+0000 7fe3c3fff700 1 -- 172.21.15.155:0/693694513 shutdown_connections 2024-02-10T09:05:20.244 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:20.243+0000 7fe3c3fff700 1 --2- 172.21.15.155:0/693694513 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fe3ac02ff10 0x7fe3ac0323d0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:05:20.244 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:20.243+0000 7fe3c3fff700 1 --2- 172.21.15.155:0/693694513 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe3c40601e0 0x7fe3c40f69a0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:05:20.244 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:20.243+0000 7fe3c3fff700 1 -- 172.21.15.155:0/693694513 >> 172.21.15.155:0/693694513 conn(0x7fe3c40f09a0 msgr2=0x7fe3c40f13d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:05:20.244 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:20.243+0000 7fe3c3fff700 1 -- 172.21.15.155:0/693694513 shutdown_connections 2024-02-10T09:05:20.244 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:20.243+0000 7fe3c3fff700 1 -- 172.21.15.155:0/693694513 wait complete. 2024-02-10T09:05:20.245 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:05:20.649 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:05:20 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:05:20.242758+0000 mon.smithi077 (mon.0) 227 : audit [DBG] from='client.? 172.21.15.155:0/693694513' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:05:21.649 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:05:21 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:05:21.016793+0000 mgr.smithi077.dxntvw (mgr.14162) 203 : cluster [DBG] pgmap v191: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:05:22.039 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:05:22.039 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:05:24.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:05:23 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:05:23.017065+0000 mgr.smithi077.dxntvw (mgr.14162) 204 : cluster [DBG] pgmap v192: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:05:24.189 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:24.187+0000 7f086b02e700 1 -- 172.21.15.155:0/61392461 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08640f5410 msgr2=0x7f08640f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:05:24.189 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:24.187+0000 7f086b02e700 1 --2- 172.21.15.155:0/61392461 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08640f5410 0x7f08640f5830 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7f0858009fb0 tx=0x7f0858009b30).stop 2024-02-10T09:05:24.189 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:24.187+0000 7f086b02e700 1 -- 172.21.15.155:0/61392461 shutdown_connections 2024-02-10T09:05:24.189 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:24.187+0000 7f086b02e700 1 --2- 172.21.15.155:0/61392461 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08640f5410 0x7f08640f5830 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:05:24.189 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:24.187+0000 7f086b02e700 1 -- 172.21.15.155:0/61392461 >> 172.21.15.155:0/61392461 conn(0x7f08640f0b70 msgr2=0x7f08640f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:05:24.189 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:24.187+0000 7f086b02e700 1 -- 172.21.15.155:0/61392461 shutdown_connections 2024-02-10T09:05:24.189 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:24.187+0000 7f086b02e700 1 -- 172.21.15.155:0/61392461 wait complete. 2024-02-10T09:05:24.190 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:24.188+0000 7f086b02e700 1 Processor -- start 2024-02-10T09:05:24.190 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:24.188+0000 7f086b02e700 1 -- start start 2024-02-10T09:05:24.190 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:24.188+0000 7f086b02e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08640f5410 0x7f08641067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:05:24.190 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:24.188+0000 7f086b02e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0864106cc0 con 0x7f08640f5410 2024-02-10T09:05:24.190 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:24.188+0000 7f0868dca700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08640f5410 0x7f08641067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:05:24.190 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:24.188+0000 7f0868dca700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08640f5410 0x7f08641067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:36708/0 (socket says 172.21.15.155:36708) 2024-02-10T09:05:24.190 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:24.188+0000 7f0868dca700 1 -- 172.21.15.155:0/3164968085 learned_addr learned my addr 172.21.15.155:0/3164968085 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:05:24.191 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:24.188+0000 7f0868dca700 1 -- 172.21.15.155:0/3164968085 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0858005040 con 0x7f08640f5410 2024-02-10T09:05:24.191 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:24.189+0000 7f0868dca700 1 --2- 172.21.15.155:0/3164968085 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08640f5410 0x7f08641067d0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f0858005010 tx=0x7f085800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:05:24.191 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:24.189+0000 7f08617fa700 1 -- 172.21.15.155:0/3164968085 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0858009320 con 0x7f08640f5410 2024-02-10T09:05:24.191 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:24.189+0000 7f086b02e700 1 -- 172.21.15.155:0/3164968085 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0864106ec0 con 0x7f08640f5410 2024-02-10T09:05:24.192 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:24.189+0000 7f08617fa700 1 -- 172.21.15.155:0/3164968085 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0858013380 con 0x7f08640f5410 2024-02-10T09:05:24.192 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:24.189+0000 7f08617fa700 1 -- 172.21.15.155:0/3164968085 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0858014e50 con 0x7f08640f5410 2024-02-10T09:05:24.192 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:24.189+0000 7f086b02e700 1 -- 172.21.15.155:0/3164968085 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0864107350 con 0x7f08640f5410 2024-02-10T09:05:24.192 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:24.190+0000 7f08617fa700 1 -- 172.21.15.155:0/3164968085 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f08580263a0 con 0x7f08640f5410 2024-02-10T09:05:24.192 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:24.190+0000 7f086b02e700 1 -- 172.21.15.155:0/3164968085 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0864045ad0 con 0x7f08640f5410 2024-02-10T09:05:24.192 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:24.190+0000 7f08617fa700 1 --2- 172.21.15.155:0/3164968085 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f085402ff10 0x7f08540323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:05:24.192 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:24.190+0000 7f08617fa700 1 -- 172.21.15.155:0/3164968085 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0858040fb0 con 0x7f08640f5410 2024-02-10T09:05:24.192 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:24.190+0000 7f0863fff700 1 --2- 172.21.15.155:0/3164968085 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f085402ff10 0x7f08540323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:05:24.192 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:24.190+0000 7f0863fff700 1 --2- 172.21.15.155:0/3164968085 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f085402ff10 0x7f08540323d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f08500096a0 tx=0x7f0850006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:05:24.195 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:24.195+0000 7f08617fa700 1 -- 172.21.15.155:0/3164968085 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f08580126f0 con 0x7f08640f5410 2024-02-10T09:05:24.404 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:24.403+0000 7f086b02e700 1 -- 172.21.15.155:0/3164968085 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0864108200 con 0x7f08640f5410 2024-02-10T09:05:24.405 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:24.404+0000 7f08617fa700 1 -- 172.21.15.155:0/3164968085 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0858012330 con 0x7f08640f5410 2024-02-10T09:05:24.405 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:05:24.405 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:05:24.407 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:24.405+0000 7f086b02e700 1 -- 172.21.15.155:0/3164968085 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f085402ff10 msgr2=0x7f08540323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:05:24.407 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:24.405+0000 7f086b02e700 1 --2- 172.21.15.155:0/3164968085 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f085402ff10 0x7f08540323d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f08500096a0 tx=0x7f0850006b40).stop 2024-02-10T09:05:24.407 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:24.405+0000 7f086b02e700 1 -- 172.21.15.155:0/3164968085 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08640f5410 msgr2=0x7f08641067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:05:24.407 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:24.405+0000 7f086b02e700 1 --2- 172.21.15.155:0/3164968085 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08640f5410 0x7f08641067d0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f0858005010 tx=0x7f085800b040).stop 2024-02-10T09:05:24.407 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:24.406+0000 7f086b02e700 1 -- 172.21.15.155:0/3164968085 shutdown_connections 2024-02-10T09:05:24.407 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:24.406+0000 7f086b02e700 1 --2- 172.21.15.155:0/3164968085 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f085402ff10 0x7f08540323d0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:05:24.407 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:24.406+0000 7f086b02e700 1 --2- 172.21.15.155:0/3164968085 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08640f5410 0x7f08641067d0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:05:24.407 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:24.406+0000 7f086b02e700 1 -- 172.21.15.155:0/3164968085 >> 172.21.15.155:0/3164968085 conn(0x7f08640f0b70 msgr2=0x7f086405fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:05:24.407 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:24.406+0000 7f086b02e700 1 -- 172.21.15.155:0/3164968085 shutdown_connections 2024-02-10T09:05:24.407 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:24.406+0000 7f086b02e700 1 -- 172.21.15.155:0/3164968085 wait complete. 2024-02-10T09:05:24.408 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:05:25.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:05:24 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:05:24.405482+0000 mon.smithi077 (mon.0) 228 : audit [DBG] from='client.? 172.21.15.155:0/3164968085' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:05:26.026 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:05:26.027 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:05:26.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:05:25 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:05:25.017354+0000 mgr.smithi077.dxntvw (mgr.14162) 205 : cluster [DBG] pgmap v193: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:05:27.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:05:27 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:05:27.017627+0000 mgr.smithi077.dxntvw (mgr.14162) 206 : cluster [DBG] pgmap v194: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:05:28.070 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:28.067+0000 7f850a6be700 1 -- 172.21.15.155:0/1473142718 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85040601e0 msgr2=0x7f85040f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:05:28.070 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:28.067+0000 7f850a6be700 1 --2- 172.21.15.155:0/1473142718 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85040601e0 0x7f85040f8f20 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f84f4009fb0 tx=0x7f84f4009b30).stop 2024-02-10T09:05:28.071 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:28.067+0000 7f850a6be700 1 -- 172.21.15.155:0/1473142718 shutdown_connections 2024-02-10T09:05:28.071 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:28.067+0000 7f850a6be700 1 --2- 172.21.15.155:0/1473142718 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85040601e0 0x7f85040f8f20 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:05:28.071 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:28.067+0000 7f850a6be700 1 -- 172.21.15.155:0/1473142718 >> 172.21.15.155:0/1473142718 conn(0x7f85040f09a0 msgr2=0x7f85040f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:05:28.071 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:28.068+0000 7f850a6be700 1 -- 172.21.15.155:0/1473142718 shutdown_connections 2024-02-10T09:05:28.071 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:28.068+0000 7f850a6be700 1 -- 172.21.15.155:0/1473142718 wait complete. 2024-02-10T09:05:28.071 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:28.068+0000 7f850a6be700 1 Processor -- start 2024-02-10T09:05:28.071 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:28.068+0000 7f850a6be700 1 -- start start 2024-02-10T09:05:28.071 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:28.068+0000 7f850a6be700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85040601e0 0x7f85040f69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:05:28.071 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:28.068+0000 7f850a6be700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85040f4e20 con 0x7f85040601e0 2024-02-10T09:05:28.071 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:28.068+0000 7f85096bc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85040601e0 0x7f85040f69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:05:28.071 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:28.068+0000 7f85096bc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85040601e0 0x7f85040f69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:36732/0 (socket says 172.21.15.155:36732) 2024-02-10T09:05:28.071 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:28.068+0000 7f85096bc700 1 -- 172.21.15.155:0/2782345021 learned_addr learned my addr 172.21.15.155:0/2782345021 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:05:28.071 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:28.069+0000 7f85096bc700 1 -- 172.21.15.155:0/2782345021 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84f4005040 con 0x7f85040601e0 2024-02-10T09:05:28.071 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:28.069+0000 7f85096bc700 1 --2- 172.21.15.155:0/2782345021 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85040601e0 0x7f85040f69a0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f84f4005010 tx=0x7f84f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:05:28.071 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:28.069+0000 7f84f9ffb700 1 -- 172.21.15.155:0/2782345021 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f84f4009320 con 0x7f85040601e0 2024-02-10T09:05:28.071 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:28.069+0000 7f850a6be700 1 -- 172.21.15.155:0/2782345021 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85040f4fc0 con 0x7f85040601e0 2024-02-10T09:05:28.072 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:28.069+0000 7f84f9ffb700 1 -- 172.21.15.155:0/2782345021 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f84f4013380 con 0x7f85040601e0 2024-02-10T09:05:28.072 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:28.069+0000 7f84f9ffb700 1 -- 172.21.15.155:0/2782345021 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f84f4014e50 con 0x7f85040601e0 2024-02-10T09:05:28.072 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:28.069+0000 7f850a6be700 1 -- 172.21.15.155:0/2782345021 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85040f5450 con 0x7f85040601e0 2024-02-10T09:05:28.072 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:28.070+0000 7f84f9ffb700 1 -- 172.21.15.155:0/2782345021 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f84f40263a0 con 0x7f85040601e0 2024-02-10T09:05:28.073 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:28.070+0000 7f84f9ffb700 1 --2- 172.21.15.155:0/2782345021 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f84f002ff10 0x7f84f00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:05:28.073 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:28.070+0000 7f850a6be700 1 -- 172.21.15.155:0/2782345021 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8504059100 con 0x7f85040601e0 2024-02-10T09:05:28.073 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:28.070+0000 7f84f9ffb700 1 -- 172.21.15.155:0/2782345021 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f84f4041240 con 0x7f85040601e0 2024-02-10T09:05:28.073 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:28.071+0000 7f8508ebb700 1 --2- 172.21.15.155:0/2782345021 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f84f002ff10 0x7f84f00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:05:28.073 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:28.071+0000 7f8508ebb700 1 --2- 172.21.15.155:0/2782345021 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f84f002ff10 0x7f84f00323d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f85000096a0 tx=0x7f8500006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:05:28.076 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:28.075+0000 7f84f9ffb700 1 -- 172.21.15.155:0/2782345021 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f84f4012920 con 0x7f85040601e0 2024-02-10T09:05:28.284 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:28.283+0000 7f850a6be700 1 -- 172.21.15.155:0/2782345021 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8504045ad0 con 0x7f85040601e0 2024-02-10T09:05:28.285 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:28.284+0000 7f84f9ffb700 1 -- 172.21.15.155:0/2782345021 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f84f402b030 con 0x7f85040601e0 2024-02-10T09:05:28.285 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:05:28.285 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:05:28.287 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:28.285+0000 7f850a6be700 1 -- 172.21.15.155:0/2782345021 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f84f002ff10 msgr2=0x7f84f00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:05:28.287 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:28.285+0000 7f850a6be700 1 --2- 172.21.15.155:0/2782345021 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f84f002ff10 0x7f84f00323d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f85000096a0 tx=0x7f8500006b40).stop 2024-02-10T09:05:28.287 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:28.285+0000 7f850a6be700 1 -- 172.21.15.155:0/2782345021 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85040601e0 msgr2=0x7f85040f69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:05:28.287 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:28.285+0000 7f850a6be700 1 --2- 172.21.15.155:0/2782345021 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85040601e0 0x7f85040f69a0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f84f4005010 tx=0x7f84f400b040).stop 2024-02-10T09:05:28.287 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:28.285+0000 7f850a6be700 1 -- 172.21.15.155:0/2782345021 shutdown_connections 2024-02-10T09:05:28.287 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:28.285+0000 7f850a6be700 1 --2- 172.21.15.155:0/2782345021 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f84f002ff10 0x7f84f00323d0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:05:28.287 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:28.285+0000 7f850a6be700 1 --2- 172.21.15.155:0/2782345021 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f85040601e0 0x7f85040f69a0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:05:28.287 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:28.285+0000 7f850a6be700 1 -- 172.21.15.155:0/2782345021 >> 172.21.15.155:0/2782345021 conn(0x7f85040f09a0 msgr2=0x7f85040f1390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:05:28.287 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:28.286+0000 7f850a6be700 1 -- 172.21.15.155:0/2782345021 shutdown_connections 2024-02-10T09:05:28.287 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:28.286+0000 7f850a6be700 1 -- 172.21.15.155:0/2782345021 wait complete. 2024-02-10T09:05:28.288 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:05:29.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:05:28 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:05:28.285392+0000 mon.smithi077 (mon.0) 229 : audit [DBG] from='client.? 172.21.15.155:0/2782345021' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:05:30.129 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:05:30.129 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:05:30.148 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:05:29 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:05:29.017997+0000 mgr.smithi077.dxntvw (mgr.14162) 207 : cluster [DBG] pgmap v195: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:05:31.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:05:31 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:05:31.018345+0000 mgr.smithi077.dxntvw (mgr.14162) 208 : cluster [DBG] pgmap v196: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:05:32.088 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:32.085+0000 7f190c170700 1 -- 172.21.15.155:0/1332512934 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1904060280 msgr2=0x7f19040f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:05:32.088 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:32.085+0000 7f190c170700 1 --2- 172.21.15.155:0/1332512934 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1904060280 0x7f19040f90f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f18f4009fb0 tx=0x7f18f4009b30).stop 2024-02-10T09:05:32.089 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:32.085+0000 7f190c170700 1 -- 172.21.15.155:0/1332512934 shutdown_connections 2024-02-10T09:05:32.089 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:32.085+0000 7f190c170700 1 --2- 172.21.15.155:0/1332512934 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1904060280 0x7f19040f90f0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:05:32.089 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:32.085+0000 7f190c170700 1 -- 172.21.15.155:0/1332512934 >> 172.21.15.155:0/1332512934 conn(0x7f19040f0b70 msgr2=0x7f19040f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:05:32.089 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:32.085+0000 7f190c170700 1 -- 172.21.15.155:0/1332512934 shutdown_connections 2024-02-10T09:05:32.089 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:32.085+0000 7f190c170700 1 -- 172.21.15.155:0/1332512934 wait complete. 2024-02-10T09:05:32.089 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:32.086+0000 7f190c170700 1 Processor -- start 2024-02-10T09:05:32.089 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:32.086+0000 7f190c170700 1 -- start start 2024-02-10T09:05:32.089 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:32.086+0000 7f190c170700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1904060280 0x7f19040f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:05:32.089 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:32.086+0000 7f190c170700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19040f4ff0 con 0x7f1904060280 2024-02-10T09:05:32.089 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:32.086+0000 7f1909f0c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1904060280 0x7f19040f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:05:32.089 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:32.086+0000 7f1909f0c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1904060280 0x7f19040f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:36752/0 (socket says 172.21.15.155:36752) 2024-02-10T09:05:32.089 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:32.086+0000 7f1909f0c700 1 -- 172.21.15.155:0/3871321532 learned_addr learned my addr 172.21.15.155:0/3871321532 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:05:32.089 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:32.087+0000 7f1909f0c700 1 -- 172.21.15.155:0/3871321532 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18f4005040 con 0x7f1904060280 2024-02-10T09:05:32.090 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:32.087+0000 7f1909f0c700 1 --2- 172.21.15.155:0/3871321532 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1904060280 0x7f19040f6b70 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f18f40089f0 tx=0x7f18f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:05:32.090 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:32.087+0000 7f18fa7fc700 1 -- 172.21.15.155:0/3871321532 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f18f4009320 con 0x7f1904060280 2024-02-10T09:05:32.090 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:32.087+0000 7f190c170700 1 -- 172.21.15.155:0/3871321532 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19040f5190 con 0x7f1904060280 2024-02-10T09:05:32.090 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:32.087+0000 7f18fa7fc700 1 -- 172.21.15.155:0/3871321532 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f18f4013380 con 0x7f1904060280 2024-02-10T09:05:32.090 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:32.087+0000 7f18fa7fc700 1 -- 172.21.15.155:0/3871321532 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f18f4014e20 con 0x7f1904060280 2024-02-10T09:05:32.090 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:32.087+0000 7f190c170700 1 -- 172.21.15.155:0/3871321532 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19040f5620 con 0x7f1904060280 2024-02-10T09:05:32.091 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:32.088+0000 7f190c170700 1 -- 172.21.15.155:0/3871321532 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1904059100 con 0x7f1904060280 2024-02-10T09:05:32.091 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:32.088+0000 7f18fa7fc700 1 -- 172.21.15.155:0/3871321532 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f18f40095f0 con 0x7f1904060280 2024-02-10T09:05:32.091 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:32.088+0000 7f18fa7fc700 1 --2- 172.21.15.155:0/3871321532 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f18f002ff10 0x7f18f00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:05:32.091 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:32.088+0000 7f18fa7fc700 1 -- 172.21.15.155:0/3871321532 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f18f4040e90 con 0x7f1904060280 2024-02-10T09:05:32.091 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:32.088+0000 7f190970b700 1 --2- 172.21.15.155:0/3871321532 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f18f002ff10 0x7f18f00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:05:32.091 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:32.089+0000 7f190970b700 1 --2- 172.21.15.155:0/3871321532 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f18f002ff10 0x7f18f00323d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f19000096a0 tx=0x7f1900006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:05:32.097 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:32.097+0000 7f18fa7fc700 1 -- 172.21.15.155:0/3871321532 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f18f4014420 con 0x7f1904060280 2024-02-10T09:05:32.303 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:32.303+0000 7f190c170700 1 -- 172.21.15.155:0/3871321532 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f19040fc090 con 0x7f1904060280 2024-02-10T09:05:32.304 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:32.304+0000 7f18fa7fc700 1 -- 172.21.15.155:0/3871321532 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f18f401d980 con 0x7f1904060280 2024-02-10T09:05:32.304 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:05:32.304 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:05:32.306 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:32.305+0000 7f190c170700 1 -- 172.21.15.155:0/3871321532 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f18f002ff10 msgr2=0x7f18f00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:05:32.306 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:32.305+0000 7f190c170700 1 --2- 172.21.15.155:0/3871321532 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f18f002ff10 0x7f18f00323d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f19000096a0 tx=0x7f1900006b40).stop 2024-02-10T09:05:32.306 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:32.305+0000 7f190c170700 1 -- 172.21.15.155:0/3871321532 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1904060280 msgr2=0x7f19040f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:05:32.306 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:32.305+0000 7f190c170700 1 --2- 172.21.15.155:0/3871321532 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1904060280 0x7f19040f6b70 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f18f40089f0 tx=0x7f18f400b040).stop 2024-02-10T09:05:32.306 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:32.305+0000 7f190c170700 1 -- 172.21.15.155:0/3871321532 shutdown_connections 2024-02-10T09:05:32.306 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:32.305+0000 7f190c170700 1 --2- 172.21.15.155:0/3871321532 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f18f002ff10 0x7f18f00323d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:05:32.306 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:32.305+0000 7f190c170700 1 --2- 172.21.15.155:0/3871321532 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1904060280 0x7f19040f6b70 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:05:32.306 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:32.305+0000 7f190c170700 1 -- 172.21.15.155:0/3871321532 >> 172.21.15.155:0/3871321532 conn(0x7f19040f0b70 msgr2=0x7f19040f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:05:32.306 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:32.305+0000 7f190c170700 1 -- 172.21.15.155:0/3871321532 shutdown_connections 2024-02-10T09:05:32.306 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:32.305+0000 7f190c170700 1 -- 172.21.15.155:0/3871321532 wait complete. 2024-02-10T09:05:32.306 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:05:32.649 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:05:32 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:05:32.304317+0000 mon.smithi077 (mon.0) 230 : audit [DBG] from='client.? 172.21.15.155:0/3871321532' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:05:34.001 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:05:34.001 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:05:34.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:05:33 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:05:33.018704+0000 mgr.smithi077.dxntvw (mgr.14162) 209 : cluster [DBG] pgmap v197: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:05:35.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:05:35 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:05:35.019047+0000 mgr.smithi077.dxntvw (mgr.14162) 210 : cluster [DBG] pgmap v198: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:05:35.996 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:35.993+0000 7fdd16ca3700 1 -- 172.21.15.155:0/1203669404 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd100f51a0 msgr2=0x7fdd100f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:05:35.996 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:35.993+0000 7fdd16ca3700 1 --2- 172.21.15.155:0/1203669404 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd100f51a0 0x7fdd100f55c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7fdd00009fb0 tx=0x7fdd00009b30).stop 2024-02-10T09:05:35.996 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:35.994+0000 7fdd16ca3700 1 -- 172.21.15.155:0/1203669404 shutdown_connections 2024-02-10T09:05:35.996 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:35.994+0000 7fdd16ca3700 1 --2- 172.21.15.155:0/1203669404 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd100f51a0 0x7fdd100f55c0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:05:35.996 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:35.994+0000 7fdd16ca3700 1 -- 172.21.15.155:0/1203669404 >> 172.21.15.155:0/1203669404 conn(0x7fdd100f09a0 msgr2=0x7fdd100f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:05:35.996 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:35.994+0000 7fdd16ca3700 1 -- 172.21.15.155:0/1203669404 shutdown_connections 2024-02-10T09:05:35.996 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:35.994+0000 7fdd16ca3700 1 -- 172.21.15.155:0/1203669404 wait complete. 2024-02-10T09:05:35.996 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:35.994+0000 7fdd16ca3700 1 Processor -- start 2024-02-10T09:05:35.996 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:35.994+0000 7fdd16ca3700 1 -- start start 2024-02-10T09:05:35.997 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:35.994+0000 7fdd16ca3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd100f51a0 0x7fdd101054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:05:35.997 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:35.994+0000 7fdd16ca3700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd101059c0 con 0x7fdd100f51a0 2024-02-10T09:05:35.997 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:35.995+0000 7fdd15ca1700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd100f51a0 0x7fdd101054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:05:35.997 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:35.995+0000 7fdd15ca1700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd100f51a0 0x7fdd101054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:51380/0 (socket says 172.21.15.155:51380) 2024-02-10T09:05:35.997 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:35.995+0000 7fdd15ca1700 1 -- 172.21.15.155:0/2201762849 learned_addr learned my addr 172.21.15.155:0/2201762849 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:05:35.997 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:35.995+0000 7fdd15ca1700 1 -- 172.21.15.155:0/2201762849 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd00005040 con 0x7fdd100f51a0 2024-02-10T09:05:35.997 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:35.995+0000 7fdd15ca1700 1 --2- 172.21.15.155:0/2201762849 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd100f51a0 0x7fdd101054d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7fdd000089f0 tx=0x7fdd0000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:05:35.997 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:35.995+0000 7fdd067fc700 1 -- 172.21.15.155:0/2201762849 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd00009320 con 0x7fdd100f51a0 2024-02-10T09:05:35.997 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:35.995+0000 7fdd067fc700 1 -- 172.21.15.155:0/2201762849 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdd00013380 con 0x7fdd100f51a0 2024-02-10T09:05:35.997 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:35.995+0000 7fdd16ca3700 1 -- 172.21.15.155:0/2201762849 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd10105bc0 con 0x7fdd100f51a0 2024-02-10T09:05:35.997 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:35.995+0000 7fdd067fc700 1 -- 172.21.15.155:0/2201762849 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd00014e20 con 0x7fdd100f51a0 2024-02-10T09:05:35.997 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:35.995+0000 7fdd16ca3700 1 -- 172.21.15.155:0/2201762849 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd10106050 con 0x7fdd100f51a0 2024-02-10T09:05:35.998 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:35.996+0000 7fdd067fc700 1 -- 172.21.15.155:0/2201762849 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdd000095f0 con 0x7fdd100f51a0 2024-02-10T09:05:35.998 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:35.996+0000 7fdd16ca3700 1 -- 172.21.15.155:0/2201762849 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd10045ad0 con 0x7fdd100f51a0 2024-02-10T09:05:35.998 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:35.996+0000 7fdd067fc700 1 --2- 172.21.15.155:0/2201762849 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fdcfc02ff60 0x7fdcfc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:05:35.999 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:35.996+0000 7fdd067fc700 1 -- 172.21.15.155:0/2201762849 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdd00040f00 con 0x7fdd100f51a0 2024-02-10T09:05:35.999 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:35.997+0000 7fdd154a0700 1 --2- 172.21.15.155:0/2201762849 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fdcfc02ff60 0x7fdcfc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:05:35.999 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:35.997+0000 7fdd154a0700 1 --2- 172.21.15.155:0/2201762849 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fdcfc02ff60 0x7fdcfc032420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fdd0c0096a0 tx=0x7fdd0c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:05:36.001 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:36.001+0000 7fdd067fc700 1 -- 172.21.15.155:0/2201762849 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdd00014420 con 0x7fdd100f51a0 2024-02-10T09:05:36.208 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:36.208+0000 7fdd16ca3700 1 -- 172.21.15.155:0/2201762849 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdd10106f00 con 0x7fdd100f51a0 2024-02-10T09:05:36.208 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:36.208+0000 7fdd067fc700 1 -- 172.21.15.155:0/2201762849 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdd00010230 con 0x7fdd100f51a0 2024-02-10T09:05:36.208 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:05:36.209 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:05:36.210 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:36.210+0000 7fdd16ca3700 1 -- 172.21.15.155:0/2201762849 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fdcfc02ff60 msgr2=0x7fdcfc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:05:36.210 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:36.210+0000 7fdd16ca3700 1 --2- 172.21.15.155:0/2201762849 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fdcfc02ff60 0x7fdcfc032420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fdd0c0096a0 tx=0x7fdd0c006b40).stop 2024-02-10T09:05:36.210 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:36.210+0000 7fdd16ca3700 1 -- 172.21.15.155:0/2201762849 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd100f51a0 msgr2=0x7fdd101054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:05:36.210 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:36.210+0000 7fdd16ca3700 1 --2- 172.21.15.155:0/2201762849 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd100f51a0 0x7fdd101054d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7fdd000089f0 tx=0x7fdd0000b040).stop 2024-02-10T09:05:36.210 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:36.210+0000 7fdd16ca3700 1 -- 172.21.15.155:0/2201762849 shutdown_connections 2024-02-10T09:05:36.210 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:36.210+0000 7fdd16ca3700 1 --2- 172.21.15.155:0/2201762849 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fdcfc02ff60 0x7fdcfc032420 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:05:36.211 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:36.210+0000 7fdd16ca3700 1 --2- 172.21.15.155:0/2201762849 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd100f51a0 0x7fdd101054d0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:05:36.211 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:36.210+0000 7fdd16ca3700 1 -- 172.21.15.155:0/2201762849 >> 172.21.15.155:0/2201762849 conn(0x7fdd100f09a0 msgr2=0x7fdd100f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:05:36.211 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:36.210+0000 7fdd16ca3700 1 -- 172.21.15.155:0/2201762849 shutdown_connections 2024-02-10T09:05:36.211 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:36.210+0000 7fdd16ca3700 1 -- 172.21.15.155:0/2201762849 wait complete. 2024-02-10T09:05:36.211 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:05:36.649 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:05:36 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:05:36.208735+0000 mon.smithi077 (mon.0) 231 : audit [DBG] from='client.? 172.21.15.155:0/2201762849' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:05:37.649 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:05:37 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:05:37.019351+0000 mgr.smithi077.dxntvw (mgr.14162) 211 : cluster [DBG] pgmap v199: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:05:38.109 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:05:38.110 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:05:39.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:05:39 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:05:39.019700+0000 mgr.smithi077.dxntvw (mgr.14162) 212 : cluster [DBG] pgmap v200: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:05:40.139 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:40.136+0000 7fd912e2c700 1 -- 172.21.15.155:0/1108185604 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd90c0f5410 msgr2=0x7fd90c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:05:40.139 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:40.136+0000 7fd912e2c700 1 --2- 172.21.15.155:0/1108185604 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd90c0f5410 0x7fd90c0f5830 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7fd900009fb0 tx=0x7fd900009b30).stop 2024-02-10T09:05:40.140 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:40.136+0000 7fd912e2c700 1 -- 172.21.15.155:0/1108185604 shutdown_connections 2024-02-10T09:05:40.140 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:40.136+0000 7fd912e2c700 1 --2- 172.21.15.155:0/1108185604 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd90c0f5410 0x7fd90c0f5830 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:05:40.140 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:40.136+0000 7fd912e2c700 1 -- 172.21.15.155:0/1108185604 >> 172.21.15.155:0/1108185604 conn(0x7fd90c0f0b70 msgr2=0x7fd90c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:05:40.140 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:40.136+0000 7fd912e2c700 1 -- 172.21.15.155:0/1108185604 shutdown_connections 2024-02-10T09:05:40.141 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:40.136+0000 7fd912e2c700 1 -- 172.21.15.155:0/1108185604 wait complete. 2024-02-10T09:05:40.141 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:40.136+0000 7fd912e2c700 1 Processor -- start 2024-02-10T09:05:40.141 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:40.136+0000 7fd912e2c700 1 -- start start 2024-02-10T09:05:40.141 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:40.137+0000 7fd912e2c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd90c0f5410 0x7fd90c1094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:05:40.141 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:40.137+0000 7fd912e2c700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd90c060240 con 0x7fd90c0f5410 2024-02-10T09:05:40.141 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:40.137+0000 7fd910bc8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd90c0f5410 0x7fd90c1094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:05:40.141 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:40.137+0000 7fd910bc8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd90c0f5410 0x7fd90c1094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:51402/0 (socket says 172.21.15.155:51402) 2024-02-10T09:05:40.141 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:40.137+0000 7fd910bc8700 1 -- 172.21.15.155:0/1540780168 learned_addr learned my addr 172.21.15.155:0/1540780168 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:05:40.142 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:40.138+0000 7fd910bc8700 1 -- 172.21.15.155:0/1540780168 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd900005040 con 0x7fd90c0f5410 2024-02-10T09:05:40.142 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:40.138+0000 7fd910bc8700 1 --2- 172.21.15.155:0/1540780168 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd90c0f5410 0x7fd90c1094e0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7fd900005010 tx=0x7fd90000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:05:40.142 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:40.138+0000 7fd9097fa700 1 -- 172.21.15.155:0/1540780168 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd900009320 con 0x7fd90c0f5410 2024-02-10T09:05:40.142 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:40.138+0000 7fd912e2c700 1 -- 172.21.15.155:0/1540780168 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd90c060440 con 0x7fd90c0f5410 2024-02-10T09:05:40.142 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:40.138+0000 7fd9097fa700 1 -- 172.21.15.155:0/1540780168 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd900013380 con 0x7fd90c0f5410 2024-02-10T09:05:40.142 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:40.138+0000 7fd9097fa700 1 -- 172.21.15.155:0/1540780168 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd900014e50 con 0x7fd90c0f5410 2024-02-10T09:05:40.142 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:40.138+0000 7fd912e2c700 1 -- 172.21.15.155:0/1540780168 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd90c109b90 con 0x7fd90c0f5410 2024-02-10T09:05:40.142 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:40.139+0000 7fd9097fa700 1 -- 172.21.15.155:0/1540780168 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd9000263a0 con 0x7fd90c0f5410 2024-02-10T09:05:40.142 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:40.139+0000 7fd912e2c700 1 -- 172.21.15.155:0/1540780168 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd90c045ad0 con 0x7fd90c0f5410 2024-02-10T09:05:40.142 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:40.139+0000 7fd9097fa700 1 --2- 172.21.15.155:0/1540780168 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fd8fc02ff10 0x7fd8fc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:05:40.142 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:40.139+0000 7fd9097fa700 1 -- 172.21.15.155:0/1540780168 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd900040f50 con 0x7fd90c0f5410 2024-02-10T09:05:40.142 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:40.139+0000 7fd90bfff700 1 --2- 172.21.15.155:0/1540780168 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fd8fc02ff10 0x7fd8fc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:05:40.143 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:40.140+0000 7fd90bfff700 1 --2- 172.21.15.155:0/1540780168 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fd8fc02ff10 0x7fd8fc0323d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fd8f80096a0 tx=0x7fd8f8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:05:40.144 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:40.144+0000 7fd9097fa700 1 -- 172.21.15.155:0/1540780168 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd9000126f0 con 0x7fd90c0f5410 2024-02-10T09:05:40.352 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:40.351+0000 7fd912e2c700 1 -- 172.21.15.155:0/1540780168 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd90c10a3b0 con 0x7fd90c0f5410 2024-02-10T09:05:40.352 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:40.352+0000 7fd9097fa700 1 -- 172.21.15.155:0/1540780168 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd900012330 con 0x7fd90c0f5410 2024-02-10T09:05:40.352 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:05:40.353 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:05:40.354 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:40.354+0000 7fd912e2c700 1 -- 172.21.15.155:0/1540780168 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fd8fc02ff10 msgr2=0x7fd8fc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:05:40.354 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:40.354+0000 7fd912e2c700 1 --2- 172.21.15.155:0/1540780168 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fd8fc02ff10 0x7fd8fc0323d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fd8f80096a0 tx=0x7fd8f8006b40).stop 2024-02-10T09:05:40.354 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:40.354+0000 7fd912e2c700 1 -- 172.21.15.155:0/1540780168 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd90c0f5410 msgr2=0x7fd90c1094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:05:40.354 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:40.354+0000 7fd912e2c700 1 --2- 172.21.15.155:0/1540780168 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd90c0f5410 0x7fd90c1094e0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7fd900005010 tx=0x7fd90000b040).stop 2024-02-10T09:05:40.354 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:40.354+0000 7fd912e2c700 1 -- 172.21.15.155:0/1540780168 shutdown_connections 2024-02-10T09:05:40.355 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:40.354+0000 7fd912e2c700 1 --2- 172.21.15.155:0/1540780168 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fd8fc02ff10 0x7fd8fc0323d0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:05:40.355 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:40.354+0000 7fd912e2c700 1 --2- 172.21.15.155:0/1540780168 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd90c0f5410 0x7fd90c1094e0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:05:40.355 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:40.354+0000 7fd912e2c700 1 -- 172.21.15.155:0/1540780168 >> 172.21.15.155:0/1540780168 conn(0x7fd90c0f0b70 msgr2=0x7fd90c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:05:40.355 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:40.354+0000 7fd912e2c700 1 -- 172.21.15.155:0/1540780168 shutdown_connections 2024-02-10T09:05:40.355 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:40.354+0000 7fd912e2c700 1 -- 172.21.15.155:0/1540780168 wait complete. 2024-02-10T09:05:40.355 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:05:40.899 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:05:40 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:05:40.352633+0000 mon.smithi077 (mon.0) 232 : audit [DBG] from='client.? 172.21.15.155:0/1540780168' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:05:41.899 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:05:41 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:05:41.020035+0000 mgr.smithi077.dxntvw (mgr.14162) 213 : cluster [DBG] pgmap v201: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:05:42.074 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:05:42.074 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:05:44.097 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:44.095+0000 7fc809cb9700 1 -- 172.21.15.155:0/114251243 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc804060280 msgr2=0x7fc8040f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:05:44.097 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:44.095+0000 7fc809cb9700 1 --2- 172.21.15.155:0/114251243 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc804060280 0x7fc8040f90f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7fc7ec009fb0 tx=0x7fc7ec009b30).stop 2024-02-10T09:05:44.097 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:44.095+0000 7fc809cb9700 1 -- 172.21.15.155:0/114251243 shutdown_connections 2024-02-10T09:05:44.097 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:44.095+0000 7fc809cb9700 1 --2- 172.21.15.155:0/114251243 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc804060280 0x7fc8040f90f0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:05:44.097 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:44.095+0000 7fc809cb9700 1 -- 172.21.15.155:0/114251243 >> 172.21.15.155:0/114251243 conn(0x7fc8040f0b70 msgr2=0x7fc8040f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:05:44.098 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:44.095+0000 7fc809cb9700 1 -- 172.21.15.155:0/114251243 shutdown_connections 2024-02-10T09:05:44.098 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:44.095+0000 7fc809cb9700 1 -- 172.21.15.155:0/114251243 wait complete. 2024-02-10T09:05:44.098 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:44.096+0000 7fc809cb9700 1 Processor -- start 2024-02-10T09:05:44.098 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:44.096+0000 7fc809cb9700 1 -- start start 2024-02-10T09:05:44.098 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:44.096+0000 7fc809cb9700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc804060280 0x7fc804106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:05:44.098 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:44.096+0000 7fc809cb9700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc804106c70 con 0x7fc804060280 2024-02-10T09:05:44.098 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:44.096+0000 7fc8037fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc804060280 0x7fc804106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:05:44.098 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:44.096+0000 7fc8037fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc804060280 0x7fc804106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:42466/0 (socket says 172.21.15.155:42466) 2024-02-10T09:05:44.098 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:44.096+0000 7fc8037fe700 1 -- 172.21.15.155:0/1025960617 learned_addr learned my addr 172.21.15.155:0/1025960617 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:05:44.099 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:44.097+0000 7fc8037fe700 1 -- 172.21.15.155:0/1025960617 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7ec005040 con 0x7fc804060280 2024-02-10T09:05:44.099 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:44.097+0000 7fc8037fe700 1 --2- 172.21.15.155:0/1025960617 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc804060280 0x7fc804106780 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7fc7ec0089f0 tx=0x7fc7ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:05:44.099 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:44.097+0000 7fc7fbfff700 1 -- 172.21.15.155:0/1025960617 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc7ec009320 con 0x7fc804060280 2024-02-10T09:05:44.099 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:44.097+0000 7fc809cb9700 1 -- 172.21.15.155:0/1025960617 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc804106e70 con 0x7fc804060280 2024-02-10T09:05:44.099 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:44.097+0000 7fc7fbfff700 1 -- 172.21.15.155:0/1025960617 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc7ec013380 con 0x7fc804060280 2024-02-10T09:05:44.099 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:44.097+0000 7fc7fbfff700 1 -- 172.21.15.155:0/1025960617 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc7ec014e20 con 0x7fc804060280 2024-02-10T09:05:44.099 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:44.097+0000 7fc809cb9700 1 -- 172.21.15.155:0/1025960617 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc804107300 con 0x7fc804060280 2024-02-10T09:05:44.100 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:44.098+0000 7fc7fbfff700 1 -- 172.21.15.155:0/1025960617 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc7ec0263a0 con 0x7fc804060280 2024-02-10T09:05:44.100 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:44.098+0000 7fc7fbfff700 1 --2- 172.21.15.155:0/1025960617 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fc7f002ff10 0x7fc7f00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:05:44.100 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:44.098+0000 7fc7fbfff700 1 -- 172.21.15.155:0/1025960617 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc7ec040f20 con 0x7fc804060280 2024-02-10T09:05:44.100 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:44.098+0000 7fc809cb9700 1 -- 172.21.15.155:0/1025960617 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc804045ad0 con 0x7fc804060280 2024-02-10T09:05:44.101 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:44.098+0000 7fc802ffd700 1 --2- 172.21.15.155:0/1025960617 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fc7f002ff10 0x7fc7f00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:05:44.101 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:44.099+0000 7fc802ffd700 1 --2- 172.21.15.155:0/1025960617 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fc7f002ff10 0x7fc7f00323d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fc7f40096a0 tx=0x7fc7f4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:05:44.103 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:44.103+0000 7fc7fbfff700 1 -- 172.21.15.155:0/1025960617 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc7ec012920 con 0x7fc804060280 2024-02-10T09:05:44.148 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:05:43 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:05:43.020351+0000 mgr.smithi077.dxntvw (mgr.14162) 214 : cluster [DBG] pgmap v202: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:05:44.310 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:44.310+0000 7fc809cb9700 1 -- 172.21.15.155:0/1025960617 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc8040fc090 con 0x7fc804060280 2024-02-10T09:05:44.311 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:44.311+0000 7fc7fbfff700 1 -- 172.21.15.155:0/1025960617 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc7ec02b020 con 0x7fc804060280 2024-02-10T09:05:44.311 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:05:44.311 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:05:44.313 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:44.313+0000 7fc809cb9700 1 -- 172.21.15.155:0/1025960617 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fc7f002ff10 msgr2=0x7fc7f00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:05:44.313 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:44.313+0000 7fc809cb9700 1 --2- 172.21.15.155:0/1025960617 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fc7f002ff10 0x7fc7f00323d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fc7f40096a0 tx=0x7fc7f4006b40).stop 2024-02-10T09:05:44.313 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:44.313+0000 7fc809cb9700 1 -- 172.21.15.155:0/1025960617 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc804060280 msgr2=0x7fc804106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:05:44.313 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:44.313+0000 7fc809cb9700 1 --2- 172.21.15.155:0/1025960617 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc804060280 0x7fc804106780 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7fc7ec0089f0 tx=0x7fc7ec00b040).stop 2024-02-10T09:05:44.313 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:44.313+0000 7fc809cb9700 1 -- 172.21.15.155:0/1025960617 shutdown_connections 2024-02-10T09:05:44.314 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:44.313+0000 7fc809cb9700 1 --2- 172.21.15.155:0/1025960617 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fc7f002ff10 0x7fc7f00323d0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:05:44.314 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:44.313+0000 7fc809cb9700 1 --2- 172.21.15.155:0/1025960617 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc804060280 0x7fc804106780 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:05:44.314 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:44.313+0000 7fc809cb9700 1 -- 172.21.15.155:0/1025960617 >> 172.21.15.155:0/1025960617 conn(0x7fc8040f0b70 msgr2=0x7fc8040f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:05:44.314 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:44.313+0000 7fc809cb9700 1 -- 172.21.15.155:0/1025960617 shutdown_connections 2024-02-10T09:05:44.314 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:44.313+0000 7fc809cb9700 1 -- 172.21.15.155:0/1025960617 wait complete. 2024-02-10T09:05:44.314 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:05:45.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:05:44 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:05:44.311497+0000 mon.smithi077 (mon.0) 233 : audit [DBG] from='client.? 172.21.15.155:0/1025960617' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:05:46.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:05:45 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:05:45.020687+0000 mgr.smithi077.dxntvw (mgr.14162) 215 : cluster [DBG] pgmap v203: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:05:46.226 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:05:46.227 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:05:47.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:05:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:05:47.021055+0000 mgr.smithi077.dxntvw (mgr.14162) 216 : cluster [DBG] pgmap v204: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:05:48.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:05:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:05:47.095062+0000 mon.smithi077 (mon.0) 234 : audit [INF] from='mgr.14162 172.21.15.77:0/2029106299' entity='mgr.smithi077.dxntvw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi077.dxntvw/mirror_snapshot_schedule"}]: dispatch 2024-02-10T09:05:48.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:05:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:05:47.101300+0000 mon.smithi077 (mon.0) 235 : audit [INF] from='mgr.14162 172.21.15.77:0/2029106299' entity='mgr.smithi077.dxntvw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi077.dxntvw/trash_purge_schedule"}]: dispatch 2024-02-10T09:05:48.535 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:48.535+0000 7f044560b700 1 -- 172.21.15.155:0/380655441 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0440060200 msgr2=0x7f04400fb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:05:48.536 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:48.535+0000 7f044560b700 1 --2- 172.21.15.155:0/380655441 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0440060200 0x7f04400fb570 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f0434009fb0 tx=0x7f0434009b30).stop 2024-02-10T09:05:48.536 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:48.535+0000 7f044560b700 1 -- 172.21.15.155:0/380655441 shutdown_connections 2024-02-10T09:05:48.536 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:48.535+0000 7f044560b700 1 --2- 172.21.15.155:0/380655441 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0440060200 0x7f04400fb570 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:05:48.536 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:48.535+0000 7f044560b700 1 -- 172.21.15.155:0/380655441 >> 172.21.15.155:0/380655441 conn(0x7f04400f0b70 msgr2=0x7f04400f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:05:48.536 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:48.535+0000 7f044560b700 1 -- 172.21.15.155:0/380655441 shutdown_connections 2024-02-10T09:05:48.536 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:48.535+0000 7f044560b700 1 -- 172.21.15.155:0/380655441 wait complete. 2024-02-10T09:05:48.538 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:48.536+0000 7f044560b700 1 Processor -- start 2024-02-10T09:05:48.538 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:48.536+0000 7f044560b700 1 -- start start 2024-02-10T09:05:48.538 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:48.536+0000 7f044560b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0440060200 0x7f04401089a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:05:48.538 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:48.536+0000 7f044560b700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0440108e90 con 0x7f0440060200 2024-02-10T09:05:48.538 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:48.537+0000 7f043effd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0440060200 0x7f04401089a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:05:48.538 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:48.537+0000 7f043effd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0440060200 0x7f04401089a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:42486/0 (socket says 172.21.15.155:42486) 2024-02-10T09:05:48.538 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:48.537+0000 7f043effd700 1 -- 172.21.15.155:0/685537509 learned_addr learned my addr 172.21.15.155:0/685537509 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:05:48.538 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:48.537+0000 7f043effd700 1 -- 172.21.15.155:0/685537509 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0434005040 con 0x7f0440060200 2024-02-10T09:05:48.538 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:48.537+0000 7f043effd700 1 --2- 172.21.15.155:0/685537509 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0440060200 0x7f04401089a0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f04340089f0 tx=0x7f043400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:05:48.538 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:48.537+0000 7f04277fe700 1 -- 172.21.15.155:0/685537509 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0434009320 con 0x7f0440060200 2024-02-10T09:05:48.538 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:48.537+0000 7f044560b700 1 -- 172.21.15.155:0/685537509 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0440109090 con 0x7f0440060200 2024-02-10T09:05:48.539 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:48.537+0000 7f04277fe700 1 -- 172.21.15.155:0/685537509 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0434013380 con 0x7f0440060200 2024-02-10T09:05:48.539 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:48.537+0000 7f04277fe700 1 -- 172.21.15.155:0/685537509 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0434014e20 con 0x7f0440060200 2024-02-10T09:05:48.539 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:48.537+0000 7f044560b700 1 -- 172.21.15.155:0/685537509 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0440109520 con 0x7f0440060200 2024-02-10T09:05:48.539 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:48.538+0000 7f04277fe700 1 -- 172.21.15.155:0/685537509 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f04340263a0 con 0x7f0440060200 2024-02-10T09:05:48.540 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:48.538+0000 7f044560b700 1 -- 172.21.15.155:0/685537509 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0440045ad0 con 0x7f0440060200 2024-02-10T09:05:48.540 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:48.538+0000 7f04277fe700 1 --2- 172.21.15.155:0/685537509 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f042802ff60 0x7f0428032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:05:48.540 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:48.538+0000 7f04277fe700 1 -- 172.21.15.155:0/685537509 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0434041140 con 0x7f0440060200 2024-02-10T09:05:48.540 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:48.539+0000 7f043e7fc700 1 --2- 172.21.15.155:0/685537509 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f042802ff60 0x7f0428032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:05:48.540 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:48.539+0000 7f043e7fc700 1 --2- 172.21.15.155:0/685537509 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f042802ff60 0x7f0428032420 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f04300096a0 tx=0x7f0430006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:05:48.543 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:48.543+0000 7f04277fe700 1 -- 172.21.15.155:0/685537509 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0434012330 con 0x7f0440060200 2024-02-10T09:05:48.749 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:48.748+0000 7f044560b700 1 -- 172.21.15.155:0/685537509 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f044010a3d0 con 0x7f0440060200 2024-02-10T09:05:48.749 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:48.749+0000 7f04277fe700 1 -- 172.21.15.155:0/685537509 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f043401ccb0 con 0x7f0440060200 2024-02-10T09:05:48.749 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:05:48.749 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:05:48.751 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:48.751+0000 7f044560b700 1 -- 172.21.15.155:0/685537509 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f042802ff60 msgr2=0x7f0428032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:05:48.751 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:48.751+0000 7f044560b700 1 --2- 172.21.15.155:0/685537509 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f042802ff60 0x7f0428032420 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f04300096a0 tx=0x7f0430006b40).stop 2024-02-10T09:05:48.751 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:48.751+0000 7f044560b700 1 -- 172.21.15.155:0/685537509 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0440060200 msgr2=0x7f04401089a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:05:48.751 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:48.751+0000 7f044560b700 1 --2- 172.21.15.155:0/685537509 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0440060200 0x7f04401089a0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f04340089f0 tx=0x7f043400b040).stop 2024-02-10T09:05:48.751 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:48.751+0000 7f044560b700 1 -- 172.21.15.155:0/685537509 shutdown_connections 2024-02-10T09:05:48.751 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:48.751+0000 7f044560b700 1 --2- 172.21.15.155:0/685537509 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f042802ff60 0x7f0428032420 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:05:48.751 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:48.751+0000 7f044560b700 1 --2- 172.21.15.155:0/685537509 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0440060200 0x7f04401089a0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:05:48.751 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:48.751+0000 7f044560b700 1 -- 172.21.15.155:0/685537509 >> 172.21.15.155:0/685537509 conn(0x7f04400f0b70 msgr2=0x7f04400f15c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:05:48.751 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:48.751+0000 7f044560b700 1 -- 172.21.15.155:0/685537509 shutdown_connections 2024-02-10T09:05:48.751 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:48.751+0000 7f044560b700 1 -- 172.21.15.155:0/685537509 wait complete. 2024-02-10T09:05:48.752 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:05:49.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:05:49 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:05:48.749571+0000 mon.smithi077 (mon.0) 236 : audit [DBG] from='client.? 172.21.15.155:0/685537509' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:05:49.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:05:49 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:05:49.021400+0000 mgr.smithi077.dxntvw (mgr.14162) 217 : cluster [DBG] pgmap v205: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:05:50.545 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:05:50.545 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:05:51.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:05:51 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:05:51.021733+0000 mgr.smithi077.dxntvw (mgr.14162) 218 : cluster [DBG] pgmap v206: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:05:52.607 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:52.576+0000 7f7a2d740700 1 -- 172.21.15.155:0/2796651360 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a280f62a0 msgr2=0x7f7a280f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:05:52.607 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:52.576+0000 7f7a2d740700 1 --2- 172.21.15.155:0/2796651360 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a280f62a0 0x7f7a280f66c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f7a1c009fb0 tx=0x7f7a1c009b30).stop 2024-02-10T09:05:52.607 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:52.576+0000 7f7a2d740700 1 -- 172.21.15.155:0/2796651360 shutdown_connections 2024-02-10T09:05:52.607 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:52.576+0000 7f7a2d740700 1 --2- 172.21.15.155:0/2796651360 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a280f62a0 0x7f7a280f66c0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:05:52.607 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:52.576+0000 7f7a2d740700 1 -- 172.21.15.155:0/2796651360 >> 172.21.15.155:0/2796651360 conn(0x7f7a280f1aa0 msgr2=0x7f7a280f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:05:52.607 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:52.576+0000 7f7a2d740700 1 -- 172.21.15.155:0/2796651360 shutdown_connections 2024-02-10T09:05:52.607 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:52.576+0000 7f7a2d740700 1 -- 172.21.15.155:0/2796651360 wait complete. 2024-02-10T09:05:52.607 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:52.577+0000 7f7a2d740700 1 Processor -- start 2024-02-10T09:05:52.608 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:52.577+0000 7f7a2d740700 1 -- start start 2024-02-10T09:05:52.608 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:52.577+0000 7f7a2d740700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a280f62a0 0x7f7a2806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:05:52.608 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:52.577+0000 7f7a2d740700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a28069e20 con 0x7f7a280f62a0 2024-02-10T09:05:52.608 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:52.577+0000 7f7a27fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a280f62a0 0x7f7a2806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:05:52.608 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:52.577+0000 7f7a27fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a280f62a0 0x7f7a2806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:46750/0 (socket says 172.21.15.155:46750) 2024-02-10T09:05:52.608 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:52.577+0000 7f7a27fff700 1 -- 172.21.15.155:0/2923806189 learned_addr learned my addr 172.21.15.155:0/2923806189 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:05:52.608 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:52.578+0000 7f7a27fff700 1 -- 172.21.15.155:0/2923806189 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a1c005040 con 0x7f7a280f62a0 2024-02-10T09:05:52.608 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:52.578+0000 7f7a27fff700 1 --2- 172.21.15.155:0/2923806189 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a280f62a0 0x7f7a2806b9a0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f7a1c0089f0 tx=0x7f7a1c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:05:52.608 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:52.578+0000 7f7a24ff9700 1 -- 172.21.15.155:0/2923806189 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a1c009320 con 0x7f7a280f62a0 2024-02-10T09:05:52.608 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:52.578+0000 7f7a2d740700 1 -- 172.21.15.155:0/2923806189 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a28069fc0 con 0x7f7a280f62a0 2024-02-10T09:05:52.608 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:52.578+0000 7f7a24ff9700 1 -- 172.21.15.155:0/2923806189 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7a1c013380 con 0x7f7a280f62a0 2024-02-10T09:05:52.608 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:52.578+0000 7f7a24ff9700 1 -- 172.21.15.155:0/2923806189 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a1c014e20 con 0x7f7a280f62a0 2024-02-10T09:05:52.608 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:52.578+0000 7f7a2d740700 1 -- 172.21.15.155:0/2923806189 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a2806a450 con 0x7f7a280f62a0 2024-02-10T09:05:52.608 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:52.579+0000 7f7a24ff9700 1 -- 172.21.15.155:0/2923806189 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7a1c0095f0 con 0x7f7a280f62a0 2024-02-10T09:05:52.609 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:52.579+0000 7f7a24ff9700 1 --2- 172.21.15.155:0/2923806189 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f7a1002ff10 0x7f7a100323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:05:52.609 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:52.579+0000 7f7a2d740700 1 -- 172.21.15.155:0/2923806189 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a28059100 con 0x7f7a280f62a0 2024-02-10T09:05:52.609 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:52.579+0000 7f7a24ff9700 1 -- 172.21.15.155:0/2923806189 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7a1c040ea0 con 0x7f7a280f62a0 2024-02-10T09:05:52.609 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:52.579+0000 7f7a277fe700 1 --2- 172.21.15.155:0/2923806189 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f7a1002ff10 0x7f7a100323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:05:52.609 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:52.580+0000 7f7a277fe700 1 --2- 172.21.15.155:0/2923806189 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f7a1002ff10 0x7f7a100323d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f7a180096a0 tx=0x7f7a18006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:05:52.609 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:52.584+0000 7f7a24ff9700 1 -- 172.21.15.155:0/2923806189 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7a1c014420 con 0x7f7a280f62a0 2024-02-10T09:05:52.795 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:52.794+0000 7f7a2d740700 1 -- 172.21.15.155:0/2923806189 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7a280fbe30 con 0x7f7a280f62a0 2024-02-10T09:05:52.796 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:52.795+0000 7f7a24ff9700 1 -- 172.21.15.155:0/2923806189 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7a1c010230 con 0x7f7a280f62a0 2024-02-10T09:05:52.797 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:05:52.797 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:05:52.798 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:52.797+0000 7f7a2d740700 1 -- 172.21.15.155:0/2923806189 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f7a1002ff10 msgr2=0x7f7a100323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:05:52.798 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:52.797+0000 7f7a2d740700 1 --2- 172.21.15.155:0/2923806189 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f7a1002ff10 0x7f7a100323d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f7a180096a0 tx=0x7f7a18006b40).stop 2024-02-10T09:05:52.798 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:52.797+0000 7f7a2d740700 1 -- 172.21.15.155:0/2923806189 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a280f62a0 msgr2=0x7f7a2806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:05:52.798 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:52.797+0000 7f7a2d740700 1 --2- 172.21.15.155:0/2923806189 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a280f62a0 0x7f7a2806b9a0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f7a1c0089f0 tx=0x7f7a1c00b040).stop 2024-02-10T09:05:52.798 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:52.797+0000 7f7a2d740700 1 -- 172.21.15.155:0/2923806189 shutdown_connections 2024-02-10T09:05:52.799 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:52.797+0000 7f7a2d740700 1 --2- 172.21.15.155:0/2923806189 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f7a1002ff10 0x7f7a100323d0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:05:52.799 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:52.797+0000 7f7a2d740700 1 --2- 172.21.15.155:0/2923806189 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7a280f62a0 0x7f7a2806b9a0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:05:52.799 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:52.797+0000 7f7a2d740700 1 -- 172.21.15.155:0/2923806189 >> 172.21.15.155:0/2923806189 conn(0x7f7a280f1aa0 msgr2=0x7f7a280f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:05:52.799 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:52.797+0000 7f7a2d740700 1 -- 172.21.15.155:0/2923806189 shutdown_connections 2024-02-10T09:05:52.799 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:52.797+0000 7f7a2d740700 1 -- 172.21.15.155:0/2923806189 wait complete. 2024-02-10T09:05:52.799 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:05:54.148 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:05:53 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:05:52.796530+0000 mon.smithi077 (mon.0) 237 : audit [DBG] from='client.? 172.21.15.155:0/2923806189' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:05:54.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:05:53 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:05:53.022080+0000 mgr.smithi077.dxntvw (mgr.14162) 219 : cluster [DBG] pgmap v207: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:05:54.597 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:05:54.598 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:05:55.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:05:55 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:05:55.022441+0000 mgr.smithi077.dxntvw (mgr.14162) 220 : cluster [DBG] pgmap v208: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:05:56.890 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:56.887+0000 7f672759e700 1 -- 172.21.15.155:0/1675609030 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67280f51a0 msgr2=0x7f67280f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:05:56.890 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:56.887+0000 7f672759e700 1 --2- 172.21.15.155:0/1675609030 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67280f51a0 0x7f67280f55c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7f671c009fb0 tx=0x7f671c009b30).stop 2024-02-10T09:05:56.891 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:56.888+0000 7f672759e700 1 -- 172.21.15.155:0/1675609030 shutdown_connections 2024-02-10T09:05:56.891 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:56.888+0000 7f672759e700 1 --2- 172.21.15.155:0/1675609030 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67280f51a0 0x7f67280f55c0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:05:56.891 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:56.888+0000 7f672759e700 1 -- 172.21.15.155:0/1675609030 >> 172.21.15.155:0/1675609030 conn(0x7f67280f09a0 msgr2=0x7f67280f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:05:56.891 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:56.888+0000 7f672759e700 1 -- 172.21.15.155:0/1675609030 shutdown_connections 2024-02-10T09:05:56.891 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:56.888+0000 7f672759e700 1 -- 172.21.15.155:0/1675609030 wait complete. 2024-02-10T09:05:56.891 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:56.888+0000 7f672759e700 1 Processor -- start 2024-02-10T09:05:56.891 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:56.888+0000 7f672759e700 1 -- start start 2024-02-10T09:05:56.891 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:56.888+0000 7f672759e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67280f51a0 0x7f67281065a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:05:56.891 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:56.888+0000 7f672759e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6728106a90 con 0x7f67280f51a0 2024-02-10T09:05:56.891 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:56.888+0000 7f672659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67280f51a0 0x7f67281065a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:05:56.891 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:56.888+0000 7f672659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67280f51a0 0x7f67281065a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:46768/0 (socket says 172.21.15.155:46768) 2024-02-10T09:05:56.891 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:56.888+0000 7f672659c700 1 -- 172.21.15.155:0/3119059301 learned_addr learned my addr 172.21.15.155:0/3119059301 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:05:56.891 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:56.889+0000 7f672659c700 1 -- 172.21.15.155:0/3119059301 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f671c005040 con 0x7f67280f51a0 2024-02-10T09:05:56.892 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:56.889+0000 7f672659c700 1 --2- 172.21.15.155:0/3119059301 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67280f51a0 0x7f67281065a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f671c0089f0 tx=0x7f671c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:05:56.892 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:56.889+0000 7f6716ffd700 1 -- 172.21.15.155:0/3119059301 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f671c009320 con 0x7f67280f51a0 2024-02-10T09:05:56.892 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:56.889+0000 7f672759e700 1 -- 172.21.15.155:0/3119059301 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6728106c90 con 0x7f67280f51a0 2024-02-10T09:05:56.892 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:56.889+0000 7f6716ffd700 1 -- 172.21.15.155:0/3119059301 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f671c013380 con 0x7f67280f51a0 2024-02-10T09:05:56.892 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:56.889+0000 7f6716ffd700 1 -- 172.21.15.155:0/3119059301 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f671c014e20 con 0x7f67280f51a0 2024-02-10T09:05:56.892 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:56.889+0000 7f672759e700 1 -- 172.21.15.155:0/3119059301 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6728107120 con 0x7f67280f51a0 2024-02-10T09:05:56.893 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:56.890+0000 7f6716ffd700 1 -- 172.21.15.155:0/3119059301 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f671c0095f0 con 0x7f67280f51a0 2024-02-10T09:05:56.893 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:56.890+0000 7f672759e700 1 -- 172.21.15.155:0/3119059301 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6728045ad0 con 0x7f67280f51a0 2024-02-10T09:05:56.893 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:56.890+0000 7f6716ffd700 1 --2- 172.21.15.155:0/3119059301 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f671002ff10 0x7f67100323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:05:56.893 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:56.890+0000 7f6716ffd700 1 -- 172.21.15.155:0/3119059301 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f671c040f30 con 0x7f67280f51a0 2024-02-10T09:05:56.893 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:56.891+0000 7f6725d9b700 1 --2- 172.21.15.155:0/3119059301 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f671002ff10 0x7f67100323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:05:56.893 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:56.891+0000 7f6725d9b700 1 --2- 172.21.15.155:0/3119059301 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f671002ff10 0x7f67100323d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f67180096a0 tx=0x7f6718006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:05:56.896 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:56.895+0000 7f6716ffd700 1 -- 172.21.15.155:0/3119059301 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f671c014420 con 0x7f67280f51a0 2024-02-10T09:05:57.101 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:57.100+0000 7f672759e700 1 -- 172.21.15.155:0/3119059301 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6728107fd0 con 0x7f67280f51a0 2024-02-10T09:05:57.101 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:57.100+0000 7f6716ffd700 1 -- 172.21.15.155:0/3119059301 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f671c01d980 con 0x7f67280f51a0 2024-02-10T09:05:57.102 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:05:57.102 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:05:57.103 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:57.102+0000 7f672759e700 1 -- 172.21.15.155:0/3119059301 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f671002ff10 msgr2=0x7f67100323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:05:57.103 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:57.102+0000 7f672759e700 1 --2- 172.21.15.155:0/3119059301 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f671002ff10 0x7f67100323d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f67180096a0 tx=0x7f6718006b40).stop 2024-02-10T09:05:57.103 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:57.102+0000 7f672759e700 1 -- 172.21.15.155:0/3119059301 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67280f51a0 msgr2=0x7f67281065a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:05:57.103 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:57.102+0000 7f672759e700 1 --2- 172.21.15.155:0/3119059301 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67280f51a0 0x7f67281065a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f671c0089f0 tx=0x7f671c00b040).stop 2024-02-10T09:05:57.103 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:57.102+0000 7f672759e700 1 -- 172.21.15.155:0/3119059301 shutdown_connections 2024-02-10T09:05:57.104 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:57.102+0000 7f672759e700 1 --2- 172.21.15.155:0/3119059301 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f671002ff10 0x7f67100323d0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:05:57.104 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:57.102+0000 7f672759e700 1 --2- 172.21.15.155:0/3119059301 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f67280f51a0 0x7f67281065a0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:05:57.104 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:57.102+0000 7f672759e700 1 -- 172.21.15.155:0/3119059301 >> 172.21.15.155:0/3119059301 conn(0x7f67280f09a0 msgr2=0x7f67280f13c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:05:57.104 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:57.102+0000 7f672759e700 1 -- 172.21.15.155:0/3119059301 shutdown_connections 2024-02-10T09:05:57.104 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:05:57.102+0000 7f672759e700 1 -- 172.21.15.155:0/3119059301 wait complete. 2024-02-10T09:05:57.104 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:05:57.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:05:57 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:05:57.022763+0000 mgr.smithi077.dxntvw (mgr.14162) 221 : cluster [DBG] pgmap v209: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:05:58.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:05:58 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:05:57.101900+0000 mon.smithi077 (mon.0) 238 : audit [DBG] from='client.? 172.21.15.155:0/3119059301' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:05:58.957 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:05:58.958 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:05:59.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:05:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:05:59.023091+0000 mgr.smithi077.dxntvw (mgr.14162) 222 : cluster [DBG] pgmap v210: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:06:01.057 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:01.054+0000 7ff651da7700 1 -- 172.21.15.155:0/2340640092 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff64c0f7b60 msgr2=0x7ff64c0f7f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:06:01.057 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:01.054+0000 7ff651da7700 1 --2- 172.21.15.155:0/2340640092 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff64c0f7b60 0x7ff64c0f7f80 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7ff63c009fb0 tx=0x7ff63c009b30).stop 2024-02-10T09:06:01.057 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:01.054+0000 7ff651da7700 1 -- 172.21.15.155:0/2340640092 shutdown_connections 2024-02-10T09:06:01.057 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:01.054+0000 7ff651da7700 1 --2- 172.21.15.155:0/2340640092 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff64c0f7b60 0x7ff64c0f7f80 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:06:01.057 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:01.054+0000 7ff651da7700 1 -- 172.21.15.155:0/2340640092 >> 172.21.15.155:0/2340640092 conn(0x7ff64c0f02e0 msgr2=0x7ff64c0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:06:01.058 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:01.054+0000 7ff651da7700 1 -- 172.21.15.155:0/2340640092 shutdown_connections 2024-02-10T09:06:01.058 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:01.055+0000 7ff651da7700 1 -- 172.21.15.155:0/2340640092 wait complete. 2024-02-10T09:06:01.058 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:01.055+0000 7ff651da7700 1 Processor -- start 2024-02-10T09:06:01.058 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:01.055+0000 7ff651da7700 1 -- start start 2024-02-10T09:06:01.058 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:01.055+0000 7ff651da7700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff64c0f7b60 0x7ff64c106830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:06:01.058 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:01.055+0000 7ff651da7700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff64c106d20 con 0x7ff64c0f7b60 2024-02-10T09:06:01.058 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:01.055+0000 7ff64b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff64c0f7b60 0x7ff64c106830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:06:01.058 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:01.055+0000 7ff64b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff64c0f7b60 0x7ff64c106830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:46788/0 (socket says 172.21.15.155:46788) 2024-02-10T09:06:01.058 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:01.055+0000 7ff64b7fe700 1 -- 172.21.15.155:0/649010355 learned_addr learned my addr 172.21.15.155:0/649010355 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:06:01.058 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:01.056+0000 7ff64b7fe700 1 -- 172.21.15.155:0/649010355 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff63c005040 con 0x7ff64c0f7b60 2024-02-10T09:06:01.058 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:01.056+0000 7ff64b7fe700 1 --2- 172.21.15.155:0/649010355 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff64c0f7b60 0x7ff64c106830 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7ff63c005010 tx=0x7ff63c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:06:01.058 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:01.056+0000 7ff643fff700 1 -- 172.21.15.155:0/649010355 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff63c009320 con 0x7ff64c0f7b60 2024-02-10T09:06:01.059 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:01.056+0000 7ff651da7700 1 -- 172.21.15.155:0/649010355 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff64c106f20 con 0x7ff64c0f7b60 2024-02-10T09:06:01.059 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:01.056+0000 7ff643fff700 1 -- 172.21.15.155:0/649010355 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff63c013380 con 0x7ff64c0f7b60 2024-02-10T09:06:01.059 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:01.056+0000 7ff643fff700 1 -- 172.21.15.155:0/649010355 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff63c014e50 con 0x7ff64c0f7b60 2024-02-10T09:06:01.059 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:01.056+0000 7ff651da7700 1 -- 172.21.15.155:0/649010355 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff64c1073b0 con 0x7ff64c0f7b60 2024-02-10T09:06:01.060 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:01.057+0000 7ff643fff700 1 -- 172.21.15.155:0/649010355 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff63c014420 con 0x7ff64c0f7b60 2024-02-10T09:06:01.060 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:01.057+0000 7ff643fff700 1 --2- 172.21.15.155:0/649010355 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7ff63802ff10 0x7ff6380323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:06:01.060 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:01.057+0000 7ff651da7700 1 -- 172.21.15.155:0/649010355 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff64c045ad0 con 0x7ff64c0f7b60 2024-02-10T09:06:01.060 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:01.057+0000 7ff643fff700 1 -- 172.21.15.155:0/649010355 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff63c040e60 con 0x7ff64c0f7b60 2024-02-10T09:06:01.060 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:01.057+0000 7ff64affd700 1 --2- 172.21.15.155:0/649010355 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7ff63802ff10 0x7ff6380323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:06:01.060 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:01.058+0000 7ff64affd700 1 --2- 172.21.15.155:0/649010355 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7ff63802ff10 0x7ff6380323d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7ff6340096a0 tx=0x7ff634006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:06:01.063 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:01.062+0000 7ff643fff700 1 -- 172.21.15.155:0/649010355 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff63c0095f0 con 0x7ff64c0f7b60 2024-02-10T09:06:01.273 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:01.272+0000 7ff651da7700 1 -- 172.21.15.155:0/649010355 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff64c108200 con 0x7ff64c0f7b60 2024-02-10T09:06:01.273 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:01.272+0000 7ff643fff700 1 -- 172.21.15.155:0/649010355 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff63c042070 con 0x7ff64c0f7b60 2024-02-10T09:06:01.274 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:06:01.274 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:06:01.276 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:01.274+0000 7ff651da7700 1 -- 172.21.15.155:0/649010355 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7ff63802ff10 msgr2=0x7ff6380323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:06:01.276 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:01.274+0000 7ff651da7700 1 --2- 172.21.15.155:0/649010355 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7ff63802ff10 0x7ff6380323d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7ff6340096a0 tx=0x7ff634006b40).stop 2024-02-10T09:06:01.276 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:01.274+0000 7ff651da7700 1 -- 172.21.15.155:0/649010355 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff64c0f7b60 msgr2=0x7ff64c106830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:06:01.276 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:01.274+0000 7ff651da7700 1 --2- 172.21.15.155:0/649010355 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff64c0f7b60 0x7ff64c106830 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7ff63c005010 tx=0x7ff63c00b040).stop 2024-02-10T09:06:01.276 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:01.274+0000 7ff651da7700 1 -- 172.21.15.155:0/649010355 shutdown_connections 2024-02-10T09:06:01.276 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:01.274+0000 7ff651da7700 1 --2- 172.21.15.155:0/649010355 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7ff63802ff10 0x7ff6380323d0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:06:01.276 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:01.274+0000 7ff651da7700 1 --2- 172.21.15.155:0/649010355 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff64c0f7b60 0x7ff64c106830 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:06:01.276 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:01.274+0000 7ff651da7700 1 -- 172.21.15.155:0/649010355 >> 172.21.15.155:0/649010355 conn(0x7ff64c0f02e0 msgr2=0x7ff64c0f0dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:06:01.276 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:01.274+0000 7ff651da7700 1 -- 172.21.15.155:0/649010355 shutdown_connections 2024-02-10T09:06:01.276 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:01.274+0000 7ff651da7700 1 -- 172.21.15.155:0/649010355 wait complete. 2024-02-10T09:06:01.277 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:06:01.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:01 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:06:01.023452+0000 mgr.smithi077.dxntvw (mgr.14162) 223 : cluster [DBG] pgmap v211: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:06:02.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:02 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:06:01.273894+0000 mon.smithi077 (mon.0) 239 : audit [DBG] from='client.? 172.21.15.155:0/649010355' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:06:03.175 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:06:03.176 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:06:04.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:03 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:06:03.023809+0000 mgr.smithi077.dxntvw (mgr.14162) 224 : cluster [DBG] pgmap v212: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:06:05.384 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:05.331+0000 7f3374a28700 1 -- 172.21.15.155:0/960010049 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33700f62a0 msgr2=0x7f33700f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:06:05.384 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:05.331+0000 7f3374a28700 1 --2- 172.21.15.155:0/960010049 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33700f62a0 0x7f33700f66c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f3360009fb0 tx=0x7f3360009b30).stop 2024-02-10T09:06:05.384 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:05.331+0000 7f3374a28700 1 -- 172.21.15.155:0/960010049 shutdown_connections 2024-02-10T09:06:05.384 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:05.332+0000 7f3374a28700 1 --2- 172.21.15.155:0/960010049 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33700f62a0 0x7f33700f66c0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:06:05.384 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:05.332+0000 7f3374a28700 1 -- 172.21.15.155:0/960010049 >> 172.21.15.155:0/960010049 conn(0x7f33700f1aa0 msgr2=0x7f33700f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:06:05.384 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:05.332+0000 7f3374a28700 1 -- 172.21.15.155:0/960010049 shutdown_connections 2024-02-10T09:06:05.384 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:05.332+0000 7f3374a28700 1 -- 172.21.15.155:0/960010049 wait complete. 2024-02-10T09:06:05.385 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:05.332+0000 7f3374a28700 1 Processor -- start 2024-02-10T09:06:05.385 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:05.332+0000 7f3374a28700 1 -- start start 2024-02-10T09:06:05.385 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:05.332+0000 7f3374a28700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33700f62a0 0x7f33701065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:06:05.385 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:05.332+0000 7f3374a28700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3370106ac0 con 0x7f33700f62a0 2024-02-10T09:06:05.385 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:05.333+0000 7f336ed9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33700f62a0 0x7f33701065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:06:05.385 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:05.333+0000 7f336ed9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33700f62a0 0x7f33701065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:34058/0 (socket says 172.21.15.155:34058) 2024-02-10T09:06:05.385 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:05.333+0000 7f336ed9d700 1 -- 172.21.15.155:0/1828545194 learned_addr learned my addr 172.21.15.155:0/1828545194 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:06:05.385 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:05.333+0000 7f336ed9d700 1 -- 172.21.15.155:0/1828545194 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3360005040 con 0x7f33700f62a0 2024-02-10T09:06:05.385 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:05.333+0000 7f336ed9d700 1 --2- 172.21.15.155:0/1828545194 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33700f62a0 0x7f33701065d0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f33600089f0 tx=0x7f336000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:06:05.385 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:05.333+0000 7f33577fe700 1 -- 172.21.15.155:0/1828545194 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3360009320 con 0x7f33700f62a0 2024-02-10T09:06:05.385 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:05.333+0000 7f3374a28700 1 -- 172.21.15.155:0/1828545194 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3370106cc0 con 0x7f33700f62a0 2024-02-10T09:06:05.385 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:05.333+0000 7f33577fe700 1 -- 172.21.15.155:0/1828545194 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3360013380 con 0x7f33700f62a0 2024-02-10T09:06:05.385 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:05.333+0000 7f33577fe700 1 -- 172.21.15.155:0/1828545194 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3360014e20 con 0x7f33700f62a0 2024-02-10T09:06:05.385 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:05.333+0000 7f3374a28700 1 -- 172.21.15.155:0/1828545194 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3370107150 con 0x7f33700f62a0 2024-02-10T09:06:05.385 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:05.334+0000 7f3374a28700 1 -- 172.21.15.155:0/1828545194 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3370045ad0 con 0x7f33700f62a0 2024-02-10T09:06:05.386 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:05.334+0000 7f33577fe700 1 -- 172.21.15.155:0/1828545194 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f33600095f0 con 0x7f33700f62a0 2024-02-10T09:06:05.386 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:05.334+0000 7f33577fe700 1 --2- 172.21.15.155:0/1828545194 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f335802ff10 0x7f33580323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:06:05.386 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:05.334+0000 7f33577fe700 1 -- 172.21.15.155:0/1828545194 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3360040ec0 con 0x7f33700f62a0 2024-02-10T09:06:05.386 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:05.334+0000 7f336e59c700 1 --2- 172.21.15.155:0/1828545194 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f335802ff10 0x7f33580323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:06:05.386 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:05.335+0000 7f336e59c700 1 --2- 172.21.15.155:0/1828545194 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f335802ff10 0x7f33580323d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f33640096a0 tx=0x7f3364006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:06:05.386 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:05.339+0000 7f33577fe700 1 -- 172.21.15.155:0/1828545194 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3360014420 con 0x7f33700f62a0 2024-02-10T09:06:05.398 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:05 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:06:05.024156+0000 mgr.smithi077.dxntvw (mgr.14162) 225 : cluster [DBG] pgmap v213: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:06:05.546 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:05.545+0000 7f3374a28700 1 -- 172.21.15.155:0/1828545194 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3370108000 con 0x7f33700f62a0 2024-02-10T09:06:05.547 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:05.546+0000 7f33577fe700 1 -- 172.21.15.155:0/1828545194 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3360010230 con 0x7f33700f62a0 2024-02-10T09:06:05.547 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:06:05.547 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:06:05.548 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:05.547+0000 7f3374a28700 1 -- 172.21.15.155:0/1828545194 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f335802ff10 msgr2=0x7f33580323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:06:05.548 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:05.547+0000 7f3374a28700 1 --2- 172.21.15.155:0/1828545194 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f335802ff10 0x7f33580323d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f33640096a0 tx=0x7f3364006b40).stop 2024-02-10T09:06:05.548 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:05.547+0000 7f3374a28700 1 -- 172.21.15.155:0/1828545194 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33700f62a0 msgr2=0x7f33701065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:06:05.548 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:05.547+0000 7f3374a28700 1 --2- 172.21.15.155:0/1828545194 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33700f62a0 0x7f33701065d0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f33600089f0 tx=0x7f336000b040).stop 2024-02-10T09:06:05.549 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:05.547+0000 7f3374a28700 1 -- 172.21.15.155:0/1828545194 shutdown_connections 2024-02-10T09:06:05.549 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:05.547+0000 7f3374a28700 1 --2- 172.21.15.155:0/1828545194 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f335802ff10 0x7f33580323d0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:06:05.549 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:05.547+0000 7f3374a28700 1 --2- 172.21.15.155:0/1828545194 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33700f62a0 0x7f33701065d0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:06:05.549 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:05.547+0000 7f3374a28700 1 -- 172.21.15.155:0/1828545194 >> 172.21.15.155:0/1828545194 conn(0x7f33700f1aa0 msgr2=0x7f33700f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:06:05.549 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:05.547+0000 7f3374a28700 1 -- 172.21.15.155:0/1828545194 shutdown_connections 2024-02-10T09:06:05.549 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:05.547+0000 7f3374a28700 1 -- 172.21.15.155:0/1828545194 wait complete. 2024-02-10T09:06:05.549 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:06:06.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:06 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:06:05.547099+0000 mon.smithi077 (mon.0) 240 : audit [DBG] from='client.? 172.21.15.155:0/1828545194' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:06:07.349 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:06:07.350 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:06:07.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:07 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:06:07.024534+0000 mgr.smithi077.dxntvw (mgr.14162) 226 : cluster [DBG] pgmap v214: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:06:09.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:09 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:06:09.024824+0000 mgr.smithi077.dxntvw (mgr.14162) 227 : cluster [DBG] pgmap v215: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:06:09.561 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:09.559+0000 7f0ec759e700 1 -- 172.21.15.155:0/2726308360 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ec80f51a0 msgr2=0x7f0ec80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:06:09.561 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:09.559+0000 7f0ec759e700 1 --2- 172.21.15.155:0/2726308360 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ec80f51a0 0x7f0ec80f55c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f0eb8009fb0 tx=0x7f0eb8009b30).stop 2024-02-10T09:06:09.562 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:09.560+0000 7f0ec759e700 1 -- 172.21.15.155:0/2726308360 shutdown_connections 2024-02-10T09:06:09.562 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:09.560+0000 7f0ec759e700 1 --2- 172.21.15.155:0/2726308360 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ec80f51a0 0x7f0ec80f55c0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:06:09.562 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:09.560+0000 7f0ec759e700 1 -- 172.21.15.155:0/2726308360 >> 172.21.15.155:0/2726308360 conn(0x7f0ec80f09a0 msgr2=0x7f0ec80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:06:09.562 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:09.560+0000 7f0ec759e700 1 -- 172.21.15.155:0/2726308360 shutdown_connections 2024-02-10T09:06:09.562 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:09.560+0000 7f0ec759e700 1 -- 172.21.15.155:0/2726308360 wait complete. 2024-02-10T09:06:09.562 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:09.560+0000 7f0ec759e700 1 Processor -- start 2024-02-10T09:06:09.562 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:09.560+0000 7f0ec759e700 1 -- start start 2024-02-10T09:06:09.562 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:09.560+0000 7f0ec759e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ec80f51a0 0x7f0ec80fd270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:06:09.562 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:09.560+0000 7f0ec759e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec80fb6f0 con 0x7f0ec80f51a0 2024-02-10T09:06:09.562 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:09.560+0000 7f0ec659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ec80f51a0 0x7f0ec80fd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:06:09.562 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:09.560+0000 7f0ec659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ec80f51a0 0x7f0ec80fd270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:34072/0 (socket says 172.21.15.155:34072) 2024-02-10T09:06:09.562 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:09.560+0000 7f0ec659c700 1 -- 172.21.15.155:0/487866863 learned_addr learned my addr 172.21.15.155:0/487866863 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:06:09.564 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:09.561+0000 7f0ec659c700 1 -- 172.21.15.155:0/487866863 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eb8005040 con 0x7f0ec80f51a0 2024-02-10T09:06:09.564 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:09.561+0000 7f0ec659c700 1 --2- 172.21.15.155:0/487866863 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ec80f51a0 0x7f0ec80fd270 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f0eb8009f80 tx=0x7f0eb800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:06:09.564 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:09.561+0000 7f0eb6ffd700 1 -- 172.21.15.155:0/487866863 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0eb80095f0 con 0x7f0ec80f51a0 2024-02-10T09:06:09.564 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:09.561+0000 7f0ec759e700 1 -- 172.21.15.155:0/487866863 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ec80fb890 con 0x7f0ec80f51a0 2024-02-10T09:06:09.564 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:09.561+0000 7f0eb6ffd700 1 -- 172.21.15.155:0/487866863 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0eb8007e90 con 0x7f0ec80f51a0 2024-02-10T09:06:09.564 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:09.561+0000 7f0eb6ffd700 1 -- 172.21.15.155:0/487866863 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0eb8014d80 con 0x7f0ec80f51a0 2024-02-10T09:06:09.564 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:09.561+0000 7f0ec759e700 1 -- 172.21.15.155:0/487866863 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ec80fbd20 con 0x7f0ec80f51a0 2024-02-10T09:06:09.564 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:09.562+0000 7f0eb6ffd700 1 -- 172.21.15.155:0/487866863 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0eb8025070 con 0x7f0ec80f51a0 2024-02-10T09:06:09.564 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:09.562+0000 7f0ec759e700 1 -- 172.21.15.155:0/487866863 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ec8059100 con 0x7f0ec80f51a0 2024-02-10T09:06:09.564 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:09.562+0000 7f0eb6ffd700 1 --2- 172.21.15.155:0/487866863 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f0eb002ff60 0x7f0eb0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:06:09.565 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:09.562+0000 7f0eb6ffd700 1 -- 172.21.15.155:0/487866863 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0eb8040830 con 0x7f0ec80f51a0 2024-02-10T09:06:09.565 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:09.562+0000 7f0ec5d9b700 1 --2- 172.21.15.155:0/487866863 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f0eb002ff60 0x7f0eb0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:06:09.565 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:09.563+0000 7f0ec5d9b700 1 --2- 172.21.15.155:0/487866863 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f0eb002ff60 0x7f0eb0032420 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f0ebc0096a0 tx=0x7f0ebc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:06:09.568 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:09.567+0000 7f0eb6ffd700 1 -- 172.21.15.155:0/487866863 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0eb800f4e0 con 0x7f0ec80f51a0 2024-02-10T09:06:09.774 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:09.773+0000 7f0ec759e700 1 -- 172.21.15.155:0/487866863 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0ec8045ad0 con 0x7f0ec80f51a0 2024-02-10T09:06:09.775 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:09.774+0000 7f0eb6ffd700 1 -- 172.21.15.155:0/487866863 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0eb800f300 con 0x7f0ec80f51a0 2024-02-10T09:06:09.775 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:06:09.775 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:06:09.777 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:09.775+0000 7f0ec759e700 1 -- 172.21.15.155:0/487866863 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f0eb002ff60 msgr2=0x7f0eb0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:06:09.777 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:09.775+0000 7f0ec759e700 1 --2- 172.21.15.155:0/487866863 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f0eb002ff60 0x7f0eb0032420 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f0ebc0096a0 tx=0x7f0ebc006b40).stop 2024-02-10T09:06:09.777 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:09.775+0000 7f0ec759e700 1 -- 172.21.15.155:0/487866863 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ec80f51a0 msgr2=0x7f0ec80fd270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:06:09.777 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:09.775+0000 7f0ec759e700 1 --2- 172.21.15.155:0/487866863 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ec80f51a0 0x7f0ec80fd270 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f0eb8009f80 tx=0x7f0eb800b040).stop 2024-02-10T09:06:09.777 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:09.775+0000 7f0ec759e700 1 -- 172.21.15.155:0/487866863 shutdown_connections 2024-02-10T09:06:09.777 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:09.775+0000 7f0ec759e700 1 --2- 172.21.15.155:0/487866863 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f0eb002ff60 0x7f0eb0032420 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:06:09.777 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:09.776+0000 7f0ec759e700 1 --2- 172.21.15.155:0/487866863 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0ec80f51a0 0x7f0ec80fd270 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:06:09.777 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:09.776+0000 7f0ec759e700 1 -- 172.21.15.155:0/487866863 >> 172.21.15.155:0/487866863 conn(0x7f0ec80f09a0 msgr2=0x7f0ec805fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:06:09.777 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:09.776+0000 7f0ec759e700 1 -- 172.21.15.155:0/487866863 shutdown_connections 2024-02-10T09:06:09.777 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:09.776+0000 7f0ec759e700 1 -- 172.21.15.155:0/487866863 wait complete. 2024-02-10T09:06:09.778 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:06:10.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:10 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:06:09.775562+0000 mon.smithi077 (mon.0) 241 : audit [DBG] from='client.? 172.21.15.155:0/487866863' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:06:11.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:11 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:06:11.025152+0000 mgr.smithi077.dxntvw (mgr.14162) 228 : cluster [DBG] pgmap v216: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:06:11.531 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:06:11.531 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:06:13.706 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:13.702+0000 7fd5760b2700 1 -- 172.21.15.155:0/1130233258 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5700f78e0 msgr2=0x7fd5700f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:06:13.706 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:13.702+0000 7fd5760b2700 1 --2- 172.21.15.155:0/1130233258 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5700f78e0 0x7fd5700f7d00 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7fd560009fb0 tx=0x7fd560009b30).stop 2024-02-10T09:06:13.706 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:13.702+0000 7fd5760b2700 1 -- 172.21.15.155:0/1130233258 shutdown_connections 2024-02-10T09:06:13.706 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:13.702+0000 7fd5760b2700 1 --2- 172.21.15.155:0/1130233258 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5700f78e0 0x7fd5700f7d00 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:06:13.706 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:13.702+0000 7fd5760b2700 1 -- 172.21.15.155:0/1130233258 >> 172.21.15.155:0/1130233258 conn(0x7fd5700f0120 msgr2=0x7fd5700f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:06:13.706 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:13.702+0000 7fd5760b2700 1 -- 172.21.15.155:0/1130233258 shutdown_connections 2024-02-10T09:06:13.706 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:13.703+0000 7fd5760b2700 1 -- 172.21.15.155:0/1130233258 wait complete. 2024-02-10T09:06:13.706 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:13.703+0000 7fd5760b2700 1 Processor -- start 2024-02-10T09:06:13.707 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:13.703+0000 7fd5760b2700 1 -- start start 2024-02-10T09:06:13.707 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:13.703+0000 7fd5760b2700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5700f78e0 0x7fd5701065e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:06:13.707 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:13.703+0000 7fd5760b2700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd570106ad0 con 0x7fd5700f78e0 2024-02-10T09:06:13.707 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:13.704+0000 7fd5750b0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5700f78e0 0x7fd5701065e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:06:13.707 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:13.704+0000 7fd5750b0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5700f78e0 0x7fd5701065e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:59180/0 (socket says 172.21.15.155:59180) 2024-02-10T09:06:13.707 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:13.704+0000 7fd5750b0700 1 -- 172.21.15.155:0/1096459832 learned_addr learned my addr 172.21.15.155:0/1096459832 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:06:13.707 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:13.704+0000 7fd5750b0700 1 -- 172.21.15.155:0/1096459832 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd560005040 con 0x7fd5700f78e0 2024-02-10T09:06:13.707 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:13.704+0000 7fd5750b0700 1 --2- 172.21.15.155:0/1096459832 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5700f78e0 0x7fd5701065e0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7fd5600089f0 tx=0x7fd56000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:06:13.707 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:13.704+0000 7fd565ffb700 1 -- 172.21.15.155:0/1096459832 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd560009320 con 0x7fd5700f78e0 2024-02-10T09:06:13.707 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:13.704+0000 7fd5760b2700 1 -- 172.21.15.155:0/1096459832 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd570106cd0 con 0x7fd5700f78e0 2024-02-10T09:06:13.707 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:13.704+0000 7fd565ffb700 1 -- 172.21.15.155:0/1096459832 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd560013380 con 0x7fd5700f78e0 2024-02-10T09:06:13.707 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:13.704+0000 7fd565ffb700 1 -- 172.21.15.155:0/1096459832 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd560014e20 con 0x7fd5700f78e0 2024-02-10T09:06:13.708 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:13.705+0000 7fd5760b2700 1 -- 172.21.15.155:0/1096459832 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd570107160 con 0x7fd5700f78e0 2024-02-10T09:06:13.708 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:13.705+0000 7fd565ffb700 1 -- 172.21.15.155:0/1096459832 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd5600095f0 con 0x7fd5700f78e0 2024-02-10T09:06:13.709 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:13.705+0000 7fd5760b2700 1 -- 172.21.15.155:0/1096459832 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd570045ad0 con 0x7fd5700f78e0 2024-02-10T09:06:13.709 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:13.705+0000 7fd565ffb700 1 --2- 172.21.15.155:0/1096459832 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fd55c02ff10 0x7fd55c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:06:13.709 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:13.705+0000 7fd565ffb700 1 -- 172.21.15.155:0/1096459832 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd560040e90 con 0x7fd5700f78e0 2024-02-10T09:06:13.709 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:13.706+0000 7fd5748af700 1 --2- 172.21.15.155:0/1096459832 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fd55c02ff10 0x7fd55c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:06:13.709 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:13.706+0000 7fd5748af700 1 --2- 172.21.15.155:0/1096459832 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fd55c02ff10 0x7fd55c0323d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fd56c0096a0 tx=0x7fd56c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:06:13.711 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:13.710+0000 7fd565ffb700 1 -- 172.21.15.155:0/1096459832 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd560014420 con 0x7fd5700f78e0 2024-02-10T09:06:13.920 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:13.919+0000 7fd5760b2700 1 -- 172.21.15.155:0/1096459832 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd570108010 con 0x7fd5700f78e0 2024-02-10T09:06:13.920 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:13.919+0000 7fd565ffb700 1 -- 172.21.15.155:0/1096459832 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd56001d980 con 0x7fd5700f78e0 2024-02-10T09:06:13.921 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:06:13.921 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:06:13.923 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:13.921+0000 7fd5760b2700 1 -- 172.21.15.155:0/1096459832 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fd55c02ff10 msgr2=0x7fd55c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:06:13.923 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:13.921+0000 7fd5760b2700 1 --2- 172.21.15.155:0/1096459832 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fd55c02ff10 0x7fd55c0323d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fd56c0096a0 tx=0x7fd56c006b40).stop 2024-02-10T09:06:13.923 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:13.921+0000 7fd5760b2700 1 -- 172.21.15.155:0/1096459832 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5700f78e0 msgr2=0x7fd5701065e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:06:13.923 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:13.921+0000 7fd5760b2700 1 --2- 172.21.15.155:0/1096459832 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5700f78e0 0x7fd5701065e0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7fd5600089f0 tx=0x7fd56000b040).stop 2024-02-10T09:06:13.923 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:13.921+0000 7fd5760b2700 1 -- 172.21.15.155:0/1096459832 shutdown_connections 2024-02-10T09:06:13.923 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:13.921+0000 7fd5760b2700 1 --2- 172.21.15.155:0/1096459832 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fd55c02ff10 0x7fd55c0323d0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:06:13.923 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:13.921+0000 7fd5760b2700 1 --2- 172.21.15.155:0/1096459832 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd5700f78e0 0x7fd5701065e0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:06:13.923 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:13.921+0000 7fd5760b2700 1 -- 172.21.15.155:0/1096459832 >> 172.21.15.155:0/1096459832 conn(0x7fd5700f0120 msgr2=0x7fd5700f0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:06:13.924 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:13.921+0000 7fd5760b2700 1 -- 172.21.15.155:0/1096459832 shutdown_connections 2024-02-10T09:06:13.924 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:13.921+0000 7fd5760b2700 1 -- 172.21.15.155:0/1096459832 wait complete. 2024-02-10T09:06:13.924 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:06:14.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:13 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:06:13.025501+0000 mgr.smithi077.dxntvw (mgr.14162) 229 : cluster [DBG] pgmap v217: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:06:15.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:14 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:06:13.920836+0000 mon.smithi077 (mon.0) 242 : audit [DBG] from='client.? 172.21.15.155:0/1096459832' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:06:15.780 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:06:15.780 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:06:16.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:15 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:06:15.025873+0000 mgr.smithi077.dxntvw (mgr.14162) 230 : cluster [DBG] pgmap v218: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:06:17.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:17 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:06:17.026167+0000 mgr.smithi077.dxntvw (mgr.14162) 231 : cluster [DBG] pgmap v219: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:06:17.912 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:17.910+0000 7f42e9db3700 1 -- 172.21.15.155:0/743164216 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42e40f5410 msgr2=0x7f42e40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:06:17.912 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:17.910+0000 7f42e9db3700 1 --2- 172.21.15.155:0/743164216 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42e40f5410 0x7f42e40f5830 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f42d8009fb0 tx=0x7f42d8009b30).stop 2024-02-10T09:06:17.912 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:17.910+0000 7f42e9db3700 1 -- 172.21.15.155:0/743164216 shutdown_connections 2024-02-10T09:06:17.912 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:17.910+0000 7f42e9db3700 1 --2- 172.21.15.155:0/743164216 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42e40f5410 0x7f42e40f5830 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:06:17.912 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:17.910+0000 7f42e9db3700 1 -- 172.21.15.155:0/743164216 >> 172.21.15.155:0/743164216 conn(0x7f42e40f0b70 msgr2=0x7f42e40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:06:17.912 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:17.910+0000 7f42e9db3700 1 -- 172.21.15.155:0/743164216 shutdown_connections 2024-02-10T09:06:17.912 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:17.910+0000 7f42e9db3700 1 -- 172.21.15.155:0/743164216 wait complete. 2024-02-10T09:06:17.912 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:17.911+0000 7f42e9db3700 1 Processor -- start 2024-02-10T09:06:17.912 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:17.911+0000 7f42e9db3700 1 -- start start 2024-02-10T09:06:17.914 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:17.911+0000 7f42e9db3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42e40f5410 0x7f42e41056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:06:17.914 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:17.911+0000 7f42e9db3700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42e4105bd0 con 0x7f42e40f5410 2024-02-10T09:06:17.914 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:17.911+0000 7f42e37fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42e40f5410 0x7f42e41056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:06:17.915 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:17.911+0000 7f42e37fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42e40f5410 0x7f42e41056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:59194/0 (socket says 172.21.15.155:59194) 2024-02-10T09:06:17.915 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:17.911+0000 7f42e37fe700 1 -- 172.21.15.155:0/110874755 learned_addr learned my addr 172.21.15.155:0/110874755 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:06:17.915 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:17.912+0000 7f42e37fe700 1 -- 172.21.15.155:0/110874755 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42d8005040 con 0x7f42e40f5410 2024-02-10T09:06:17.915 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:17.912+0000 7f42e37fe700 1 --2- 172.21.15.155:0/110874755 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42e40f5410 0x7f42e41056e0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f42d8005010 tx=0x7f42d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:06:17.915 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:17.912+0000 7f42cbfff700 1 -- 172.21.15.155:0/110874755 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f42d8009320 con 0x7f42e40f5410 2024-02-10T09:06:17.915 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:17.912+0000 7f42e9db3700 1 -- 172.21.15.155:0/110874755 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42e4105dd0 con 0x7f42e40f5410 2024-02-10T09:06:17.915 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:17.912+0000 7f42cbfff700 1 -- 172.21.15.155:0/110874755 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f42d8013380 con 0x7f42e40f5410 2024-02-10T09:06:17.915 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:17.912+0000 7f42cbfff700 1 -- 172.21.15.155:0/110874755 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f42d8014e50 con 0x7f42e40f5410 2024-02-10T09:06:17.915 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:17.912+0000 7f42e9db3700 1 -- 172.21.15.155:0/110874755 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42e4106260 con 0x7f42e40f5410 2024-02-10T09:06:17.915 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:17.913+0000 7f42cbfff700 1 -- 172.21.15.155:0/110874755 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f42d80095f0 con 0x7f42e40f5410 2024-02-10T09:06:17.915 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:17.913+0000 7f42e9db3700 1 -- 172.21.15.155:0/110874755 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42e4045ad0 con 0x7f42e40f5410 2024-02-10T09:06:17.915 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:17.913+0000 7f42cbfff700 1 --2- 172.21.15.155:0/110874755 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f42cc02ff10 0x7f42cc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:06:17.915 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:17.913+0000 7f42cbfff700 1 -- 172.21.15.155:0/110874755 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f42d8040fa0 con 0x7f42e40f5410 2024-02-10T09:06:17.916 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:17.913+0000 7f42e2ffd700 1 --2- 172.21.15.155:0/110874755 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f42cc02ff10 0x7f42cc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:06:17.916 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:17.914+0000 7f42e2ffd700 1 --2- 172.21.15.155:0/110874755 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f42cc02ff10 0x7f42cc0323d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f42d40096a0 tx=0x7f42d4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:06:17.919 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:17.918+0000 7f42cbfff700 1 -- 172.21.15.155:0/110874755 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f42d8014420 con 0x7f42e40f5410 2024-02-10T09:06:18.129 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:18.127+0000 7f42e9db3700 1 -- 172.21.15.155:0/110874755 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f42e4107110 con 0x7f42e40f5410 2024-02-10T09:06:18.129 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:18.128+0000 7f42cbfff700 1 -- 172.21.15.155:0/110874755 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f42d8010230 con 0x7f42e40f5410 2024-02-10T09:06:18.129 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:06:18.130 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:06:18.131 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:18.130+0000 7f42e9db3700 1 -- 172.21.15.155:0/110874755 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f42cc02ff10 msgr2=0x7f42cc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:06:18.131 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:18.130+0000 7f42e9db3700 1 --2- 172.21.15.155:0/110874755 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f42cc02ff10 0x7f42cc0323d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f42d40096a0 tx=0x7f42d4006b40).stop 2024-02-10T09:06:18.131 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:18.130+0000 7f42e9db3700 1 -- 172.21.15.155:0/110874755 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42e40f5410 msgr2=0x7f42e41056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:06:18.131 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:18.130+0000 7f42e9db3700 1 --2- 172.21.15.155:0/110874755 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42e40f5410 0x7f42e41056e0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f42d8005010 tx=0x7f42d800b040).stop 2024-02-10T09:06:18.131 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:18.130+0000 7f42e9db3700 1 -- 172.21.15.155:0/110874755 shutdown_connections 2024-02-10T09:06:18.131 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:18.130+0000 7f42e9db3700 1 --2- 172.21.15.155:0/110874755 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f42cc02ff10 0x7f42cc0323d0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:06:18.131 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:18.130+0000 7f42e9db3700 1 --2- 172.21.15.155:0/110874755 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f42e40f5410 0x7f42e41056e0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:06:18.131 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:18.130+0000 7f42e9db3700 1 -- 172.21.15.155:0/110874755 >> 172.21.15.155:0/110874755 conn(0x7f42e40f0b70 msgr2=0x7f42e40f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:06:18.131 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:18.130+0000 7f42e9db3700 1 -- 172.21.15.155:0/110874755 shutdown_connections 2024-02-10T09:06:18.132 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:18.130+0000 7f42e9db3700 1 -- 172.21.15.155:0/110874755 wait complete. 2024-02-10T09:06:18.132 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:06:19.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:18 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:06:18.129645+0000 mon.smithi077 (mon.0) 243 : audit [DBG] from='client.? 172.21.15.155:0/110874755' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:06:19.975 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:06:19.976 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:06:20.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:19 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:06:19.026492+0000 mgr.smithi077.dxntvw (mgr.14162) 232 : cluster [DBG] pgmap v220: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:06:21.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:21 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cephadm 2024-02-10T09:06:20.189325+0000 mgr.smithi077.dxntvw (mgr.14162) 233 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi077 ... 2024-02-10T09:06:21.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:21 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: Verifying port 9100 ... 2024-02-10T09:06:21.400 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:21 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: Non-zero exit code 1 from systemctl start ceph-6081b54a-c7f2-11ee-95b8-87774f69a715@node-exporter.smithi077 2024-02-10T09:06:21.400 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:21 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: systemctl: stderr Job for ceph-6081b54a-c7f2-11ee-95b8-87774f69a715@node-exporter.smithi077.service failed because a timeout was exceeded. 2024-02-10T09:06:21.400 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:21 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: systemctl: stderr See "systemctl status ceph-6081b54a-c7f2-11ee-95b8-87774f69a715@node-exporter.smithi077.service" and "journalctl -xe" for details. 2024-02-10T09:06:21.400 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:21 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: Traceback (most recent call last): 2024-02-10T09:06:21.400 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:21 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: File "/var/lib/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-10T09:06:21.400 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:21 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: main() 2024-02-10T09:06:21.400 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:21 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: File "/var/lib/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-10T09:06:21.400 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:21 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: r = ctx.func(ctx) 2024-02-10T09:06:21.400 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:21 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: File "/var/lib/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-10T09:06:21.400 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:21 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: return func(ctx) 2024-02-10T09:06:21.400 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:21 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: File "/var/lib/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-02-10T09:06:21.400 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:21 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: ports=daemon_ports) 2024-02-10T09:06:21.400 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:21 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: File "/var/lib/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2024-02-10T09:06:21.401 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:21 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: c, osd_fsid=osd_fsid, ports=ports) 2024-02-10T09:06:21.401 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:21 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: File "/var/lib/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2024-02-10T09:06:21.401 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:21 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-02-10T09:06:21.401 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:21 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: File "/var/lib/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-02-10T09:06:21.401 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:21 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-02-10T09:06:21.401 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:21 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: RuntimeError: Failed command: systemctl start ceph-6081b54a-c7f2-11ee-95b8-87774f69a715@node-exporter.smithi077 2024-02-10T09:06:21.401 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:21 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: Traceback (most recent call last): 2024-02-10T09:06:21.401 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:21 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-02-10T09:06:21.401 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:21 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: yield (conn, connr) 2024-02-10T09:06:21.401 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:21 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-02-10T09:06:21.401 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:21 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: code, '\n'.join(err))) 2024-02-10T09:06:21.401 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:21 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi077 ... 2024-02-10T09:06:21.401 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:21 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: Verifying port 9100 ... 2024-02-10T09:06:21.402 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:21 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: Non-zero exit code 1 from systemctl start ceph-6081b54a-c7f2-11ee-95b8-87774f69a715@node-exporter.smithi077 2024-02-10T09:06:21.402 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:21 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: systemctl: stderr Job for ceph-6081b54a-c7f2-11ee-95b8-87774f69a715@node-exporter.smithi077.service failed because a timeout was exceeded. 2024-02-10T09:06:21.402 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:21 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: systemctl: stderr See "systemctl status ceph-6081b54a-c7f2-11ee-95b8-87774f69a715@node-exporter.smithi077.service" and "journalctl -xe" for details. 2024-02-10T09:06:21.402 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:21 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: Traceback (most recent call last): 2024-02-10T09:06:21.402 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:21 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: File "/var/lib/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-10T09:06:21.402 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:21 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: main() 2024-02-10T09:06:21.402 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:21 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: File "/var/lib/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-10T09:06:21.402 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:21 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: r = ctx.func(ctx) 2024-02-10T09:06:21.402 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:21 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: File "/var/lib/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-10T09:06:21.402 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:21 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: return func(ctx) 2024-02-10T09:06:21.402 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:21 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: File "/var/lib/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-02-10T09:06:21.402 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:21 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: ports=daemon_ports) 2024-02-10T09:06:21.402 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:21 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: File "/var/lib/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2024-02-10T09:06:21.402 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:21 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: c, osd_fsid=osd_fsid, ports=ports) 2024-02-10T09:06:21.403 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:21 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: File "/var/lib/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2024-02-10T09:06:21.403 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:21 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-02-10T09:06:21.403 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:21 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: File "/var/lib/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-02-10T09:06:21.403 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:21 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-02-10T09:06:21.403 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:21 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: RuntimeError: Failed command: systemctl start ceph-6081b54a-c7f2-11ee-95b8-87774f69a715@node-exporter.smithi077 2024-02-10T09:06:21.403 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:21 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cephadm 2024-02-10T09:06:20.194898+0000 mgr.smithi077.dxntvw (mgr.14162) 234 : cephadm [INF] Deploying daemon prometheus.smithi077 on smithi077 2024-02-10T09:06:21.403 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:21 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:06:21.026897+0000 mgr.smithi077.dxntvw (mgr.14162) 235 : cluster [DBG] pgmap v221: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:06:22.066 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:22.064+0000 7fecc879b700 1 -- 172.21.15.155:0/589236736 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fecc00f6510 msgr2=0x7fecc00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:06:22.067 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:22.064+0000 7fecc879b700 1 --2- 172.21.15.155:0/589236736 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fecc00f6510 0x7fecc00f6930 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7fecb0009fb0 tx=0x7fecb0009b30).stop 2024-02-10T09:06:22.067 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:22.064+0000 7fecc879b700 1 -- 172.21.15.155:0/589236736 shutdown_connections 2024-02-10T09:06:22.067 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:22.064+0000 7fecc879b700 1 --2- 172.21.15.155:0/589236736 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fecc00f6510 0x7fecc00f6930 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:06:22.067 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:22.064+0000 7fecc879b700 1 -- 172.21.15.155:0/589236736 >> 172.21.15.155:0/589236736 conn(0x7fecc00f1c70 msgr2=0x7fecc00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:06:22.067 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:22.064+0000 7fecc879b700 1 -- 172.21.15.155:0/589236736 shutdown_connections 2024-02-10T09:06:22.067 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:22.064+0000 7fecc879b700 1 -- 172.21.15.155:0/589236736 wait complete. 2024-02-10T09:06:22.067 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:22.065+0000 7fecc879b700 1 Processor -- start 2024-02-10T09:06:22.067 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:22.065+0000 7fecc879b700 1 -- start start 2024-02-10T09:06:22.067 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:22.065+0000 7fecc879b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fecc00f6510 0x7fecc01067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:06:22.067 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:22.065+0000 7fecc879b700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecc0106cd0 con 0x7fecc00f6510 2024-02-10T09:06:22.068 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:22.065+0000 7fecc6537700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fecc00f6510 0x7fecc01067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:06:22.068 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:22.065+0000 7fecc6537700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fecc00f6510 0x7fecc01067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:59208/0 (socket says 172.21.15.155:59208) 2024-02-10T09:06:22.068 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:22.065+0000 7fecc6537700 1 -- 172.21.15.155:0/522309023 learned_addr learned my addr 172.21.15.155:0/522309023 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:06:22.068 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:22.065+0000 7fecc6537700 1 -- 172.21.15.155:0/522309023 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecb0005040 con 0x7fecc00f6510 2024-02-10T09:06:22.068 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:22.066+0000 7fecc6537700 1 --2- 172.21.15.155:0/522309023 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fecc00f6510 0x7fecc01067e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7fecb0005010 tx=0x7fecb000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:06:22.068 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:22.066+0000 7fecb6ffd700 1 -- 172.21.15.155:0/522309023 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fecb0009320 con 0x7fecc00f6510 2024-02-10T09:06:22.068 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:22.066+0000 7fecb6ffd700 1 -- 172.21.15.155:0/522309023 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fecb0013380 con 0x7fecc00f6510 2024-02-10T09:06:22.068 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:22.066+0000 7fecb6ffd700 1 -- 172.21.15.155:0/522309023 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fecb0014e50 con 0x7fecc00f6510 2024-02-10T09:06:22.068 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:22.066+0000 7fecc879b700 1 -- 172.21.15.155:0/522309023 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecc0106ed0 con 0x7fecc00f6510 2024-02-10T09:06:22.068 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:22.066+0000 7fecc879b700 1 -- 172.21.15.155:0/522309023 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecc0107360 con 0x7fecc00f6510 2024-02-10T09:06:22.068 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:22.067+0000 7fecb6ffd700 1 -- 172.21.15.155:0/522309023 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fecb00095f0 con 0x7fecc00f6510 2024-02-10T09:06:22.069 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:22.067+0000 7fecb6ffd700 1 --2- 172.21.15.155:0/522309023 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fecac02ff10 0x7fecac0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:06:22.069 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:22.067+0000 7fecc879b700 1 -- 172.21.15.155:0/522309023 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecc0045ad0 con 0x7fecc00f6510 2024-02-10T09:06:22.069 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:22.067+0000 7fecb6ffd700 1 -- 172.21.15.155:0/522309023 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fecb0040e50 con 0x7fecc00f6510 2024-02-10T09:06:22.069 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:22.067+0000 7fecc5d36700 1 --2- 172.21.15.155:0/522309023 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fecac02ff10 0x7fecac0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:06:22.070 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:22.067+0000 7fecc5d36700 1 --2- 172.21.15.155:0/522309023 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fecac02ff10 0x7fecac0323d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fecbc0096a0 tx=0x7fecbc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:06:22.071 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:22.071+0000 7fecb6ffd700 1 -- 172.21.15.155:0/522309023 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fecb001d980 con 0x7fecc00f6510 2024-02-10T09:06:22.278 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:22.277+0000 7fecc879b700 1 -- 172.21.15.155:0/522309023 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fecc0108210 con 0x7fecc00f6510 2024-02-10T09:06:22.278 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:22.278+0000 7fecb6ffd700 1 -- 172.21.15.155:0/522309023 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fecb0007dd0 con 0x7fecc00f6510 2024-02-10T09:06:22.278 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:06:22.278 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:06:22.280 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:22.279+0000 7fecc879b700 1 -- 172.21.15.155:0/522309023 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fecac02ff10 msgr2=0x7fecac0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:06:22.280 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:22.279+0000 7fecc879b700 1 --2- 172.21.15.155:0/522309023 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fecac02ff10 0x7fecac0323d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fecbc0096a0 tx=0x7fecbc006b40).stop 2024-02-10T09:06:22.280 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:22.279+0000 7fecc879b700 1 -- 172.21.15.155:0/522309023 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fecc00f6510 msgr2=0x7fecc01067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:06:22.280 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:22.279+0000 7fecc879b700 1 --2- 172.21.15.155:0/522309023 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fecc00f6510 0x7fecc01067e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7fecb0005010 tx=0x7fecb000b040).stop 2024-02-10T09:06:22.280 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:22.279+0000 7fecc879b700 1 -- 172.21.15.155:0/522309023 shutdown_connections 2024-02-10T09:06:22.280 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:22.279+0000 7fecc879b700 1 --2- 172.21.15.155:0/522309023 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fecac02ff10 0x7fecac0323d0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:06:22.280 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:22.279+0000 7fecc879b700 1 --2- 172.21.15.155:0/522309023 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fecc00f6510 0x7fecc01067e0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:06:22.280 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:22.279+0000 7fecc879b700 1 -- 172.21.15.155:0/522309023 >> 172.21.15.155:0/522309023 conn(0x7fecc00f1c70 msgr2=0x7fecc00f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:06:22.280 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:22.279+0000 7fecc879b700 1 -- 172.21.15.155:0/522309023 shutdown_connections 2024-02-10T09:06:22.280 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:22.280+0000 7fecc879b700 1 -- 172.21.15.155:0/522309023 wait complete. 2024-02-10T09:06:22.281 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:06:22.649 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:22 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:06:22.278214+0000 mon.smithi077 (mon.0) 244 : audit [DBG] from='client.? 172.21.15.155:0/522309023' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:06:24.054 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:06:24.055 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:06:24.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:23 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:06:23.027207+0000 mgr.smithi077.dxntvw (mgr.14162) 236 : cluster [DBG] pgmap v222: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:06:25.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:25 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:06:25.027481+0000 mgr.smithi077.dxntvw (mgr.14162) 237 : cluster [DBG] pgmap v223: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:06:26.183 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:26.183+0000 7f96bab44700 1 -- 172.21.15.155:0/2607916396 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96b40f4860 msgr2=0x7f96b40f4c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:06:26.184 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:26.183+0000 7f96bab44700 1 --2- 172.21.15.155:0/2607916396 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96b40f4860 0x7f96b40f4c80 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f96a8009fb0 tx=0x7f96a8009b30).stop 2024-02-10T09:06:26.184 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:26.183+0000 7f96bab44700 1 -- 172.21.15.155:0/2607916396 shutdown_connections 2024-02-10T09:06:26.184 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:26.183+0000 7f96bab44700 1 --2- 172.21.15.155:0/2607916396 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96b40f4860 0x7f96b40f4c80 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:06:26.184 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:26.183+0000 7f96bab44700 1 -- 172.21.15.155:0/2607916396 >> 172.21.15.155:0/2607916396 conn(0x7f96b40f02e0 msgr2=0x7f96b40f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:06:26.184 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:26.183+0000 7f96bab44700 1 -- 172.21.15.155:0/2607916396 shutdown_connections 2024-02-10T09:06:26.184 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:26.183+0000 7f96bab44700 1 -- 172.21.15.155:0/2607916396 wait complete. 2024-02-10T09:06:26.186 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:26.184+0000 7f96bab44700 1 Processor -- start 2024-02-10T09:06:26.186 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:26.184+0000 7f96bab44700 1 -- start start 2024-02-10T09:06:26.186 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:26.184+0000 7f96bab44700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96b40f4860 0x7f96b41069c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:06:26.186 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:26.184+0000 7f96bab44700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96b4106eb0 con 0x7f96b40f4860 2024-02-10T09:06:26.186 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:26.185+0000 7f96b88e0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96b40f4860 0x7f96b41069c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:06:26.186 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:26.185+0000 7f96b88e0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96b40f4860 0x7f96b41069c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:49214/0 (socket says 172.21.15.155:49214) 2024-02-10T09:06:26.186 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:26.185+0000 7f96b88e0700 1 -- 172.21.15.155:0/2889721264 learned_addr learned my addr 172.21.15.155:0/2889721264 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:06:26.186 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:26.185+0000 7f96b88e0700 1 -- 172.21.15.155:0/2889721264 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96a8005040 con 0x7f96b40f4860 2024-02-10T09:06:26.186 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:26.185+0000 7f96b88e0700 1 --2- 172.21.15.155:0/2889721264 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96b40f4860 0x7f96b41069c0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f96a80089f0 tx=0x7f96a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:06:26.186 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:26.185+0000 7f96b17fa700 1 -- 172.21.15.155:0/2889721264 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f96a8009320 con 0x7f96b40f4860 2024-02-10T09:06:26.187 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:26.185+0000 7f96bab44700 1 -- 172.21.15.155:0/2889721264 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96b41070b0 con 0x7f96b40f4860 2024-02-10T09:06:26.187 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:26.185+0000 7f96b17fa700 1 -- 172.21.15.155:0/2889721264 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f96a8013380 con 0x7f96b40f4860 2024-02-10T09:06:26.187 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:26.185+0000 7f96b17fa700 1 -- 172.21.15.155:0/2889721264 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f96a8014e20 con 0x7f96b40f4860 2024-02-10T09:06:26.187 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:26.185+0000 7f96bab44700 1 -- 172.21.15.155:0/2889721264 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96b4107540 con 0x7f96b40f4860 2024-02-10T09:06:26.187 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:26.186+0000 7f96bab44700 1 -- 172.21.15.155:0/2889721264 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96b4045ad0 con 0x7f96b40f4860 2024-02-10T09:06:26.187 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:26.186+0000 7f96b17fa700 1 -- 172.21.15.155:0/2889721264 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f96a80095f0 con 0x7f96b40f4860 2024-02-10T09:06:26.188 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:26.186+0000 7f96b17fa700 1 --2- 172.21.15.155:0/2889721264 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f96a402ff10 0x7f96a40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:06:26.188 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:26.186+0000 7f96b17fa700 1 -- 172.21.15.155:0/2889721264 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f96a8040e90 con 0x7f96b40f4860 2024-02-10T09:06:26.188 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:26.187+0000 7f96b3fff700 1 --2- 172.21.15.155:0/2889721264 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f96a402ff10 0x7f96a40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:06:26.188 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:26.187+0000 7f96b3fff700 1 --2- 172.21.15.155:0/2889721264 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f96a402ff10 0x7f96a40323d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f96a00096a0 tx=0x7f96a0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:06:26.191 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:26.191+0000 7f96b17fa700 1 -- 172.21.15.155:0/2889721264 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f96a8014420 con 0x7f96b40f4860 2024-02-10T09:06:26.400 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:26.400+0000 7f96bab44700 1 -- 172.21.15.155:0/2889721264 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f96b405f370 con 0x7f96b40f4860 2024-02-10T09:06:26.400 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:26.400+0000 7f96b17fa700 1 -- 172.21.15.155:0/2889721264 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f96a801d980 con 0x7f96b40f4860 2024-02-10T09:06:26.400 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:06:26.401 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:06:26.402 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:26.402+0000 7f96bab44700 1 -- 172.21.15.155:0/2889721264 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f96a402ff10 msgr2=0x7f96a40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:06:26.402 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:26.402+0000 7f96bab44700 1 --2- 172.21.15.155:0/2889721264 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f96a402ff10 0x7f96a40323d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f96a00096a0 tx=0x7f96a0006b40).stop 2024-02-10T09:06:26.402 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:26.402+0000 7f96bab44700 1 -- 172.21.15.155:0/2889721264 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96b40f4860 msgr2=0x7f96b41069c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:06:26.402 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:26.402+0000 7f96bab44700 1 --2- 172.21.15.155:0/2889721264 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96b40f4860 0x7f96b41069c0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f96a80089f0 tx=0x7f96a800b040).stop 2024-02-10T09:06:26.402 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:26.402+0000 7f96bab44700 1 -- 172.21.15.155:0/2889721264 shutdown_connections 2024-02-10T09:06:26.402 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:26.402+0000 7f96bab44700 1 --2- 172.21.15.155:0/2889721264 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f96a402ff10 0x7f96a40323d0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:06:26.403 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:26.402+0000 7f96bab44700 1 --2- 172.21.15.155:0/2889721264 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96b40f4860 0x7f96b41069c0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:06:26.403 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:26.402+0000 7f96bab44700 1 -- 172.21.15.155:0/2889721264 >> 172.21.15.155:0/2889721264 conn(0x7f96b40f02e0 msgr2=0x7f96b40f0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:06:26.403 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:26.402+0000 7f96bab44700 1 -- 172.21.15.155:0/2889721264 shutdown_connections 2024-02-10T09:06:26.403 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:26.402+0000 7f96bab44700 1 -- 172.21.15.155:0/2889721264 wait complete. 2024-02-10T09:06:26.403 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:06:26.899 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:26 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:06:26.400919+0000 mon.smithi077 (mon.0) 245 : audit [DBG] from='client.? 172.21.15.155:0/2889721264' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:06:27.899 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:27 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:06:27.027831+0000 mgr.smithi077.dxntvw (mgr.14162) 238 : cluster [DBG] pgmap v224: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:06:28.184 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:06:28.184 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:06:29.398 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:29 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:06:29.028165+0000 mgr.smithi077.dxntvw (mgr.14162) 239 : cluster [DBG] pgmap v225: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:06:30.376 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:30.374+0000 7f2b60b48700 1 -- 172.21.15.155:0/271808027 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b5c0f5410 msgr2=0x7f2b5c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:06:30.377 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:30.374+0000 7f2b60b48700 1 --2- 172.21.15.155:0/271808027 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b5c0f5410 0x7f2b5c0f5830 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f2b50009fb0 tx=0x7f2b50009b30).stop 2024-02-10T09:06:30.377 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:30.374+0000 7f2b60b48700 1 -- 172.21.15.155:0/271808027 shutdown_connections 2024-02-10T09:06:30.377 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:30.374+0000 7f2b60b48700 1 --2- 172.21.15.155:0/271808027 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b5c0f5410 0x7f2b5c0f5830 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:06:30.377 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:30.374+0000 7f2b60b48700 1 -- 172.21.15.155:0/271808027 >> 172.21.15.155:0/271808027 conn(0x7f2b5c0f0b70 msgr2=0x7f2b5c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:06:30.377 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:30.374+0000 7f2b60b48700 1 -- 172.21.15.155:0/271808027 shutdown_connections 2024-02-10T09:06:30.377 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:30.375+0000 7f2b60b48700 1 -- 172.21.15.155:0/271808027 wait complete. 2024-02-10T09:06:30.377 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:30.375+0000 7f2b60b48700 1 Processor -- start 2024-02-10T09:06:30.377 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:30.375+0000 7f2b60b48700 1 -- start start 2024-02-10T09:06:30.377 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:30.375+0000 7f2b60b48700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b5c0f5410 0x7f2b5c10b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:06:30.377 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:30.375+0000 7f2b60b48700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b5c107330 con 0x7f2b5c0f5410 2024-02-10T09:06:30.377 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:30.376+0000 7f2b5a59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b5c0f5410 0x7f2b5c10b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:06:30.377 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:30.376+0000 7f2b5a59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b5c0f5410 0x7f2b5c10b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:49232/0 (socket says 172.21.15.155:49232) 2024-02-10T09:06:30.377 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:30.376+0000 7f2b5a59c700 1 -- 172.21.15.155:0/1257797982 learned_addr learned my addr 172.21.15.155:0/1257797982 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:06:30.378 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:30.376+0000 7f2b5a59c700 1 -- 172.21.15.155:0/1257797982 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b50005040 con 0x7f2b5c0f5410 2024-02-10T09:06:30.378 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:30.376+0000 7f2b5a59c700 1 --2- 172.21.15.155:0/1257797982 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b5c0f5410 0x7f2b5c10b940 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f2b500089f0 tx=0x7f2b5000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:06:30.378 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:30.376+0000 7f2b4affd700 1 -- 172.21.15.155:0/1257797982 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b50009320 con 0x7f2b5c0f5410 2024-02-10T09:06:30.378 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:30.376+0000 7f2b60b48700 1 -- 172.21.15.155:0/1257797982 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b5c060220 con 0x7f2b5c0f5410 2024-02-10T09:06:30.378 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:30.376+0000 7f2b4affd700 1 -- 172.21.15.155:0/1257797982 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2b50013380 con 0x7f2b5c0f5410 2024-02-10T09:06:30.378 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:30.376+0000 7f2b4affd700 1 -- 172.21.15.155:0/1257797982 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b5001ed80 con 0x7f2b5c0f5410 2024-02-10T09:06:30.378 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:30.376+0000 7f2b60b48700 1 -- 172.21.15.155:0/1257797982 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b5c10be30 con 0x7f2b5c0f5410 2024-02-10T09:06:30.379 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:30.377+0000 7f2b4affd700 1 -- 172.21.15.155:0/1257797982 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2b5001e420 con 0x7f2b5c0f5410 2024-02-10T09:06:30.379 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:30.377+0000 7f2b60b48700 1 -- 172.21.15.155:0/1257797982 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b5c045ad0 con 0x7f2b5c0f5410 2024-02-10T09:06:30.379 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:30.377+0000 7f2b4affd700 1 --2- 172.21.15.155:0/1257797982 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f2b4402ff60 0x7f2b44032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:06:30.379 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:30.377+0000 7f2b4affd700 1 -- 172.21.15.155:0/1257797982 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2b50041120 con 0x7f2b5c0f5410 2024-02-10T09:06:30.379 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:30.377+0000 7f2b59d9b700 1 --2- 172.21.15.155:0/1257797982 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f2b4402ff60 0x7f2b44032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:06:30.379 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:30.378+0000 7f2b59d9b700 1 --2- 172.21.15.155:0/1257797982 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f2b4402ff60 0x7f2b44032420 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f2b4c0096a0 tx=0x7f2b4c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:06:30.382 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:30.382+0000 7f2b4affd700 1 -- 172.21.15.155:0/1257797982 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2b5001cb10 con 0x7f2b5c0f5410 2024-02-10T09:06:30.587 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:30.587+0000 7f2b60b48700 1 -- 172.21.15.155:0/1257797982 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2b5c10c5f0 con 0x7f2b5c0f5410 2024-02-10T09:06:30.587 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:30.587+0000 7f2b4affd700 1 -- 172.21.15.155:0/1257797982 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2b5002c2d0 con 0x7f2b5c0f5410 2024-02-10T09:06:30.588 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:06:30.588 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:06:30.589 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:30.589+0000 7f2b60b48700 1 -- 172.21.15.155:0/1257797982 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f2b4402ff60 msgr2=0x7f2b44032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:06:30.589 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:30.589+0000 7f2b60b48700 1 --2- 172.21.15.155:0/1257797982 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f2b4402ff60 0x7f2b44032420 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f2b4c0096a0 tx=0x7f2b4c006b40).stop 2024-02-10T09:06:30.589 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:30.589+0000 7f2b60b48700 1 -- 172.21.15.155:0/1257797982 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b5c0f5410 msgr2=0x7f2b5c10b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:06:30.589 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:30.589+0000 7f2b60b48700 1 --2- 172.21.15.155:0/1257797982 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b5c0f5410 0x7f2b5c10b940 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f2b500089f0 tx=0x7f2b5000b040).stop 2024-02-10T09:06:30.589 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:30.589+0000 7f2b60b48700 1 -- 172.21.15.155:0/1257797982 shutdown_connections 2024-02-10T09:06:30.589 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:30.589+0000 7f2b60b48700 1 --2- 172.21.15.155:0/1257797982 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f2b4402ff60 0x7f2b44032420 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:06:30.590 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:30.589+0000 7f2b60b48700 1 --2- 172.21.15.155:0/1257797982 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2b5c0f5410 0x7f2b5c10b940 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:06:30.590 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:30.589+0000 7f2b60b48700 1 -- 172.21.15.155:0/1257797982 >> 172.21.15.155:0/1257797982 conn(0x7f2b5c0f0b70 msgr2=0x7f2b5c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:06:30.590 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:30.589+0000 7f2b60b48700 1 -- 172.21.15.155:0/1257797982 shutdown_connections 2024-02-10T09:06:30.590 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:30.589+0000 7f2b60b48700 1 -- 172.21.15.155:0/1257797982 wait complete. 2024-02-10T09:06:30.590 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:06:30.899 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:30 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:06:30.587924+0000 mon.smithi077 (mon.0) 246 : audit [DBG] from='client.? 172.21.15.155:0/1257797982' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:06:31.899 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:31 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:06:31.028452+0000 mgr.smithi077.dxntvw (mgr.14162) 240 : cluster [DBG] pgmap v226: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:06:32.552 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:06:32.552 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:06:34.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:33 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:06:33.028745+0000 mgr.smithi077.dxntvw (mgr.14162) 241 : cluster [DBG] pgmap v227: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:06:34.791 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:34.789+0000 7fbd21811700 1 -- 172.21.15.155:0/1598922551 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd1c0f5410 msgr2=0x7fbd1c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:06:34.791 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:34.789+0000 7fbd21811700 1 --2- 172.21.15.155:0/1598922551 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd1c0f5410 0x7fbd1c0f5830 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7fbd04009fb0 tx=0x7fbd04009b30).stop 2024-02-10T09:06:34.792 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:34.789+0000 7fbd21811700 1 -- 172.21.15.155:0/1598922551 shutdown_connections 2024-02-10T09:06:34.792 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:34.789+0000 7fbd21811700 1 --2- 172.21.15.155:0/1598922551 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd1c0f5410 0x7fbd1c0f5830 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:06:34.792 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:34.789+0000 7fbd21811700 1 -- 172.21.15.155:0/1598922551 >> 172.21.15.155:0/1598922551 conn(0x7fbd1c0f0b70 msgr2=0x7fbd1c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:06:34.792 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:34.789+0000 7fbd21811700 1 -- 172.21.15.155:0/1598922551 shutdown_connections 2024-02-10T09:06:34.792 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:34.789+0000 7fbd21811700 1 -- 172.21.15.155:0/1598922551 wait complete. 2024-02-10T09:06:34.792 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:34.790+0000 7fbd21811700 1 Processor -- start 2024-02-10T09:06:34.792 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:34.790+0000 7fbd21811700 1 -- start start 2024-02-10T09:06:34.792 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:34.790+0000 7fbd21811700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd1c0f5410 0x7fbd1c105720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:06:34.792 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:34.790+0000 7fbd21811700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd1c105c10 con 0x7fbd1c0f5410 2024-02-10T09:06:34.793 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:34.790+0000 7fbd1affd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd1c0f5410 0x7fbd1c105720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:06:34.793 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:34.790+0000 7fbd1affd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd1c0f5410 0x7fbd1c105720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:52574/0 (socket says 172.21.15.155:52574) 2024-02-10T09:06:34.793 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:34.790+0000 7fbd1affd700 1 -- 172.21.15.155:0/3437922 learned_addr learned my addr 172.21.15.155:0/3437922 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:06:34.793 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:34.791+0000 7fbd1affd700 1 -- 172.21.15.155:0/3437922 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd04005040 con 0x7fbd1c0f5410 2024-02-10T09:06:34.793 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:34.791+0000 7fbd1affd700 1 --2- 172.21.15.155:0/3437922 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd1c0f5410 0x7fbd1c105720 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7fbd040089f0 tx=0x7fbd0400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:06:34.793 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:34.791+0000 7fbd137fe700 1 -- 172.21.15.155:0/3437922 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd04009320 con 0x7fbd1c0f5410 2024-02-10T09:06:34.793 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:34.791+0000 7fbd21811700 1 -- 172.21.15.155:0/3437922 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd1c105e10 con 0x7fbd1c0f5410 2024-02-10T09:06:34.793 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:34.791+0000 7fbd137fe700 1 -- 172.21.15.155:0/3437922 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbd04013380 con 0x7fbd1c0f5410 2024-02-10T09:06:34.793 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:34.791+0000 7fbd137fe700 1 -- 172.21.15.155:0/3437922 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd0401ed80 con 0x7fbd1c0f5410 2024-02-10T09:06:34.793 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:34.791+0000 7fbd21811700 1 -- 172.21.15.155:0/3437922 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd1c1062a0 con 0x7fbd1c0f5410 2024-02-10T09:06:34.794 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:34.792+0000 7fbd137fe700 1 -- 172.21.15.155:0/3437922 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbd0401e420 con 0x7fbd1c0f5410 2024-02-10T09:06:34.794 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:34.792+0000 7fbd21811700 1 -- 172.21.15.155:0/3437922 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd1c045ad0 con 0x7fbd1c0f5410 2024-02-10T09:06:34.794 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:34.792+0000 7fbd137fe700 1 --2- 172.21.15.155:0/3437922 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fbd0802ff60 0x7fbd08032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:06:34.794 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:34.792+0000 7fbd137fe700 1 -- 172.21.15.155:0/3437922 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbd04041050 con 0x7fbd1c0f5410 2024-02-10T09:06:34.794 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:34.792+0000 7fbd1a7fc700 1 --2- 172.21.15.155:0/3437922 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fbd0802ff60 0x7fbd08032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:06:34.795 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:34.793+0000 7fbd1a7fc700 1 --2- 172.21.15.155:0/3437922 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fbd0802ff60 0x7fbd08032420 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fbd0c0096a0 tx=0x7fbd0c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:06:34.797 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:34.797+0000 7fbd137fe700 1 -- 172.21.15.155:0/3437922 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbd0401d380 con 0x7fbd1c0f5410 2024-02-10T09:06:35.002 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:35.002+0000 7fbd21811700 1 -- 172.21.15.155:0/3437922 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbd1c107150 con 0x7fbd1c0f5410 2024-02-10T09:06:35.002 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:35.002+0000 7fbd137fe700 1 -- 172.21.15.155:0/3437922 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbd04004490 con 0x7fbd1c0f5410 2024-02-10T09:06:35.003 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:06:35.003 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:06:35.004 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:35.004+0000 7fbd21811700 1 -- 172.21.15.155:0/3437922 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fbd0802ff60 msgr2=0x7fbd08032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:06:35.004 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:35.004+0000 7fbd21811700 1 --2- 172.21.15.155:0/3437922 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fbd0802ff60 0x7fbd08032420 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fbd0c0096a0 tx=0x7fbd0c006b40).stop 2024-02-10T09:06:35.004 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:35.004+0000 7fbd21811700 1 -- 172.21.15.155:0/3437922 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd1c0f5410 msgr2=0x7fbd1c105720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:06:35.004 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:35.004+0000 7fbd21811700 1 --2- 172.21.15.155:0/3437922 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd1c0f5410 0x7fbd1c105720 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7fbd040089f0 tx=0x7fbd0400b040).stop 2024-02-10T09:06:35.004 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:35.004+0000 7fbd21811700 1 -- 172.21.15.155:0/3437922 shutdown_connections 2024-02-10T09:06:35.004 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:35.004+0000 7fbd21811700 1 --2- 172.21.15.155:0/3437922 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fbd0802ff60 0x7fbd08032420 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:06:35.005 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:35.004+0000 7fbd21811700 1 --2- 172.21.15.155:0/3437922 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd1c0f5410 0x7fbd1c105720 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:06:35.005 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:35.004+0000 7fbd21811700 1 -- 172.21.15.155:0/3437922 >> 172.21.15.155:0/3437922 conn(0x7fbd1c0f0b70 msgr2=0x7fbd1c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:06:35.005 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:35.004+0000 7fbd21811700 1 -- 172.21.15.155:0/3437922 shutdown_connections 2024-02-10T09:06:35.005 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:35.004+0000 7fbd21811700 1 -- 172.21.15.155:0/3437922 wait complete. 2024-02-10T09:06:35.005 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:06:35.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:35 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:06:35.002867+0000 mon.smithi077 (mon.0) 247 : audit [DBG] from='client.? 172.21.15.155:0/3437922' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:06:35.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:35 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:06:35.029037+0000 mgr.smithi077.dxntvw (mgr.14162) 242 : cluster [DBG] pgmap v228: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:06:36.891 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:06:36.891 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:06:37.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:37 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:06:37.029347+0000 mgr.smithi077.dxntvw (mgr.14162) 243 : cluster [DBG] pgmap v229: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:06:39.218 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:39.217+0000 7fb566b4c700 1 -- 172.21.15.155:0/137686377 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5600f6510 msgr2=0x7fb5600f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:06:39.219 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:39.217+0000 7fb566b4c700 1 --2- 172.21.15.155:0/137686377 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5600f6510 0x7fb5600f6930 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7fb554009fb0 tx=0x7fb554009b30).stop 2024-02-10T09:06:39.219 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:39.217+0000 7fb566b4c700 1 -- 172.21.15.155:0/137686377 shutdown_connections 2024-02-10T09:06:39.219 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:39.217+0000 7fb566b4c700 1 --2- 172.21.15.155:0/137686377 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5600f6510 0x7fb5600f6930 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:06:39.219 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:39.217+0000 7fb566b4c700 1 -- 172.21.15.155:0/137686377 >> 172.21.15.155:0/137686377 conn(0x7fb5600f1c70 msgr2=0x7fb5600f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:06:39.219 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:39.217+0000 7fb566b4c700 1 -- 172.21.15.155:0/137686377 shutdown_connections 2024-02-10T09:06:39.219 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:39.217+0000 7fb566b4c700 1 -- 172.21.15.155:0/137686377 wait complete. 2024-02-10T09:06:39.219 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:39.217+0000 7fb566b4c700 1 Processor -- start 2024-02-10T09:06:39.219 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:39.218+0000 7fb566b4c700 1 -- start start 2024-02-10T09:06:39.220 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:39.218+0000 7fb566b4c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5600f6510 0x7fb5601067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:06:39.220 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:39.218+0000 7fb566b4c700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb560106cd0 con 0x7fb5600f6510 2024-02-10T09:06:39.220 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:39.218+0000 7fb5648e8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5600f6510 0x7fb5601067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:06:39.221 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:39.218+0000 7fb5648e8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5600f6510 0x7fb5601067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:52594/0 (socket says 172.21.15.155:52594) 2024-02-10T09:06:39.221 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:39.218+0000 7fb5648e8700 1 -- 172.21.15.155:0/927890372 learned_addr learned my addr 172.21.15.155:0/927890372 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:06:39.221 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:39.219+0000 7fb5648e8700 1 -- 172.21.15.155:0/927890372 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb554005040 con 0x7fb5600f6510 2024-02-10T09:06:39.221 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:39.219+0000 7fb5648e8700 1 --2- 172.21.15.155:0/927890372 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5600f6510 0x7fb5601067e0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7fb554005010 tx=0x7fb55400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:06:39.221 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:39.219+0000 7fb55d7fa700 1 -- 172.21.15.155:0/927890372 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb554009320 con 0x7fb5600f6510 2024-02-10T09:06:39.221 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:39.219+0000 7fb566b4c700 1 -- 172.21.15.155:0/927890372 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb560106ed0 con 0x7fb5600f6510 2024-02-10T09:06:39.221 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:39.219+0000 7fb55d7fa700 1 -- 172.21.15.155:0/927890372 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb554013380 con 0x7fb5600f6510 2024-02-10T09:06:39.221 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:39.219+0000 7fb55d7fa700 1 -- 172.21.15.155:0/927890372 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb554014e50 con 0x7fb5600f6510 2024-02-10T09:06:39.221 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:39.219+0000 7fb566b4c700 1 -- 172.21.15.155:0/927890372 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb560107360 con 0x7fb5600f6510 2024-02-10T09:06:39.222 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:39.220+0000 7fb55d7fa700 1 -- 172.21.15.155:0/927890372 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb5540095f0 con 0x7fb5600f6510 2024-02-10T09:06:39.222 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:39.220+0000 7fb566b4c700 1 -- 172.21.15.155:0/927890372 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb560045ad0 con 0x7fb5600f6510 2024-02-10T09:06:39.222 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:39.220+0000 7fb55d7fa700 1 --2- 172.21.15.155:0/927890372 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb55002ff10 0x7fb5500323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:06:39.222 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:39.220+0000 7fb55d7fa700 1 -- 172.21.15.155:0/927890372 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb554040e90 con 0x7fb5600f6510 2024-02-10T09:06:39.222 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:39.220+0000 7fb55ffff700 1 --2- 172.21.15.155:0/927890372 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb55002ff10 0x7fb5500323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:06:39.222 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:39.221+0000 7fb55ffff700 1 --2- 172.21.15.155:0/927890372 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb55002ff10 0x7fb5500323d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fb54c0096a0 tx=0x7fb54c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:06:39.225 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:39.225+0000 7fb55d7fa700 1 -- 172.21.15.155:0/927890372 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb554014420 con 0x7fb5600f6510 2024-02-10T09:06:39.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:39 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:06:39.029612+0000 mgr.smithi077.dxntvw (mgr.14162) 244 : cluster [DBG] pgmap v230: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:06:39.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:39.431+0000 7fb566b4c700 1 -- 172.21.15.155:0/927890372 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb5601081b0 con 0x7fb5600f6510 2024-02-10T09:06:39.433 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:39.432+0000 7fb55d7fa700 1 -- 172.21.15.155:0/927890372 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb554010230 con 0x7fb5600f6510 2024-02-10T09:06:39.433 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:06:39.433 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:06:39.435 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:39.433+0000 7fb566b4c700 1 -- 172.21.15.155:0/927890372 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb55002ff10 msgr2=0x7fb5500323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:06:39.435 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:39.433+0000 7fb566b4c700 1 --2- 172.21.15.155:0/927890372 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb55002ff10 0x7fb5500323d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fb54c0096a0 tx=0x7fb54c006b40).stop 2024-02-10T09:06:39.435 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:39.434+0000 7fb566b4c700 1 -- 172.21.15.155:0/927890372 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5600f6510 msgr2=0x7fb5601067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:06:39.435 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:39.434+0000 7fb566b4c700 1 --2- 172.21.15.155:0/927890372 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5600f6510 0x7fb5601067e0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7fb554005010 tx=0x7fb55400b040).stop 2024-02-10T09:06:39.435 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:39.434+0000 7fb566b4c700 1 -- 172.21.15.155:0/927890372 shutdown_connections 2024-02-10T09:06:39.435 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:39.434+0000 7fb566b4c700 1 --2- 172.21.15.155:0/927890372 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb55002ff10 0x7fb5500323d0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:06:39.435 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:39.434+0000 7fb566b4c700 1 --2- 172.21.15.155:0/927890372 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb5600f6510 0x7fb5601067e0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:06:39.435 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:39.434+0000 7fb566b4c700 1 -- 172.21.15.155:0/927890372 >> 172.21.15.155:0/927890372 conn(0x7fb5600f1c70 msgr2=0x7fb5600f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:06:39.435 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:39.434+0000 7fb566b4c700 1 -- 172.21.15.155:0/927890372 shutdown_connections 2024-02-10T09:06:39.436 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:39.434+0000 7fb566b4c700 1 -- 172.21.15.155:0/927890372 wait complete. 2024-02-10T09:06:39.436 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:06:40.398 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:40 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:06:39.433519+0000 mon.smithi077 (mon.0) 248 : audit [DBG] from='client.? 172.21.15.155:0/927890372' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:06:41.298 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:06:41.298 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:06:41.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:41 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:06:41.029953+0000 mgr.smithi077.dxntvw (mgr.14162) 245 : cluster [DBG] pgmap v231: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:06:43.261 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:43.253+0000 7f4b4b559700 1 -- 172.21.15.155:0/4110629375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b440f6510 msgr2=0x7f4b440f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:06:43.261 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:43.253+0000 7f4b4b559700 1 --2- 172.21.15.155:0/4110629375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b440f6510 0x7f4b440f6930 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f4b34009fb0 tx=0x7f4b34009b30).stop 2024-02-10T09:06:43.261 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:43.253+0000 7f4b4b559700 1 -- 172.21.15.155:0/4110629375 shutdown_connections 2024-02-10T09:06:43.261 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:43.253+0000 7f4b4b559700 1 --2- 172.21.15.155:0/4110629375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b440f6510 0x7f4b440f6930 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:06:43.262 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:43.253+0000 7f4b4b559700 1 -- 172.21.15.155:0/4110629375 >> 172.21.15.155:0/4110629375 conn(0x7f4b440f1c70 msgr2=0x7f4b440f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:06:43.262 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:43.253+0000 7f4b4b559700 1 -- 172.21.15.155:0/4110629375 shutdown_connections 2024-02-10T09:06:43.262 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:43.253+0000 7f4b4b559700 1 -- 172.21.15.155:0/4110629375 wait complete. 2024-02-10T09:06:43.262 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:43.254+0000 7f4b4b559700 1 Processor -- start 2024-02-10T09:06:43.262 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:43.254+0000 7f4b4b559700 1 -- start start 2024-02-10T09:06:43.262 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:43.254+0000 7f4b4b559700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b440f6510 0x7f4b4406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:06:43.262 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:43.254+0000 7f4b4b559700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b44069e20 con 0x7f4b440f6510 2024-02-10T09:06:43.262 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:43.254+0000 7f4b492f5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b440f6510 0x7f4b4406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:06:43.262 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:43.254+0000 7f4b492f5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b440f6510 0x7f4b4406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:53478/0 (socket says 172.21.15.155:53478) 2024-02-10T09:06:43.262 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:43.254+0000 7f4b492f5700 1 -- 172.21.15.155:0/4267400555 learned_addr learned my addr 172.21.15.155:0/4267400555 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:06:43.262 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:43.255+0000 7f4b492f5700 1 -- 172.21.15.155:0/4267400555 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b34005040 con 0x7f4b440f6510 2024-02-10T09:06:43.262 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:43.255+0000 7f4b492f5700 1 --2- 172.21.15.155:0/4267400555 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b440f6510 0x7f4b4406b9a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f4b34005010 tx=0x7f4b3400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:06:43.262 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:43.255+0000 7f4b39ffb700 1 -- 172.21.15.155:0/4267400555 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b34009320 con 0x7f4b440f6510 2024-02-10T09:06:43.263 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:43.255+0000 7f4b4b559700 1 -- 172.21.15.155:0/4267400555 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b44069fc0 con 0x7f4b440f6510 2024-02-10T09:06:43.263 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:43.255+0000 7f4b39ffb700 1 -- 172.21.15.155:0/4267400555 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4b34013380 con 0x7f4b440f6510 2024-02-10T09:06:43.263 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:43.255+0000 7f4b39ffb700 1 -- 172.21.15.155:0/4267400555 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b34014e50 con 0x7f4b440f6510 2024-02-10T09:06:43.263 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:43.255+0000 7f4b4b559700 1 -- 172.21.15.155:0/4267400555 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b4406a450 con 0x7f4b440f6510 2024-02-10T09:06:43.263 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:43.256+0000 7f4b39ffb700 1 -- 172.21.15.155:0/4267400555 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4b340095f0 con 0x7f4b440f6510 2024-02-10T09:06:43.263 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:43.256+0000 7f4b4b559700 1 -- 172.21.15.155:0/4267400555 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b44059100 con 0x7f4b440f6510 2024-02-10T09:06:43.263 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:43.256+0000 7f4b39ffb700 1 --2- 172.21.15.155:0/4267400555 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f4b3002ff10 0x7f4b300323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:06:43.263 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:43.256+0000 7f4b39ffb700 1 -- 172.21.15.155:0/4267400555 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4b34040e70 con 0x7f4b440f6510 2024-02-10T09:06:43.264 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:43.256+0000 7f4b48af4700 1 --2- 172.21.15.155:0/4267400555 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f4b3002ff10 0x7f4b300323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:06:43.264 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:43.257+0000 7f4b48af4700 1 --2- 172.21.15.155:0/4267400555 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f4b3002ff10 0x7f4b300323d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f4b400096a0 tx=0x7f4b40006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:06:43.264 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:43.261+0000 7f4b39ffb700 1 -- 172.21.15.155:0/4267400555 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4b34014420 con 0x7f4b440f6510 2024-02-10T09:06:43.467 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:43.466+0000 7f4b4b559700 1 -- 172.21.15.155:0/4267400555 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4b440fc0a0 con 0x7f4b440f6510 2024-02-10T09:06:43.468 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:43.467+0000 7f4b39ffb700 1 -- 172.21.15.155:0/4267400555 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4b34010230 con 0x7f4b440f6510 2024-02-10T09:06:43.468 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:06:43.468 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:06:43.470 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:43.468+0000 7f4b4b559700 1 -- 172.21.15.155:0/4267400555 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f4b3002ff10 msgr2=0x7f4b300323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:06:43.470 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:43.469+0000 7f4b4b559700 1 --2- 172.21.15.155:0/4267400555 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f4b3002ff10 0x7f4b300323d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f4b400096a0 tx=0x7f4b40006b40).stop 2024-02-10T09:06:43.470 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:43.469+0000 7f4b4b559700 1 -- 172.21.15.155:0/4267400555 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b440f6510 msgr2=0x7f4b4406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:06:43.470 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:43.469+0000 7f4b4b559700 1 --2- 172.21.15.155:0/4267400555 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b440f6510 0x7f4b4406b9a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f4b34005010 tx=0x7f4b3400b040).stop 2024-02-10T09:06:43.470 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:43.469+0000 7f4b4b559700 1 -- 172.21.15.155:0/4267400555 shutdown_connections 2024-02-10T09:06:43.470 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:43.469+0000 7f4b4b559700 1 --2- 172.21.15.155:0/4267400555 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f4b3002ff10 0x7f4b300323d0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:06:43.470 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:43.469+0000 7f4b4b559700 1 --2- 172.21.15.155:0/4267400555 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4b440f6510 0x7f4b4406b9a0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:06:43.470 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:43.469+0000 7f4b4b559700 1 -- 172.21.15.155:0/4267400555 >> 172.21.15.155:0/4267400555 conn(0x7f4b440f1c70 msgr2=0x7f4b440f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:06:43.470 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:43.469+0000 7f4b4b559700 1 -- 172.21.15.155:0/4267400555 shutdown_connections 2024-02-10T09:06:43.470 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:43.469+0000 7f4b4b559700 1 -- 172.21.15.155:0/4267400555 wait complete. 2024-02-10T09:06:43.471 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:06:44.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:43 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:06:43.030330+0000 mgr.smithi077.dxntvw (mgr.14162) 246 : cluster [DBG] pgmap v232: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:06:44.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:43 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:06:43.468366+0000 mon.smithi077 (mon.0) 249 : audit [DBG] from='client.? 172.21.15.155:0/4267400555' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:06:45.279 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:06:45.280 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:06:45.398 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:45 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:06:45.030595+0000 mgr.smithi077.dxntvw (mgr.14162) 247 : cluster [DBG] pgmap v233: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:06:47.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:06:47.030975+0000 mgr.smithi077.dxntvw (mgr.14162) 248 : cluster [DBG] pgmap v234: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:06:47.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:06:47.096188+0000 mon.smithi077 (mon.0) 250 : audit [INF] from='mgr.14162 172.21.15.77:0/2029106299' entity='mgr.smithi077.dxntvw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi077.dxntvw/mirror_snapshot_schedule"}]: dispatch 2024-02-10T09:06:47.549 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:47.548+0000 7fb065805700 1 -- 172.21.15.155:0/53363835 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0600f5410 msgr2=0x7fb0600f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:06:47.549 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:47.548+0000 7fb065805700 1 --2- 172.21.15.155:0/53363835 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0600f5410 0x7fb0600f5830 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7fb054009fb0 tx=0x7fb054009b30).stop 2024-02-10T09:06:47.550 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:47.548+0000 7fb065805700 1 -- 172.21.15.155:0/53363835 shutdown_connections 2024-02-10T09:06:47.550 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:47.548+0000 7fb065805700 1 --2- 172.21.15.155:0/53363835 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0600f5410 0x7fb0600f5830 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:06:47.550 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:47.548+0000 7fb065805700 1 -- 172.21.15.155:0/53363835 >> 172.21.15.155:0/53363835 conn(0x7fb0600f0b70 msgr2=0x7fb0600f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:06:47.550 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:47.548+0000 7fb065805700 1 -- 172.21.15.155:0/53363835 shutdown_connections 2024-02-10T09:06:47.550 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:47.548+0000 7fb065805700 1 -- 172.21.15.155:0/53363835 wait complete. 2024-02-10T09:06:47.551 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:47.549+0000 7fb065805700 1 Processor -- start 2024-02-10T09:06:47.552 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:47.549+0000 7fb065805700 1 -- start start 2024-02-10T09:06:47.552 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:47.549+0000 7fb065805700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0600f5410 0x7fb0601094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:06:47.552 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:47.549+0000 7fb065805700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb060060240 con 0x7fb0600f5410 2024-02-10T09:06:47.552 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:47.549+0000 7fb05effd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0600f5410 0x7fb0601094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:06:47.552 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:47.549+0000 7fb05effd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0600f5410 0x7fb0601094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:53508/0 (socket says 172.21.15.155:53508) 2024-02-10T09:06:47.552 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:47.549+0000 7fb05effd700 1 -- 172.21.15.155:0/2587944700 learned_addr learned my addr 172.21.15.155:0/2587944700 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:06:47.552 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:47.550+0000 7fb05effd700 1 -- 172.21.15.155:0/2587944700 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb054005040 con 0x7fb0600f5410 2024-02-10T09:06:47.552 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:47.550+0000 7fb05effd700 1 --2- 172.21.15.155:0/2587944700 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0600f5410 0x7fb0601094e0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7fb054005010 tx=0x7fb05400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:06:47.552 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:47.550+0000 7fb0477fe700 1 -- 172.21.15.155:0/2587944700 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb054009320 con 0x7fb0600f5410 2024-02-10T09:06:47.553 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:47.550+0000 7fb065805700 1 -- 172.21.15.155:0/2587944700 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb060060440 con 0x7fb0600f5410 2024-02-10T09:06:47.553 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:47.550+0000 7fb0477fe700 1 -- 172.21.15.155:0/2587944700 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb054013380 con 0x7fb0600f5410 2024-02-10T09:06:47.553 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:47.550+0000 7fb0477fe700 1 -- 172.21.15.155:0/2587944700 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb054014e50 con 0x7fb0600f5410 2024-02-10T09:06:47.553 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:47.550+0000 7fb065805700 1 -- 172.21.15.155:0/2587944700 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb060109b90 con 0x7fb0600f5410 2024-02-10T09:06:47.554 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:47.551+0000 7fb0477fe700 1 -- 172.21.15.155:0/2587944700 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb0540095f0 con 0x7fb0600f5410 2024-02-10T09:06:47.554 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:47.551+0000 7fb065805700 1 -- 172.21.15.155:0/2587944700 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb060045ad0 con 0x7fb0600f5410 2024-02-10T09:06:47.554 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:47.551+0000 7fb0477fe700 1 --2- 172.21.15.155:0/2587944700 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb04802ff10 0x7fb0480323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:06:47.554 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:47.551+0000 7fb0477fe700 1 -- 172.21.15.155:0/2587944700 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb054040e90 con 0x7fb0600f5410 2024-02-10T09:06:47.555 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:47.551+0000 7fb05e7fc700 1 --2- 172.21.15.155:0/2587944700 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb04802ff10 0x7fb0480323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:06:47.555 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:47.552+0000 7fb05e7fc700 1 --2- 172.21.15.155:0/2587944700 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb04802ff10 0x7fb0480323d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fb0500096a0 tx=0x7fb050006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:06:47.557 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:47.556+0000 7fb0477fe700 1 -- 172.21.15.155:0/2587944700 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb054014420 con 0x7fb0600f5410 2024-02-10T09:06:47.762 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:47.761+0000 7fb065805700 1 -- 172.21.15.155:0/2587944700 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb06010a3b0 con 0x7fb0600f5410 2024-02-10T09:06:47.763 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:47.762+0000 7fb0477fe700 1 -- 172.21.15.155:0/2587944700 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb054010230 con 0x7fb0600f5410 2024-02-10T09:06:47.763 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:06:47.763 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:06:47.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:47.763+0000 7fb065805700 1 -- 172.21.15.155:0/2587944700 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb04802ff10 msgr2=0x7fb0480323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:06:47.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:47.763+0000 7fb065805700 1 --2- 172.21.15.155:0/2587944700 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb04802ff10 0x7fb0480323d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fb0500096a0 tx=0x7fb050006b40).stop 2024-02-10T09:06:47.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:47.763+0000 7fb065805700 1 -- 172.21.15.155:0/2587944700 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0600f5410 msgr2=0x7fb0601094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:06:47.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:47.763+0000 7fb065805700 1 --2- 172.21.15.155:0/2587944700 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0600f5410 0x7fb0601094e0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7fb054005010 tx=0x7fb05400b040).stop 2024-02-10T09:06:47.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:47.764+0000 7fb065805700 1 -- 172.21.15.155:0/2587944700 shutdown_connections 2024-02-10T09:06:47.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:47.764+0000 7fb065805700 1 --2- 172.21.15.155:0/2587944700 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb04802ff10 0x7fb0480323d0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:06:47.768 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:47.764+0000 7fb065805700 1 --2- 172.21.15.155:0/2587944700 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0600f5410 0x7fb0601094e0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:06:47.768 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:47.764+0000 7fb065805700 1 -- 172.21.15.155:0/2587944700 >> 172.21.15.155:0/2587944700 conn(0x7fb0600f0b70 msgr2=0x7fb0600f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:06:47.768 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:47.764+0000 7fb065805700 1 -- 172.21.15.155:0/2587944700 shutdown_connections 2024-02-10T09:06:47.768 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:47.764+0000 7fb065805700 1 -- 172.21.15.155:0/2587944700 wait complete. 2024-02-10T09:06:47.768 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:06:48.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:06:47.102248+0000 mon.smithi077 (mon.0) 251 : audit [INF] from='mgr.14162 172.21.15.77:0/2029106299' entity='mgr.smithi077.dxntvw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi077.dxntvw/trash_purge_schedule"}]: dispatch 2024-02-10T09:06:48.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:06:47.763578+0000 mon.smithi077 (mon.0) 252 : audit [DBG] from='client.? 172.21.15.155:0/2587944700' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:06:49.398 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:49 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:06:49.031312+0000 mgr.smithi077.dxntvw (mgr.14162) 249 : cluster [DBG] pgmap v235: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:06:49.691 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:06:49.691 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:06:51.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:51 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:06:51.031602+0000 mgr.smithi077.dxntvw (mgr.14162) 250 : cluster [DBG] pgmap v236: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:06:52.025 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:52.022+0000 7f0d5a813700 1 -- 172.21.15.155:0/3621325571 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d540f5410 msgr2=0x7f0d540f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:06:52.026 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:52.022+0000 7f0d5a813700 1 --2- 172.21.15.155:0/3621325571 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d540f5410 0x7f0d540f5830 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7f0d44009fb0 tx=0x7f0d44009b30).stop 2024-02-10T09:06:52.026 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:52.022+0000 7f0d5a813700 1 -- 172.21.15.155:0/3621325571 shutdown_connections 2024-02-10T09:06:52.026 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:52.022+0000 7f0d5a813700 1 --2- 172.21.15.155:0/3621325571 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d540f5410 0x7f0d540f5830 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:06:52.026 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:52.022+0000 7f0d5a813700 1 -- 172.21.15.155:0/3621325571 >> 172.21.15.155:0/3621325571 conn(0x7f0d540f0b70 msgr2=0x7f0d540f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:06:52.026 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:52.022+0000 7f0d5a813700 1 -- 172.21.15.155:0/3621325571 shutdown_connections 2024-02-10T09:06:52.026 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:52.022+0000 7f0d5a813700 1 -- 172.21.15.155:0/3621325571 wait complete. 2024-02-10T09:06:52.026 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:52.023+0000 7f0d5a813700 1 Processor -- start 2024-02-10T09:06:52.026 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:52.023+0000 7f0d5a813700 1 -- start start 2024-02-10T09:06:52.026 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:52.023+0000 7f0d5a813700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d540f5410 0x7f0d541089f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:06:52.026 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:52.023+0000 7f0d5a813700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d54108ee0 con 0x7f0d540f5410 2024-02-10T09:06:52.026 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:52.023+0000 7f0d53fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d540f5410 0x7f0d541089f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:06:52.026 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:52.023+0000 7f0d53fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d540f5410 0x7f0d541089f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:53528/0 (socket says 172.21.15.155:53528) 2024-02-10T09:06:52.026 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:52.023+0000 7f0d53fff700 1 -- 172.21.15.155:0/3873470162 learned_addr learned my addr 172.21.15.155:0/3873470162 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:06:52.027 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:52.024+0000 7f0d53fff700 1 -- 172.21.15.155:0/3873470162 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d44005040 con 0x7f0d540f5410 2024-02-10T09:06:52.027 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:52.024+0000 7f0d53fff700 1 --2- 172.21.15.155:0/3873470162 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d540f5410 0x7f0d541089f0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f0d44005010 tx=0x7f0d4400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:06:52.027 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:52.024+0000 7f0d50ff9700 1 -- 172.21.15.155:0/3873470162 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d44009320 con 0x7f0d540f5410 2024-02-10T09:06:52.027 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:52.024+0000 7f0d5a813700 1 -- 172.21.15.155:0/3873470162 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d541090e0 con 0x7f0d540f5410 2024-02-10T09:06:52.027 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:52.024+0000 7f0d50ff9700 1 -- 172.21.15.155:0/3873470162 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0d44013380 con 0x7f0d540f5410 2024-02-10T09:06:52.027 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:52.024+0000 7f0d50ff9700 1 -- 172.21.15.155:0/3873470162 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d44014e50 con 0x7f0d540f5410 2024-02-10T09:06:52.027 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:52.024+0000 7f0d5a813700 1 -- 172.21.15.155:0/3873470162 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d54109570 con 0x7f0d540f5410 2024-02-10T09:06:52.028 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:52.025+0000 7f0d50ff9700 1 -- 172.21.15.155:0/3873470162 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0d440263a0 con 0x7f0d540f5410 2024-02-10T09:06:52.028 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:52.025+0000 7f0d50ff9700 1 --2- 172.21.15.155:0/3873470162 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f0d3c02ff10 0x7f0d3c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:06:52.028 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:52.025+0000 7f0d50ff9700 1 -- 172.21.15.155:0/3873470162 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0d44040eb0 con 0x7f0d540f5410 2024-02-10T09:06:52.028 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:52.025+0000 7f0d5a813700 1 -- 172.21.15.155:0/3873470162 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d54045ad0 con 0x7f0d540f5410 2024-02-10T09:06:52.028 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:52.025+0000 7f0d537fe700 1 --2- 172.21.15.155:0/3873470162 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f0d3c02ff10 0x7f0d3c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:06:52.028 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:52.026+0000 7f0d537fe700 1 --2- 172.21.15.155:0/3873470162 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f0d3c02ff10 0x7f0d3c0323d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f0d480096a0 tx=0x7f0d48006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:06:52.031 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:52.030+0000 7f0d50ff9700 1 -- 172.21.15.155:0/3873470162 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0d44012920 con 0x7f0d540f5410 2024-02-10T09:06:52.234 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:52.233+0000 7f0d5a813700 1 -- 172.21.15.155:0/3873470162 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0d540fb090 con 0x7f0d540f5410 2024-02-10T09:06:52.234 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:52.233+0000 7f0d50ff9700 1 -- 172.21.15.155:0/3873470162 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0d4402b020 con 0x7f0d540f5410 2024-02-10T09:06:52.235 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:06:52.235 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:06:52.236 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:52.235+0000 7f0d5a813700 1 -- 172.21.15.155:0/3873470162 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f0d3c02ff10 msgr2=0x7f0d3c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:06:52.236 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:52.235+0000 7f0d5a813700 1 --2- 172.21.15.155:0/3873470162 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f0d3c02ff10 0x7f0d3c0323d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f0d480096a0 tx=0x7f0d48006b40).stop 2024-02-10T09:06:52.236 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:52.235+0000 7f0d5a813700 1 -- 172.21.15.155:0/3873470162 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d540f5410 msgr2=0x7f0d541089f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:06:52.236 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:52.235+0000 7f0d5a813700 1 --2- 172.21.15.155:0/3873470162 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d540f5410 0x7f0d541089f0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f0d44005010 tx=0x7f0d4400b040).stop 2024-02-10T09:06:52.236 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:52.235+0000 7f0d5a813700 1 -- 172.21.15.155:0/3873470162 shutdown_connections 2024-02-10T09:06:52.236 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:52.235+0000 7f0d5a813700 1 --2- 172.21.15.155:0/3873470162 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f0d3c02ff10 0x7f0d3c0323d0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:06:52.237 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:52.235+0000 7f0d5a813700 1 --2- 172.21.15.155:0/3873470162 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0d540f5410 0x7f0d541089f0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:06:52.237 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:52.235+0000 7f0d5a813700 1 -- 172.21.15.155:0/3873470162 >> 172.21.15.155:0/3873470162 conn(0x7f0d540f0b70 msgr2=0x7f0d5405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:06:52.237 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:52.235+0000 7f0d5a813700 1 -- 172.21.15.155:0/3873470162 shutdown_connections 2024-02-10T09:06:52.237 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:52.235+0000 7f0d5a813700 1 -- 172.21.15.155:0/3873470162 wait complete. 2024-02-10T09:06:52.237 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:06:52.649 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:52 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:06:52.234881+0000 mon.smithi077 (mon.0) 253 : audit [DBG] from='client.? 172.21.15.155:0/3873470162' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:06:54.060 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:06:54.060 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:06:54.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:53 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:06:53.031941+0000 mgr.smithi077.dxntvw (mgr.14162) 251 : cluster [DBG] pgmap v237: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:06:55.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:55 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:06:55.032209+0000 mgr.smithi077.dxntvw (mgr.14162) 252 : cluster [DBG] pgmap v238: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:06:56.155 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:56.151+0000 7f9c6d0e5700 1 -- 172.21.15.155:0/391646774 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c680f5410 msgr2=0x7f9c680f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:06:56.155 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:56.151+0000 7f9c6d0e5700 1 --2- 172.21.15.155:0/391646774 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c680f5410 0x7f9c680f5830 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f9c50009fb0 tx=0x7f9c50009b30).stop 2024-02-10T09:06:56.155 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:56.151+0000 7f9c6d0e5700 1 -- 172.21.15.155:0/391646774 shutdown_connections 2024-02-10T09:06:56.155 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:56.151+0000 7f9c6d0e5700 1 --2- 172.21.15.155:0/391646774 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c680f5410 0x7f9c680f5830 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:06:56.155 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:56.151+0000 7f9c6d0e5700 1 -- 172.21.15.155:0/391646774 >> 172.21.15.155:0/391646774 conn(0x7f9c680f0b70 msgr2=0x7f9c680f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:06:56.155 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:56.151+0000 7f9c6d0e5700 1 -- 172.21.15.155:0/391646774 shutdown_connections 2024-02-10T09:06:56.155 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:56.151+0000 7f9c6d0e5700 1 -- 172.21.15.155:0/391646774 wait complete. 2024-02-10T09:06:56.156 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:56.152+0000 7f9c6d0e5700 1 Processor -- start 2024-02-10T09:06:56.156 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:56.152+0000 7f9c6d0e5700 1 -- start start 2024-02-10T09:06:56.156 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:56.152+0000 7f9c6d0e5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c680f5410 0x7f9c681094a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:06:56.156 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:56.152+0000 7f9c6d0e5700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c68060280 con 0x7f9c680f5410 2024-02-10T09:06:56.156 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:56.152+0000 7f9c66d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c680f5410 0x7f9c681094a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:06:56.156 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:56.153+0000 7f9c66d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c680f5410 0x7f9c681094a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:55712/0 (socket says 172.21.15.155:55712) 2024-02-10T09:06:56.156 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:56.153+0000 7f9c66d9d700 1 -- 172.21.15.155:0/787325192 learned_addr learned my addr 172.21.15.155:0/787325192 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:06:56.156 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:56.153+0000 7f9c66d9d700 1 -- 172.21.15.155:0/787325192 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c50005040 con 0x7f9c680f5410 2024-02-10T09:06:56.156 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:56.153+0000 7f9c66d9d700 1 --2- 172.21.15.155:0/787325192 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c680f5410 0x7f9c681094a0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f9c50005010 tx=0x7f9c5000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:06:56.156 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:56.153+0000 7f9c5f7fe700 1 -- 172.21.15.155:0/787325192 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c50009320 con 0x7f9c680f5410 2024-02-10T09:06:56.156 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:56.153+0000 7f9c6d0e5700 1 -- 172.21.15.155:0/787325192 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c68060480 con 0x7f9c680f5410 2024-02-10T09:06:56.156 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:56.153+0000 7f9c5f7fe700 1 -- 172.21.15.155:0/787325192 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9c50013380 con 0x7f9c680f5410 2024-02-10T09:06:56.156 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:56.153+0000 7f9c5f7fe700 1 -- 172.21.15.155:0/787325192 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c50014e50 con 0x7f9c680f5410 2024-02-10T09:06:56.157 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:56.153+0000 7f9c6d0e5700 1 -- 172.21.15.155:0/787325192 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c68109bd0 con 0x7f9c680f5410 2024-02-10T09:06:56.157 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:56.154+0000 7f9c5f7fe700 1 -- 172.21.15.155:0/787325192 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9c500263a0 con 0x7f9c680f5410 2024-02-10T09:06:56.157 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:56.154+0000 7f9c5f7fe700 1 --2- 172.21.15.155:0/787325192 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f9c5402ff10 0x7f9c540323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:06:56.157 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:56.154+0000 7f9c6d0e5700 1 -- 172.21.15.155:0/787325192 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c68045ad0 con 0x7f9c680f5410 2024-02-10T09:06:56.157 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:56.154+0000 7f9c5f7fe700 1 -- 172.21.15.155:0/787325192 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9c50040fa0 con 0x7f9c680f5410 2024-02-10T09:06:56.157 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:56.154+0000 7f9c6659c700 1 --2- 172.21.15.155:0/787325192 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f9c5402ff10 0x7f9c540323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:06:56.157 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:56.155+0000 7f9c6659c700 1 --2- 172.21.15.155:0/787325192 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f9c5402ff10 0x7f9c540323d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f9c580096a0 tx=0x7f9c58006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:06:56.160 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:56.159+0000 7f9c5f7fe700 1 -- 172.21.15.155:0/787325192 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9c50012920 con 0x7f9c680f5410 2024-02-10T09:06:56.367 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:56.366+0000 7f9c6d0e5700 1 -- 172.21.15.155:0/787325192 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9c680fafa0 con 0x7f9c680f5410 2024-02-10T09:06:56.367 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:56.366+0000 7f9c5f7fe700 1 -- 172.21.15.155:0/787325192 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9c5002b020 con 0x7f9c680f5410 2024-02-10T09:06:56.371 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:06:56.371 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:06:56.372 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:56.368+0000 7f9c6d0e5700 1 -- 172.21.15.155:0/787325192 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f9c5402ff10 msgr2=0x7f9c540323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:06:56.372 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:56.368+0000 7f9c6d0e5700 1 --2- 172.21.15.155:0/787325192 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f9c5402ff10 0x7f9c540323d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f9c580096a0 tx=0x7f9c58006b40).stop 2024-02-10T09:06:56.372 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:56.368+0000 7f9c6d0e5700 1 -- 172.21.15.155:0/787325192 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c680f5410 msgr2=0x7f9c681094a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:06:56.372 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:56.368+0000 7f9c6d0e5700 1 --2- 172.21.15.155:0/787325192 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c680f5410 0x7f9c681094a0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f9c50005010 tx=0x7f9c5000b040).stop 2024-02-10T09:06:56.372 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:56.368+0000 7f9c6d0e5700 1 -- 172.21.15.155:0/787325192 shutdown_connections 2024-02-10T09:06:56.372 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:56.368+0000 7f9c6d0e5700 1 --2- 172.21.15.155:0/787325192 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f9c5402ff10 0x7f9c540323d0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:06:56.373 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:56.368+0000 7f9c6d0e5700 1 --2- 172.21.15.155:0/787325192 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c680f5410 0x7f9c681094a0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:06:56.373 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:56.368+0000 7f9c6d0e5700 1 -- 172.21.15.155:0/787325192 >> 172.21.15.155:0/787325192 conn(0x7f9c680f0b70 msgr2=0x7f9c680f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:06:56.373 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:56.368+0000 7f9c6d0e5700 1 -- 172.21.15.155:0/787325192 shutdown_connections 2024-02-10T09:06:56.373 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:06:56.368+0000 7f9c6d0e5700 1 -- 172.21.15.155:0/787325192 wait complete. 2024-02-10T09:06:56.373 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:06:56.899 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:56 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:06:56.367842+0000 mon.smithi077 (mon.0) 254 : audit [DBG] from='client.? 172.21.15.155:0/787325192' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:06:57.899 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:57 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:06:57.032514+0000 mgr.smithi077.dxntvw (mgr.14162) 253 : cluster [DBG] pgmap v239: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:06:58.174 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:06:58.175 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:06:59.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:06:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:06:59.032844+0000 mgr.smithi077.dxntvw (mgr.14162) 254 : cluster [DBG] pgmap v240: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:07:00.270 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:00.269+0000 7f07cd40c700 1 -- 172.21.15.155:0/1302958670 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07c80f6510 msgr2=0x7f07c80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:07:00.270 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:00.269+0000 7f07cd40c700 1 --2- 172.21.15.155:0/1302958670 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07c80f6510 0x7f07c80f6930 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7f07b0009fb0 tx=0x7f07b0009b30).stop 2024-02-10T09:07:00.270 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:00.269+0000 7f07cd40c700 1 -- 172.21.15.155:0/1302958670 shutdown_connections 2024-02-10T09:07:00.271 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:00.269+0000 7f07cd40c700 1 --2- 172.21.15.155:0/1302958670 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07c80f6510 0x7f07c80f6930 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:07:00.271 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:00.269+0000 7f07cd40c700 1 -- 172.21.15.155:0/1302958670 >> 172.21.15.155:0/1302958670 conn(0x7f07c80f1c70 msgr2=0x7f07c80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:07:00.271 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:00.269+0000 7f07cd40c700 1 -- 172.21.15.155:0/1302958670 shutdown_connections 2024-02-10T09:07:00.271 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:00.269+0000 7f07cd40c700 1 -- 172.21.15.155:0/1302958670 wait complete. 2024-02-10T09:07:00.275 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:00.269+0000 7f07cd40c700 1 Processor -- start 2024-02-10T09:07:00.276 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:00.269+0000 7f07cd40c700 1 -- start start 2024-02-10T09:07:00.276 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:00.270+0000 7f07cd40c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07c80f6510 0x7f07c81067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:07:00.276 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:00.270+0000 7f07cd40c700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07c8106cd0 con 0x7f07c80f6510 2024-02-10T09:07:00.276 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:00.270+0000 7f07c6ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07c80f6510 0x7f07c81067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:07:00.276 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:00.270+0000 7f07c6ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07c80f6510 0x7f07c81067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:55736/0 (socket says 172.21.15.155:55736) 2024-02-10T09:07:00.276 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:00.270+0000 7f07c6ffd700 1 -- 172.21.15.155:0/206391983 learned_addr learned my addr 172.21.15.155:0/206391983 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:07:00.277 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:00.270+0000 7f07c6ffd700 1 -- 172.21.15.155:0/206391983 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07b0005040 con 0x7f07c80f6510 2024-02-10T09:07:00.277 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:00.270+0000 7f07c6ffd700 1 --2- 172.21.15.155:0/206391983 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07c80f6510 0x7f07c81067e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f07b0005010 tx=0x7f07b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:07:00.277 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:00.271+0000 7f07bf7fe700 1 -- 172.21.15.155:0/206391983 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07b0009320 con 0x7f07c80f6510 2024-02-10T09:07:00.277 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:00.271+0000 7f07cd40c700 1 -- 172.21.15.155:0/206391983 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07c8106ed0 con 0x7f07c80f6510 2024-02-10T09:07:00.277 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:00.271+0000 7f07bf7fe700 1 -- 172.21.15.155:0/206391983 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f07b0013380 con 0x7f07c80f6510 2024-02-10T09:07:00.277 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:00.271+0000 7f07bf7fe700 1 -- 172.21.15.155:0/206391983 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07b0014e50 con 0x7f07c80f6510 2024-02-10T09:07:00.278 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:00.271+0000 7f07cd40c700 1 -- 172.21.15.155:0/206391983 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07c8107360 con 0x7f07c80f6510 2024-02-10T09:07:00.278 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:00.271+0000 7f07bf7fe700 1 -- 172.21.15.155:0/206391983 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f07b00095f0 con 0x7f07c80f6510 2024-02-10T09:07:00.278 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:00.271+0000 7f07cd40c700 1 -- 172.21.15.155:0/206391983 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07c8045ad0 con 0x7f07c80f6510 2024-02-10T09:07:00.278 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:00.272+0000 7f07bf7fe700 1 --2- 172.21.15.155:0/206391983 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f07b402ff10 0x7f07b40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:07:00.278 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:00.272+0000 7f07bf7fe700 1 -- 172.21.15.155:0/206391983 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f07b0040e90 con 0x7f07c80f6510 2024-02-10T09:07:00.278 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:00.272+0000 7f07c67fc700 1 --2- 172.21.15.155:0/206391983 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f07b402ff10 0x7f07b40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:07:00.278 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:00.272+0000 7f07c67fc700 1 --2- 172.21.15.155:0/206391983 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f07b402ff10 0x7f07b40323d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f07b80096a0 tx=0x7f07b8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:07:00.280 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:00.276+0000 7f07bf7fe700 1 -- 172.21.15.155:0/206391983 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f07b0014420 con 0x7f07c80f6510 2024-02-10T09:07:00.484 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:00.482+0000 7f07cd40c700 1 -- 172.21.15.155:0/206391983 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f07c8108210 con 0x7f07c80f6510 2024-02-10T09:07:00.484 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:00.483+0000 7f07bf7fe700 1 -- 172.21.15.155:0/206391983 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f07b0010230 con 0x7f07c80f6510 2024-02-10T09:07:00.484 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:07:00.485 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:07:00.486 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:00.485+0000 7f07cd40c700 1 -- 172.21.15.155:0/206391983 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f07b402ff10 msgr2=0x7f07b40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:07:00.486 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:00.485+0000 7f07cd40c700 1 --2- 172.21.15.155:0/206391983 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f07b402ff10 0x7f07b40323d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f07b80096a0 tx=0x7f07b8006b40).stop 2024-02-10T09:07:00.486 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:00.485+0000 7f07cd40c700 1 -- 172.21.15.155:0/206391983 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07c80f6510 msgr2=0x7f07c81067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:07:00.486 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:00.485+0000 7f07cd40c700 1 --2- 172.21.15.155:0/206391983 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07c80f6510 0x7f07c81067e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f07b0005010 tx=0x7f07b000b040).stop 2024-02-10T09:07:00.487 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:00.485+0000 7f07cd40c700 1 -- 172.21.15.155:0/206391983 shutdown_connections 2024-02-10T09:07:00.487 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:00.485+0000 7f07cd40c700 1 --2- 172.21.15.155:0/206391983 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f07b402ff10 0x7f07b40323d0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:07:00.487 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:00.485+0000 7f07cd40c700 1 --2- 172.21.15.155:0/206391983 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f07c80f6510 0x7f07c81067e0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:07:00.487 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:00.485+0000 7f07cd40c700 1 -- 172.21.15.155:0/206391983 >> 172.21.15.155:0/206391983 conn(0x7f07c80f1c70 msgr2=0x7f07c80f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:07:00.487 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:00.485+0000 7f07cd40c700 1 -- 172.21.15.155:0/206391983 shutdown_connections 2024-02-10T09:07:00.487 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:00.485+0000 7f07cd40c700 1 -- 172.21.15.155:0/206391983 wait complete. 2024-02-10T09:07:00.487 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:07:00.899 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:07:00 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:07:00.484689+0000 mon.smithi077 (mon.0) 255 : audit [DBG] from='client.? 172.21.15.155:0/206391983' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:07:01.899 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:07:01 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:07:01.033133+0000 mgr.smithi077.dxntvw (mgr.14162) 255 : cluster [DBG] pgmap v241: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:07:02.166 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:07:02.166 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:07:04.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:07:03 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:07:03.033388+0000 mgr.smithi077.dxntvw (mgr.14162) 256 : cluster [DBG] pgmap v242: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:07:04.306 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:04.275+0000 7f5a294ca700 1 -- 172.21.15.155:0/3117307770 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a240f51a0 msgr2=0x7f5a240f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:07:04.306 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:04.275+0000 7f5a294ca700 1 --2- 172.21.15.155:0/3117307770 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a240f51a0 0x7f5a240f55c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7f5a14009fb0 tx=0x7f5a14009b30).stop 2024-02-10T09:07:04.306 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:04.275+0000 7f5a294ca700 1 -- 172.21.15.155:0/3117307770 shutdown_connections 2024-02-10T09:07:04.306 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:04.275+0000 7f5a294ca700 1 --2- 172.21.15.155:0/3117307770 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a240f51a0 0x7f5a240f55c0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:07:04.306 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:04.275+0000 7f5a294ca700 1 -- 172.21.15.155:0/3117307770 >> 172.21.15.155:0/3117307770 conn(0x7f5a240f09a0 msgr2=0x7f5a240f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:07:04.306 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:04.275+0000 7f5a294ca700 1 -- 172.21.15.155:0/3117307770 shutdown_connections 2024-02-10T09:07:04.306 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:04.275+0000 7f5a294ca700 1 -- 172.21.15.155:0/3117307770 wait complete. 2024-02-10T09:07:04.306 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:04.275+0000 7f5a294ca700 1 Processor -- start 2024-02-10T09:07:04.307 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:04.276+0000 7f5a294ca700 1 -- start start 2024-02-10T09:07:04.307 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:04.276+0000 7f5a294ca700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a240f51a0 0x7f5a241054c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:07:04.307 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:04.276+0000 7f5a294ca700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a241059b0 con 0x7f5a240f51a0 2024-02-10T09:07:04.307 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:04.276+0000 7f5a23fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a240f51a0 0x7f5a241054c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:07:04.307 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:04.276+0000 7f5a23fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a240f51a0 0x7f5a241054c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:32832/0 (socket says 172.21.15.155:32832) 2024-02-10T09:07:04.307 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:04.276+0000 7f5a23fff700 1 -- 172.21.15.155:0/1283874607 learned_addr learned my addr 172.21.15.155:0/1283874607 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:07:04.307 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:04.276+0000 7f5a23fff700 1 -- 172.21.15.155:0/1283874607 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a14005040 con 0x7f5a240f51a0 2024-02-10T09:07:04.307 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:04.277+0000 7f5a23fff700 1 --2- 172.21.15.155:0/1283874607 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a240f51a0 0x7f5a241054c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f5a140089f0 tx=0x7f5a1400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:07:04.307 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:04.277+0000 7f5a20ff9700 1 -- 172.21.15.155:0/1283874607 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a14009320 con 0x7f5a240f51a0 2024-02-10T09:07:04.307 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:04.277+0000 7f5a294ca700 1 -- 172.21.15.155:0/1283874607 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a24105bb0 con 0x7f5a240f51a0 2024-02-10T09:07:04.307 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:04.277+0000 7f5a20ff9700 1 -- 172.21.15.155:0/1283874607 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5a14013380 con 0x7f5a240f51a0 2024-02-10T09:07:04.308 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:04.277+0000 7f5a20ff9700 1 -- 172.21.15.155:0/1283874607 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a14014e20 con 0x7f5a240f51a0 2024-02-10T09:07:04.308 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:04.277+0000 7f5a294ca700 1 -- 172.21.15.155:0/1283874607 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a24106040 con 0x7f5a240f51a0 2024-02-10T09:07:04.308 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:04.278+0000 7f5a20ff9700 1 -- 172.21.15.155:0/1283874607 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5a140095f0 con 0x7f5a240f51a0 2024-02-10T09:07:04.308 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:04.278+0000 7f5a294ca700 1 -- 172.21.15.155:0/1283874607 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a24045ad0 con 0x7f5a240f51a0 2024-02-10T09:07:04.308 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:04.278+0000 7f5a20ff9700 1 --2- 172.21.15.155:0/1283874607 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f5a0c02ff10 0x7f5a0c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:07:04.308 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:04.278+0000 7f5a20ff9700 1 -- 172.21.15.155:0/1283874607 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5a14040e90 con 0x7f5a240f51a0 2024-02-10T09:07:04.308 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:04.278+0000 7f5a237fe700 1 --2- 172.21.15.155:0/1283874607 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f5a0c02ff10 0x7f5a0c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:07:04.308 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:04.278+0000 7f5a237fe700 1 --2- 172.21.15.155:0/1283874607 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f5a0c02ff10 0x7f5a0c0323d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f5a180096a0 tx=0x7f5a18006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:07:04.309 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:04.283+0000 7f5a20ff9700 1 -- 172.21.15.155:0/1283874607 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5a14014420 con 0x7f5a240f51a0 2024-02-10T09:07:04.495 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:04.493+0000 7f5a294ca700 1 -- 172.21.15.155:0/1283874607 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5a24106ef0 con 0x7f5a240f51a0 2024-02-10T09:07:04.495 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:04.494+0000 7f5a20ff9700 1 -- 172.21.15.155:0/1283874607 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5a1401d980 con 0x7f5a240f51a0 2024-02-10T09:07:04.496 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:07:04.496 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:07:04.497 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:04.496+0000 7f5a294ca700 1 -- 172.21.15.155:0/1283874607 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f5a0c02ff10 msgr2=0x7f5a0c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:07:04.497 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:04.496+0000 7f5a294ca700 1 --2- 172.21.15.155:0/1283874607 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f5a0c02ff10 0x7f5a0c0323d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f5a180096a0 tx=0x7f5a18006b40).stop 2024-02-10T09:07:04.497 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:04.496+0000 7f5a294ca700 1 -- 172.21.15.155:0/1283874607 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a240f51a0 msgr2=0x7f5a241054c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:07:04.497 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:04.496+0000 7f5a294ca700 1 --2- 172.21.15.155:0/1283874607 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a240f51a0 0x7f5a241054c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f5a140089f0 tx=0x7f5a1400b040).stop 2024-02-10T09:07:04.497 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:04.496+0000 7f5a294ca700 1 -- 172.21.15.155:0/1283874607 shutdown_connections 2024-02-10T09:07:04.497 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:04.496+0000 7f5a294ca700 1 --2- 172.21.15.155:0/1283874607 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f5a0c02ff10 0x7f5a0c0323d0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:07:04.497 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:04.496+0000 7f5a294ca700 1 --2- 172.21.15.155:0/1283874607 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5a240f51a0 0x7f5a241054c0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:07:04.497 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:04.496+0000 7f5a294ca700 1 -- 172.21.15.155:0/1283874607 >> 172.21.15.155:0/1283874607 conn(0x7f5a240f09a0 msgr2=0x7f5a240f13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:07:04.498 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:04.496+0000 7f5a294ca700 1 -- 172.21.15.155:0/1283874607 shutdown_connections 2024-02-10T09:07:04.498 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:04.496+0000 7f5a294ca700 1 -- 172.21.15.155:0/1283874607 wait complete. 2024-02-10T09:07:04.498 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:07:05.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:07:04 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:07:04.495599+0000 mon.smithi077 (mon.0) 256 : audit [DBG] from='client.? 172.21.15.155:0/1283874607' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:07:06.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:07:05 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:07:05.033736+0000 mgr.smithi077.dxntvw (mgr.14162) 257 : cluster [DBG] pgmap v243: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:07:06.350 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:07:06.350 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:07:07.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:07:07 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:07:07.034018+0000 mgr.smithi077.dxntvw (mgr.14162) 258 : cluster [DBG] pgmap v244: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:07:08.449 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:08.425+0000 7f119ba87700 1 -- 172.21.15.155:0/2275552575 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11940f6510 msgr2=0x7f11940f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:07:08.449 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:08.425+0000 7f119ba87700 1 --2- 172.21.15.155:0/2275552575 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11940f6510 0x7f11940f6930 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7f1184009fb0 tx=0x7f1184009b30).stop 2024-02-10T09:07:08.449 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:08.425+0000 7f119ba87700 1 -- 172.21.15.155:0/2275552575 shutdown_connections 2024-02-10T09:07:08.449 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:08.425+0000 7f119ba87700 1 --2- 172.21.15.155:0/2275552575 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11940f6510 0x7f11940f6930 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:07:08.449 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:08.425+0000 7f119ba87700 1 -- 172.21.15.155:0/2275552575 >> 172.21.15.155:0/2275552575 conn(0x7f11940f1c70 msgr2=0x7f11940f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:07:08.449 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:08.425+0000 7f119ba87700 1 -- 172.21.15.155:0/2275552575 shutdown_connections 2024-02-10T09:07:08.449 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:08.425+0000 7f119ba87700 1 -- 172.21.15.155:0/2275552575 wait complete. 2024-02-10T09:07:08.449 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:08.426+0000 7f119ba87700 1 Processor -- start 2024-02-10T09:07:08.449 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:08.426+0000 7f119ba87700 1 -- start start 2024-02-10T09:07:08.449 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:08.426+0000 7f119ba87700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11940f6510 0x7f11941067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:07:08.449 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:08.426+0000 7f119ba87700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1194106cd0 con 0x7f11940f6510 2024-02-10T09:07:08.450 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:08.426+0000 7f1199823700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11940f6510 0x7f11941067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:07:08.450 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:08.426+0000 7f1199823700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11940f6510 0x7f11941067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:32844/0 (socket says 172.21.15.155:32844) 2024-02-10T09:07:08.450 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:08.426+0000 7f1199823700 1 -- 172.21.15.155:0/580964520 learned_addr learned my addr 172.21.15.155:0/580964520 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:07:08.450 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:08.426+0000 7f1199823700 1 -- 172.21.15.155:0/580964520 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1184005040 con 0x7f11940f6510 2024-02-10T09:07:08.450 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:08.427+0000 7f1199823700 1 --2- 172.21.15.155:0/580964520 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11940f6510 0x7f11941067e0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f1184005010 tx=0x7f118400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:07:08.450 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:08.427+0000 7f118a7fc700 1 -- 172.21.15.155:0/580964520 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1184009320 con 0x7f11940f6510 2024-02-10T09:07:08.450 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:08.427+0000 7f119ba87700 1 -- 172.21.15.155:0/580964520 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1194106ed0 con 0x7f11940f6510 2024-02-10T09:07:08.450 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:08.427+0000 7f118a7fc700 1 -- 172.21.15.155:0/580964520 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1184013380 con 0x7f11940f6510 2024-02-10T09:07:08.450 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:08.427+0000 7f118a7fc700 1 -- 172.21.15.155:0/580964520 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1184014e50 con 0x7f11940f6510 2024-02-10T09:07:08.450 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:08.427+0000 7f119ba87700 1 -- 172.21.15.155:0/580964520 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1194107360 con 0x7f11940f6510 2024-02-10T09:07:08.450 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:08.428+0000 7f118a7fc700 1 -- 172.21.15.155:0/580964520 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f11840095f0 con 0x7f11940f6510 2024-02-10T09:07:08.450 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:08.428+0000 7f119ba87700 1 -- 172.21.15.155:0/580964520 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1194045ad0 con 0x7f11940f6510 2024-02-10T09:07:08.451 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:08.428+0000 7f118a7fc700 1 --2- 172.21.15.155:0/580964520 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f118002ff10 0x7f11800323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:07:08.451 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:08.428+0000 7f118a7fc700 1 -- 172.21.15.155:0/580964520 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1184040e90 con 0x7f11940f6510 2024-02-10T09:07:08.451 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:08.428+0000 7f1199022700 1 --2- 172.21.15.155:0/580964520 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f118002ff10 0x7f11800323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:07:08.451 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:08.428+0000 7f1199022700 1 --2- 172.21.15.155:0/580964520 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f118002ff10 0x7f11800323d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f11900096a0 tx=0x7f1190006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:07:08.451 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:08.434+0000 7f118a7fc700 1 -- 172.21.15.155:0/580964520 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1184014420 con 0x7f11940f6510 2024-02-10T09:07:08.642 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:08.642+0000 7f119ba87700 1 -- 172.21.15.155:0/580964520 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1194108210 con 0x7f11940f6510 2024-02-10T09:07:08.643 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:08.643+0000 7f118a7fc700 1 -- 172.21.15.155:0/580964520 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1184010230 con 0x7f11940f6510 2024-02-10T09:07:08.643 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:07:08.643 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:07:08.645 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:08.644+0000 7f119ba87700 1 -- 172.21.15.155:0/580964520 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f118002ff10 msgr2=0x7f11800323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:07:08.645 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:08.644+0000 7f119ba87700 1 --2- 172.21.15.155:0/580964520 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f118002ff10 0x7f11800323d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f11900096a0 tx=0x7f1190006b40).stop 2024-02-10T09:07:08.645 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:08.644+0000 7f119ba87700 1 -- 172.21.15.155:0/580964520 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11940f6510 msgr2=0x7f11941067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:07:08.645 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:08.645+0000 7f119ba87700 1 --2- 172.21.15.155:0/580964520 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11940f6510 0x7f11941067e0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f1184005010 tx=0x7f118400b040).stop 2024-02-10T09:07:08.645 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:08.645+0000 7f119ba87700 1 -- 172.21.15.155:0/580964520 shutdown_connections 2024-02-10T09:07:08.645 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:08.645+0000 7f119ba87700 1 --2- 172.21.15.155:0/580964520 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f118002ff10 0x7f11800323d0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:07:08.645 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:08.645+0000 7f119ba87700 1 --2- 172.21.15.155:0/580964520 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f11940f6510 0x7f11941067e0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:07:08.645 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:08.645+0000 7f119ba87700 1 -- 172.21.15.155:0/580964520 >> 172.21.15.155:0/580964520 conn(0x7f11940f1c70 msgr2=0x7f11940f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:07:08.645 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:08.645+0000 7f119ba87700 1 -- 172.21.15.155:0/580964520 shutdown_connections 2024-02-10T09:07:08.645 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:08.645+0000 7f119ba87700 1 -- 172.21.15.155:0/580964520 wait complete. 2024-02-10T09:07:08.646 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:07:09.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:07:08 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:07:08.643300+0000 mon.smithi077 (mon.0) 257 : audit [DBG] from='client.? 172.21.15.155:0/580964520' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:07:10.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:07:09 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:07:09.034400+0000 mgr.smithi077.dxntvw (mgr.14162) 259 : cluster [DBG] pgmap v245: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:07:10.582 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:07:10.583 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:07:11.398 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:07:11 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:07:11.034778+0000 mgr.smithi077.dxntvw (mgr.14162) 260 : cluster [DBG] pgmap v246: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:07:12.818 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:12.816+0000 7f0c4d2db700 1 -- 172.21.15.155:0/4218085436 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c480f47c0 msgr2=0x7f0c480f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:07:12.818 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:12.816+0000 7f0c4d2db700 1 --2- 172.21.15.155:0/4218085436 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c480f47c0 0x7f0c480f4be0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f0c38009fb0 tx=0x7f0c38009b30).stop 2024-02-10T09:07:12.818 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:12.816+0000 7f0c4d2db700 1 -- 172.21.15.155:0/4218085436 shutdown_connections 2024-02-10T09:07:12.818 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:12.816+0000 7f0c4d2db700 1 --2- 172.21.15.155:0/4218085436 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c480f47c0 0x7f0c480f4be0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:07:12.818 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:12.816+0000 7f0c4d2db700 1 -- 172.21.15.155:0/4218085436 >> 172.21.15.155:0/4218085436 conn(0x7f0c480f02e0 msgr2=0x7f0c480f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:07:12.818 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:12.816+0000 7f0c4d2db700 1 -- 172.21.15.155:0/4218085436 shutdown_connections 2024-02-10T09:07:12.818 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:12.816+0000 7f0c4d2db700 1 -- 172.21.15.155:0/4218085436 wait complete. 2024-02-10T09:07:12.818 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:12.816+0000 7f0c4d2db700 1 Processor -- start 2024-02-10T09:07:12.818 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:12.817+0000 7f0c4d2db700 1 -- start start 2024-02-10T09:07:12.819 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:12.817+0000 7f0c4d2db700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c480f47c0 0x7f0c481067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:07:12.819 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:12.817+0000 7f0c4d2db700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c48106cc0 con 0x7f0c480f47c0 2024-02-10T09:07:12.819 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:12.817+0000 7f0c46ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c480f47c0 0x7f0c481067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:07:12.819 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:12.817+0000 7f0c46ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c480f47c0 0x7f0c481067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:36246/0 (socket says 172.21.15.155:36246) 2024-02-10T09:07:12.819 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:12.817+0000 7f0c46ffd700 1 -- 172.21.15.155:0/3919741891 learned_addr learned my addr 172.21.15.155:0/3919741891 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:07:12.819 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:12.817+0000 7f0c46ffd700 1 -- 172.21.15.155:0/3919741891 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c38005040 con 0x7f0c480f47c0 2024-02-10T09:07:12.819 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:12.817+0000 7f0c46ffd700 1 --2- 172.21.15.155:0/3919741891 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c480f47c0 0x7f0c481067d0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f0c38005010 tx=0x7f0c3800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:07:12.819 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:12.818+0000 7f0c2f7fe700 1 -- 172.21.15.155:0/3919741891 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c38009320 con 0x7f0c480f47c0 2024-02-10T09:07:12.819 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:12.818+0000 7f0c4d2db700 1 -- 172.21.15.155:0/3919741891 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c48106ec0 con 0x7f0c480f47c0 2024-02-10T09:07:12.819 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:12.818+0000 7f0c2f7fe700 1 -- 172.21.15.155:0/3919741891 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0c38013380 con 0x7f0c480f47c0 2024-02-10T09:07:12.820 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:12.818+0000 7f0c2f7fe700 1 -- 172.21.15.155:0/3919741891 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0c38014e50 con 0x7f0c480f47c0 2024-02-10T09:07:12.820 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:12.818+0000 7f0c4d2db700 1 -- 172.21.15.155:0/3919741891 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c48107350 con 0x7f0c480f47c0 2024-02-10T09:07:12.821 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:12.818+0000 7f0c2f7fe700 1 -- 172.21.15.155:0/3919741891 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0c380263a0 con 0x7f0c480f47c0 2024-02-10T09:07:12.821 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:12.819+0000 7f0c2f7fe700 1 --2- 172.21.15.155:0/3919741891 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f0c3002ff10 0x7f0c300323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:07:12.821 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:12.819+0000 7f0c4d2db700 1 -- 172.21.15.155:0/3919741891 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c48045ad0 con 0x7f0c480f47c0 2024-02-10T09:07:12.821 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:12.819+0000 7f0c2f7fe700 1 -- 172.21.15.155:0/3919741891 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0c38041140 con 0x7f0c480f47c0 2024-02-10T09:07:12.821 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:12.819+0000 7f0c467fc700 1 --2- 172.21.15.155:0/3919741891 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f0c3002ff10 0x7f0c300323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:07:12.821 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:12.819+0000 7f0c467fc700 1 --2- 172.21.15.155:0/3919741891 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f0c3002ff10 0x7f0c300323d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f0c3c0096a0 tx=0x7f0c3c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:07:12.823 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:12.823+0000 7f0c2f7fe700 1 -- 172.21.15.155:0/3919741891 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0c3801cb10 con 0x7f0c480f47c0 2024-02-10T09:07:13.029 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:13.029+0000 7f0c4d2db700 1 -- 172.21.15.155:0/3919741891 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0c481081a0 con 0x7f0c480f47c0 2024-02-10T09:07:13.030 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:13.029+0000 7f0c2f7fe700 1 -- 172.21.15.155:0/3919741891 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0c3801cb10 con 0x7f0c480f47c0 2024-02-10T09:07:13.030 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:07:13.030 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:07:13.032 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:13.031+0000 7f0c4d2db700 1 -- 172.21.15.155:0/3919741891 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f0c3002ff10 msgr2=0x7f0c300323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:07:13.032 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:13.031+0000 7f0c4d2db700 1 --2- 172.21.15.155:0/3919741891 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f0c3002ff10 0x7f0c300323d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f0c3c0096a0 tx=0x7f0c3c006b40).stop 2024-02-10T09:07:13.032 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:13.031+0000 7f0c4d2db700 1 -- 172.21.15.155:0/3919741891 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c480f47c0 msgr2=0x7f0c481067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:07:13.032 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:13.031+0000 7f0c4d2db700 1 --2- 172.21.15.155:0/3919741891 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c480f47c0 0x7f0c481067d0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f0c38005010 tx=0x7f0c3800b040).stop 2024-02-10T09:07:13.032 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:13.031+0000 7f0c4d2db700 1 -- 172.21.15.155:0/3919741891 shutdown_connections 2024-02-10T09:07:13.032 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:13.031+0000 7f0c4d2db700 1 --2- 172.21.15.155:0/3919741891 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f0c3002ff10 0x7f0c300323d0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:07:13.032 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:13.031+0000 7f0c4d2db700 1 --2- 172.21.15.155:0/3919741891 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0c480f47c0 0x7f0c481067d0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:07:13.033 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:13.031+0000 7f0c4d2db700 1 -- 172.21.15.155:0/3919741891 >> 172.21.15.155:0/3919741891 conn(0x7f0c480f02e0 msgr2=0x7f0c480f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:07:13.033 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:13.031+0000 7f0c4d2db700 1 -- 172.21.15.155:0/3919741891 shutdown_connections 2024-02-10T09:07:13.033 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:13.031+0000 7f0c4d2db700 1 -- 172.21.15.155:0/3919741891 wait complete. 2024-02-10T09:07:13.033 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:07:14.148 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:07:13 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:07:13.030135+0000 mon.smithi077 (mon.0) 258 : audit [DBG] from='client.? 172.21.15.155:0/3919741891' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:07:14.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:07:13 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:07:13.035166+0000 mgr.smithi077.dxntvw (mgr.14162) 261 : cluster [DBG] pgmap v247: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:07:14.851 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:07:14.851 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:07:15.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:07:15 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:07:15.035464+0000 mgr.smithi077.dxntvw (mgr.14162) 262 : cluster [DBG] pgmap v248: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:07:17.038 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:17.036+0000 7fdcd7df7700 1 -- 172.21.15.155:0/4201092057 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdcd00f5410 msgr2=0x7fdcd00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:07:17.038 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:17.036+0000 7fdcd7df7700 1 --2- 172.21.15.155:0/4201092057 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdcd00f5410 0x7fdcd00f5830 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7fdcc0009fb0 tx=0x7fdcc0009b30).stop 2024-02-10T09:07:17.038 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:17.036+0000 7fdcd7df7700 1 -- 172.21.15.155:0/4201092057 shutdown_connections 2024-02-10T09:07:17.038 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:17.036+0000 7fdcd7df7700 1 --2- 172.21.15.155:0/4201092057 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdcd00f5410 0x7fdcd00f5830 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:07:17.038 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:17.036+0000 7fdcd7df7700 1 -- 172.21.15.155:0/4201092057 >> 172.21.15.155:0/4201092057 conn(0x7fdcd00f0b70 msgr2=0x7fdcd00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:07:17.038 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:17.037+0000 7fdcd7df7700 1 -- 172.21.15.155:0/4201092057 shutdown_connections 2024-02-10T09:07:17.039 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:17.037+0000 7fdcd7df7700 1 -- 172.21.15.155:0/4201092057 wait complete. 2024-02-10T09:07:17.039 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:17.037+0000 7fdcd7df7700 1 Processor -- start 2024-02-10T09:07:17.039 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:17.037+0000 7fdcd7df7700 1 -- start start 2024-02-10T09:07:17.039 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:17.037+0000 7fdcd7df7700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdcd00f5410 0x7fdcd0105720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:07:17.039 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:17.037+0000 7fdcd7df7700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd0105c10 con 0x7fdcd00f5410 2024-02-10T09:07:17.039 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:17.037+0000 7fdcd5b93700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdcd00f5410 0x7fdcd0105720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:07:17.039 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:17.037+0000 7fdcd5b93700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdcd00f5410 0x7fdcd0105720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:36272/0 (socket says 172.21.15.155:36272) 2024-02-10T09:07:17.039 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:17.037+0000 7fdcd5b93700 1 -- 172.21.15.155:0/1713308744 learned_addr learned my addr 172.21.15.155:0/1713308744 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:07:17.039 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:17.038+0000 7fdcd5b93700 1 -- 172.21.15.155:0/1713308744 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcc0005040 con 0x7fdcd00f5410 2024-02-10T09:07:17.039 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:17.038+0000 7fdcd5b93700 1 --2- 172.21.15.155:0/1713308744 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdcd00f5410 0x7fdcd0105720 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7fdcc0005010 tx=0x7fdcc000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:07:17.039 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:17.038+0000 7fdcc67fc700 1 -- 172.21.15.155:0/1713308744 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdcc0009320 con 0x7fdcd00f5410 2024-02-10T09:07:17.039 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:17.038+0000 7fdcd7df7700 1 -- 172.21.15.155:0/1713308744 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcd0105e10 con 0x7fdcd00f5410 2024-02-10T09:07:17.039 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:17.038+0000 7fdcc67fc700 1 -- 172.21.15.155:0/1713308744 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdcc0013380 con 0x7fdcd00f5410 2024-02-10T09:07:17.039 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:17.038+0000 7fdcc67fc700 1 -- 172.21.15.155:0/1713308744 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdcc0014e50 con 0x7fdcd00f5410 2024-02-10T09:07:17.040 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:17.038+0000 7fdcd7df7700 1 -- 172.21.15.155:0/1713308744 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcd01062a0 con 0x7fdcd00f5410 2024-02-10T09:07:17.040 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:17.039+0000 7fdcc67fc700 1 -- 172.21.15.155:0/1713308744 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdcc00095f0 con 0x7fdcd00f5410 2024-02-10T09:07:17.040 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:17.039+0000 7fdcd7df7700 1 -- 172.21.15.155:0/1713308744 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcd0045ad0 con 0x7fdcd00f5410 2024-02-10T09:07:17.041 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:17.039+0000 7fdcc67fc700 1 --2- 172.21.15.155:0/1713308744 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fdcbc02ff10 0x7fdcbc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:07:17.041 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:17.039+0000 7fdcc67fc700 1 -- 172.21.15.155:0/1713308744 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdcc0040f60 con 0x7fdcd00f5410 2024-02-10T09:07:17.041 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:17.039+0000 7fdcd5392700 1 --2- 172.21.15.155:0/1713308744 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fdcbc02ff10 0x7fdcbc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:07:17.041 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:17.040+0000 7fdcd5392700 1 --2- 172.21.15.155:0/1713308744 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fdcbc02ff10 0x7fdcbc0323d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fdccc0096a0 tx=0x7fdccc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:07:17.043 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:17.044+0000 7fdcc67fc700 1 -- 172.21.15.155:0/1713308744 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdcc0014420 con 0x7fdcd00f5410 2024-02-10T09:07:17.252 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:17.252+0000 7fdcd7df7700 1 -- 172.21.15.155:0/1713308744 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdcd0107150 con 0x7fdcd00f5410 2024-02-10T09:07:17.252 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:17.252+0000 7fdcc67fc700 1 -- 172.21.15.155:0/1713308744 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdcc0010230 con 0x7fdcd00f5410 2024-02-10T09:07:17.252 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:07:17.252 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:07:17.254 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:17.254+0000 7fdcd7df7700 1 -- 172.21.15.155:0/1713308744 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fdcbc02ff10 msgr2=0x7fdcbc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:07:17.254 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:17.254+0000 7fdcd7df7700 1 --2- 172.21.15.155:0/1713308744 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fdcbc02ff10 0x7fdcbc0323d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fdccc0096a0 tx=0x7fdccc006b40).stop 2024-02-10T09:07:17.254 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:17.254+0000 7fdcd7df7700 1 -- 172.21.15.155:0/1713308744 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdcd00f5410 msgr2=0x7fdcd0105720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:07:17.254 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:17.254+0000 7fdcd7df7700 1 --2- 172.21.15.155:0/1713308744 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdcd00f5410 0x7fdcd0105720 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7fdcc0005010 tx=0x7fdcc000b040).stop 2024-02-10T09:07:17.254 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:17.254+0000 7fdcd7df7700 1 -- 172.21.15.155:0/1713308744 shutdown_connections 2024-02-10T09:07:17.254 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:17.254+0000 7fdcd7df7700 1 --2- 172.21.15.155:0/1713308744 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fdcbc02ff10 0x7fdcbc0323d0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:07:17.254 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:17.254+0000 7fdcd7df7700 1 --2- 172.21.15.155:0/1713308744 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdcd00f5410 0x7fdcd0105720 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:07:17.254 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:17.254+0000 7fdcd7df7700 1 -- 172.21.15.155:0/1713308744 >> 172.21.15.155:0/1713308744 conn(0x7fdcd00f0b70 msgr2=0x7fdcd00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:07:17.255 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:17.254+0000 7fdcd7df7700 1 -- 172.21.15.155:0/1713308744 shutdown_connections 2024-02-10T09:07:17.255 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:17.254+0000 7fdcd7df7700 1 -- 172.21.15.155:0/1713308744 wait complete. 2024-02-10T09:07:17.255 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:07:17.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:07:17 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:07:17.035693+0000 mgr.smithi077.dxntvw (mgr.14162) 263 : cluster [DBG] pgmap v249: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:07:18.398 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:07:18 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:07:17.252757+0000 mon.smithi077 (mon.0) 259 : audit [DBG] from='client.? 172.21.15.155:0/1713308744' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:07:19.102 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:07:19.103 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:07:19.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:07:19 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:07:19.035966+0000 mgr.smithi077.dxntvw (mgr.14162) 264 : cluster [DBG] pgmap v250: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:07:21.262 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:21.236+0000 7fbd9e5ec700 1 -- 172.21.15.155:0/2160994443 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd980f62a0 msgr2=0x7fbd980f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:07:21.262 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:21.236+0000 7fbd9e5ec700 1 --2- 172.21.15.155:0/2160994443 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd980f62a0 0x7fbd980f66c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7fbd94009fb0 tx=0x7fbd94009b30).stop 2024-02-10T09:07:21.262 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:21.236+0000 7fbd9e5ec700 1 -- 172.21.15.155:0/2160994443 shutdown_connections 2024-02-10T09:07:21.263 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:21.236+0000 7fbd9e5ec700 1 --2- 172.21.15.155:0/2160994443 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd980f62a0 0x7fbd980f66c0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:07:21.263 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:21.236+0000 7fbd9e5ec700 1 -- 172.21.15.155:0/2160994443 >> 172.21.15.155:0/2160994443 conn(0x7fbd980f1aa0 msgr2=0x7fbd980f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:07:21.263 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:21.236+0000 7fbd9e5ec700 1 -- 172.21.15.155:0/2160994443 shutdown_connections 2024-02-10T09:07:21.263 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:21.236+0000 7fbd9e5ec700 1 -- 172.21.15.155:0/2160994443 wait complete. 2024-02-10T09:07:21.263 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:21.236+0000 7fbd9e5ec700 1 Processor -- start 2024-02-10T09:07:21.263 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:21.236+0000 7fbd9e5ec700 1 -- start start 2024-02-10T09:07:21.263 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:21.236+0000 7fbd9e5ec700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd980f62a0 0x7fbd981065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:07:21.263 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:21.236+0000 7fbd9e5ec700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd98106ac0 con 0x7fbd980f62a0 2024-02-10T09:07:21.263 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:21.236+0000 7fbd9d5ea700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd980f62a0 0x7fbd981065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:07:21.263 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:21.237+0000 7fbd9d5ea700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd980f62a0 0x7fbd981065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:36278/0 (socket says 172.21.15.155:36278) 2024-02-10T09:07:21.263 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:21.237+0000 7fbd9d5ea700 1 -- 172.21.15.155:0/65737781 learned_addr learned my addr 172.21.15.155:0/65737781 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:07:21.263 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:21.237+0000 7fbd9d5ea700 1 -- 172.21.15.155:0/65737781 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd94005040 con 0x7fbd980f62a0 2024-02-10T09:07:21.263 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:21.237+0000 7fbd9d5ea700 1 --2- 172.21.15.155:0/65737781 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd980f62a0 0x7fbd981065d0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7fbd940089f0 tx=0x7fbd9400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:07:21.263 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:21.237+0000 7fbd8dffb700 1 -- 172.21.15.155:0/65737781 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd94009320 con 0x7fbd980f62a0 2024-02-10T09:07:21.264 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:21.237+0000 7fbd9e5ec700 1 -- 172.21.15.155:0/65737781 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd98106cc0 con 0x7fbd980f62a0 2024-02-10T09:07:21.264 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:21.237+0000 7fbd8dffb700 1 -- 172.21.15.155:0/65737781 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbd94013380 con 0x7fbd980f62a0 2024-02-10T09:07:21.264 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:21.237+0000 7fbd8dffb700 1 -- 172.21.15.155:0/65737781 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd94014e20 con 0x7fbd980f62a0 2024-02-10T09:07:21.264 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:21.237+0000 7fbd9e5ec700 1 -- 172.21.15.155:0/65737781 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd98107150 con 0x7fbd980f62a0 2024-02-10T09:07:21.264 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:21.238+0000 7fbd8dffb700 1 -- 172.21.15.155:0/65737781 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbd940095f0 con 0x7fbd980f62a0 2024-02-10T09:07:21.264 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:21.238+0000 7fbd9e5ec700 1 -- 172.21.15.155:0/65737781 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd98045ad0 con 0x7fbd980f62a0 2024-02-10T09:07:21.264 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:21.238+0000 7fbd8dffb700 1 --2- 172.21.15.155:0/65737781 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fbd8402ff10 0x7fbd840323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:07:21.264 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:21.238+0000 7fbd8dffb700 1 -- 172.21.15.155:0/65737781 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbd94040f80 con 0x7fbd980f62a0 2024-02-10T09:07:21.264 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:21.239+0000 7fbd9cde9700 1 --2- 172.21.15.155:0/65737781 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fbd8402ff10 0x7fbd840323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:07:21.264 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:21.239+0000 7fbd9cde9700 1 --2- 172.21.15.155:0/65737781 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fbd8402ff10 0x7fbd840323d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fbd880096a0 tx=0x7fbd88006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:07:21.264 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:21.243+0000 7fbd8dffb700 1 -- 172.21.15.155:0/65737781 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbd94014420 con 0x7fbd980f62a0 2024-02-10T09:07:21.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:07:21 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:07:21.036307+0000 mgr.smithi077.dxntvw (mgr.14162) 265 : cluster [DBG] pgmap v251: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:07:21.450 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:21.450+0000 7fbd9e5ec700 1 -- 172.21.15.155:0/65737781 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbd98107fa0 con 0x7fbd980f62a0 2024-02-10T09:07:21.451 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:21.451+0000 7fbd8dffb700 1 -- 172.21.15.155:0/65737781 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbd94010230 con 0x7fbd980f62a0 2024-02-10T09:07:21.451 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:07:21.451 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:07:21.453 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:21.452+0000 7fbd9e5ec700 1 -- 172.21.15.155:0/65737781 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fbd8402ff10 msgr2=0x7fbd840323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:07:21.453 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:21.452+0000 7fbd9e5ec700 1 --2- 172.21.15.155:0/65737781 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fbd8402ff10 0x7fbd840323d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fbd880096a0 tx=0x7fbd88006b40).stop 2024-02-10T09:07:21.453 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:21.452+0000 7fbd9e5ec700 1 -- 172.21.15.155:0/65737781 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd980f62a0 msgr2=0x7fbd981065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:07:21.453 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:21.452+0000 7fbd9e5ec700 1 --2- 172.21.15.155:0/65737781 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd980f62a0 0x7fbd981065d0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7fbd940089f0 tx=0x7fbd9400b040).stop 2024-02-10T09:07:21.453 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:21.452+0000 7fbd9e5ec700 1 -- 172.21.15.155:0/65737781 shutdown_connections 2024-02-10T09:07:21.453 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:21.452+0000 7fbd9e5ec700 1 --2- 172.21.15.155:0/65737781 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fbd8402ff10 0x7fbd840323d0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:07:21.453 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:21.452+0000 7fbd9e5ec700 1 --2- 172.21.15.155:0/65737781 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbd980f62a0 0x7fbd981065d0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:07:21.453 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:21.452+0000 7fbd9e5ec700 1 -- 172.21.15.155:0/65737781 >> 172.21.15.155:0/65737781 conn(0x7fbd980f1aa0 msgr2=0x7fbd980f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:07:21.453 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:21.452+0000 7fbd9e5ec700 1 -- 172.21.15.155:0/65737781 shutdown_connections 2024-02-10T09:07:21.453 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:21.452+0000 7fbd9e5ec700 1 -- 172.21.15.155:0/65737781 wait complete. 2024-02-10T09:07:21.453 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:07:22.398 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:07:22 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:07:21.451209+0000 mon.smithi077 (mon.0) 260 : audit [DBG] from='client.? 172.21.15.155:0/65737781' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:07:23.264 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:07:23.264 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:07:24.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:07:23 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:07:23.036592+0000 mgr.smithi077.dxntvw (mgr.14162) 266 : cluster [DBG] pgmap v252: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:07:25.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:07:25 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:07:25.036921+0000 mgr.smithi077.dxntvw (mgr.14162) 267 : cluster [DBG] pgmap v253: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:07:25.422 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:25.420+0000 7f2120554700 1 -- 172.21.15.155:0/2719197558 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21180f6510 msgr2=0x7f21180f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:07:25.422 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:25.420+0000 7f2120554700 1 --2- 172.21.15.155:0/2719197558 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21180f6510 0x7f21180f6930 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7f2108009fb0 tx=0x7f2108009b30).stop 2024-02-10T09:07:25.422 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:25.420+0000 7f2120554700 1 -- 172.21.15.155:0/2719197558 shutdown_connections 2024-02-10T09:07:25.422 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:25.420+0000 7f2120554700 1 --2- 172.21.15.155:0/2719197558 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21180f6510 0x7f21180f6930 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:07:25.422 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:25.420+0000 7f2120554700 1 -- 172.21.15.155:0/2719197558 >> 172.21.15.155:0/2719197558 conn(0x7f21180f1c70 msgr2=0x7f21180f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:07:25.422 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:25.420+0000 7f2120554700 1 -- 172.21.15.155:0/2719197558 shutdown_connections 2024-02-10T09:07:25.422 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:25.420+0000 7f2120554700 1 -- 172.21.15.155:0/2719197558 wait complete. 2024-02-10T09:07:25.422 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:25.421+0000 7f2120554700 1 Processor -- start 2024-02-10T09:07:25.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:25.421+0000 7f2120554700 1 -- start start 2024-02-10T09:07:25.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:25.421+0000 7f2120554700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21180f6510 0x7f21181067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:07:25.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:25.421+0000 7f2120554700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2118106cd0 con 0x7f21180f6510 2024-02-10T09:07:25.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:25.421+0000 7f211e2f0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21180f6510 0x7f21181067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:07:25.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:25.421+0000 7f211e2f0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21180f6510 0x7f21181067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:43132/0 (socket says 172.21.15.155:43132) 2024-02-10T09:07:25.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:25.421+0000 7f211e2f0700 1 -- 172.21.15.155:0/1316147648 learned_addr learned my addr 172.21.15.155:0/1316147648 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:07:25.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:25.421+0000 7f211e2f0700 1 -- 172.21.15.155:0/1316147648 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2108005040 con 0x7f21180f6510 2024-02-10T09:07:25.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:25.422+0000 7f211e2f0700 1 --2- 172.21.15.155:0/1316147648 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21180f6510 0x7f21181067e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f2108005010 tx=0x7f210800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:07:25.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:25.422+0000 7f210effd700 1 -- 172.21.15.155:0/1316147648 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2108009320 con 0x7f21180f6510 2024-02-10T09:07:25.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:25.422+0000 7f210effd700 1 -- 172.21.15.155:0/1316147648 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2108013380 con 0x7f21180f6510 2024-02-10T09:07:25.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:25.422+0000 7f2120554700 1 -- 172.21.15.155:0/1316147648 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2118106ed0 con 0x7f21180f6510 2024-02-10T09:07:25.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:25.422+0000 7f210effd700 1 -- 172.21.15.155:0/1316147648 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2108014e50 con 0x7f21180f6510 2024-02-10T09:07:25.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:25.422+0000 7f2120554700 1 -- 172.21.15.155:0/1316147648 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2118107360 con 0x7f21180f6510 2024-02-10T09:07:25.425 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:25.422+0000 7f210effd700 1 -- 172.21.15.155:0/1316147648 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2108014420 con 0x7f21180f6510 2024-02-10T09:07:25.425 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:25.423+0000 7f2120554700 1 -- 172.21.15.155:0/1316147648 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2118045ad0 con 0x7f21180f6510 2024-02-10T09:07:25.425 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:25.423+0000 7f210effd700 1 --2- 172.21.15.155:0/1316147648 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f210402ff10 0x7f21040323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:07:25.425 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:25.423+0000 7f210effd700 1 -- 172.21.15.155:0/1316147648 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2108040ee0 con 0x7f21180f6510 2024-02-10T09:07:25.425 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:25.423+0000 7f211daef700 1 --2- 172.21.15.155:0/1316147648 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f210402ff10 0x7f21040323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:07:25.425 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:25.423+0000 7f211daef700 1 --2- 172.21.15.155:0/1316147648 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f210402ff10 0x7f21040323d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f21140096a0 tx=0x7f2114006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:07:25.427 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:25.428+0000 7f210effd700 1 -- 172.21.15.155:0/1316147648 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f21080095f0 con 0x7f21180f6510 2024-02-10T09:07:25.634 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:25.634+0000 7f2120554700 1 -- 172.21.15.155:0/1316147648 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2118108210 con 0x7f21180f6510 2024-02-10T09:07:25.634 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:25.634+0000 7f210effd700 1 -- 172.21.15.155:0/1316147648 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f210801db70 con 0x7f21180f6510 2024-02-10T09:07:25.635 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:07:25.635 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:07:25.636 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:25.636+0000 7f2120554700 1 -- 172.21.15.155:0/1316147648 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f210402ff10 msgr2=0x7f21040323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:07:25.636 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:25.636+0000 7f2120554700 1 --2- 172.21.15.155:0/1316147648 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f210402ff10 0x7f21040323d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f21140096a0 tx=0x7f2114006b40).stop 2024-02-10T09:07:25.636 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:25.636+0000 7f2120554700 1 -- 172.21.15.155:0/1316147648 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21180f6510 msgr2=0x7f21181067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:07:25.636 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:25.636+0000 7f2120554700 1 --2- 172.21.15.155:0/1316147648 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21180f6510 0x7f21181067e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f2108005010 tx=0x7f210800b040).stop 2024-02-10T09:07:25.636 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:25.636+0000 7f2120554700 1 -- 172.21.15.155:0/1316147648 shutdown_connections 2024-02-10T09:07:25.636 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:25.636+0000 7f2120554700 1 --2- 172.21.15.155:0/1316147648 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f210402ff10 0x7f21040323d0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:07:25.636 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:25.636+0000 7f2120554700 1 --2- 172.21.15.155:0/1316147648 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f21180f6510 0x7f21181067e0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:07:25.636 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:25.636+0000 7f2120554700 1 -- 172.21.15.155:0/1316147648 >> 172.21.15.155:0/1316147648 conn(0x7f21180f1c70 msgr2=0x7f21180f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:07:25.637 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:25.636+0000 7f2120554700 1 -- 172.21.15.155:0/1316147648 shutdown_connections 2024-02-10T09:07:25.637 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:25.636+0000 7f2120554700 1 -- 172.21.15.155:0/1316147648 wait complete. 2024-02-10T09:07:25.637 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:07:26.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:07:26 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:07:25.635005+0000 mon.smithi077 (mon.0) 261 : audit [DBG] from='client.? 172.21.15.155:0/1316147648' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:07:27.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:07:27 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:07:27.037311+0000 mgr.smithi077.dxntvw (mgr.14162) 268 : cluster [DBG] pgmap v254: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:07:27.407 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:07:27.407 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:07:29.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:07:29 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:07:29.037619+0000 mgr.smithi077.dxntvw (mgr.14162) 269 : cluster [DBG] pgmap v255: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:07:29.499 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:29.497+0000 7efcc5936700 1 -- 172.21.15.155:0/407379124 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efcc00f5410 msgr2=0x7efcc00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:07:29.499 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:29.497+0000 7efcc5936700 1 --2- 172.21.15.155:0/407379124 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efcc00f5410 0x7efcc00f5830 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7efcb0009fb0 tx=0x7efcb0009b30).stop 2024-02-10T09:07:29.499 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:29.497+0000 7efcc5936700 1 -- 172.21.15.155:0/407379124 shutdown_connections 2024-02-10T09:07:29.499 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:29.497+0000 7efcc5936700 1 --2- 172.21.15.155:0/407379124 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efcc00f5410 0x7efcc00f5830 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:07:29.499 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:29.497+0000 7efcc5936700 1 -- 172.21.15.155:0/407379124 >> 172.21.15.155:0/407379124 conn(0x7efcc00f0b70 msgr2=0x7efcc00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:07:29.499 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:29.497+0000 7efcc5936700 1 -- 172.21.15.155:0/407379124 shutdown_connections 2024-02-10T09:07:29.499 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:29.497+0000 7efcc5936700 1 -- 172.21.15.155:0/407379124 wait complete. 2024-02-10T09:07:29.501 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:29.498+0000 7efcc5936700 1 Processor -- start 2024-02-10T09:07:29.502 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:29.498+0000 7efcc5936700 1 -- start start 2024-02-10T09:07:29.502 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:29.498+0000 7efcc5936700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efcc00f5410 0x7efcc0105720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:07:29.502 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:29.498+0000 7efcc5936700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcc0105c10 con 0x7efcc00f5410 2024-02-10T09:07:29.502 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:29.499+0000 7efcbeffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efcc00f5410 0x7efcc0105720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:07:29.502 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:29.499+0000 7efcbeffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efcc00f5410 0x7efcc0105720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:43150/0 (socket says 172.21.15.155:43150) 2024-02-10T09:07:29.502 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:29.499+0000 7efcbeffd700 1 -- 172.21.15.155:0/2459292167 learned_addr learned my addr 172.21.15.155:0/2459292167 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:07:29.502 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:29.499+0000 7efcbeffd700 1 -- 172.21.15.155:0/2459292167 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcb0005040 con 0x7efcc00f5410 2024-02-10T09:07:29.502 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:29.499+0000 7efcbeffd700 1 --2- 172.21.15.155:0/2459292167 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efcc00f5410 0x7efcc0105720 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7efcb0005010 tx=0x7efcb000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:07:29.502 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:29.499+0000 7efca77fe700 1 -- 172.21.15.155:0/2459292167 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efcb0009320 con 0x7efcc00f5410 2024-02-10T09:07:29.502 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:29.499+0000 7efcc5936700 1 -- 172.21.15.155:0/2459292167 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcc0105e10 con 0x7efcc00f5410 2024-02-10T09:07:29.502 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:29.499+0000 7efca77fe700 1 -- 172.21.15.155:0/2459292167 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efcb0013380 con 0x7efcc00f5410 2024-02-10T09:07:29.502 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:29.499+0000 7efca77fe700 1 -- 172.21.15.155:0/2459292167 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efcb0014e50 con 0x7efcc00f5410 2024-02-10T09:07:29.502 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:29.499+0000 7efcc5936700 1 -- 172.21.15.155:0/2459292167 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcc01062a0 con 0x7efcc00f5410 2024-02-10T09:07:29.503 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:29.500+0000 7efca77fe700 1 -- 172.21.15.155:0/2459292167 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efcb00263a0 con 0x7efcc00f5410 2024-02-10T09:07:29.503 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:29.500+0000 7efca77fe700 1 --2- 172.21.15.155:0/2459292167 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7efca802ff10 0x7efca80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:07:29.503 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:29.500+0000 7efcc5936700 1 -- 172.21.15.155:0/2459292167 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcc0045ad0 con 0x7efcc00f5410 2024-02-10T09:07:29.503 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:29.500+0000 7efca77fe700 1 -- 172.21.15.155:0/2459292167 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efcb0040e90 con 0x7efcc00f5410 2024-02-10T09:07:29.503 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:29.501+0000 7efcbe7fc700 1 --2- 172.21.15.155:0/2459292167 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7efca802ff10 0x7efca80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:07:29.503 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:29.501+0000 7efcbe7fc700 1 --2- 172.21.15.155:0/2459292167 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7efca802ff10 0x7efca80323d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7efcb40096a0 tx=0x7efcb4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:07:29.506 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:29.505+0000 7efca77fe700 1 -- 172.21.15.155:0/2459292167 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efcb0012920 con 0x7efcc00f5410 2024-02-10T09:07:29.713 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:29.712+0000 7efcc5936700 1 -- 172.21.15.155:0/2459292167 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efcc01070f0 con 0x7efcc00f5410 2024-02-10T09:07:29.713 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:29.713+0000 7efca77fe700 1 -- 172.21.15.155:0/2459292167 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7efcb002c090 con 0x7efcc00f5410 2024-02-10T09:07:29.714 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:07:29.714 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:07:29.715 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:29.714+0000 7efcc5936700 1 -- 172.21.15.155:0/2459292167 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7efca802ff10 msgr2=0x7efca80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:07:29.716 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:29.714+0000 7efcc5936700 1 --2- 172.21.15.155:0/2459292167 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7efca802ff10 0x7efca80323d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7efcb40096a0 tx=0x7efcb4006b40).stop 2024-02-10T09:07:29.716 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:29.714+0000 7efcc5936700 1 -- 172.21.15.155:0/2459292167 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efcc00f5410 msgr2=0x7efcc0105720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:07:29.716 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:29.714+0000 7efcc5936700 1 --2- 172.21.15.155:0/2459292167 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efcc00f5410 0x7efcc0105720 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7efcb0005010 tx=0x7efcb000b040).stop 2024-02-10T09:07:29.716 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:29.714+0000 7efcc5936700 1 -- 172.21.15.155:0/2459292167 shutdown_connections 2024-02-10T09:07:29.716 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:29.714+0000 7efcc5936700 1 --2- 172.21.15.155:0/2459292167 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7efca802ff10 0x7efca80323d0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:07:29.716 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:29.714+0000 7efcc5936700 1 --2- 172.21.15.155:0/2459292167 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efcc00f5410 0x7efcc0105720 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:07:29.716 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:29.714+0000 7efcc5936700 1 -- 172.21.15.155:0/2459292167 >> 172.21.15.155:0/2459292167 conn(0x7efcc00f0b70 msgr2=0x7efcc00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:07:29.716 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:29.714+0000 7efcc5936700 1 -- 172.21.15.155:0/2459292167 shutdown_connections 2024-02-10T09:07:29.716 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:29.714+0000 7efcc5936700 1 -- 172.21.15.155:0/2459292167 wait complete. 2024-02-10T09:07:29.717 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:07:30.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:07:30 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:07:29.714178+0000 mon.smithi077 (mon.0) 262 : audit [DBG] from='client.? 172.21.15.155:0/2459292167' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:07:31.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:07:31 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:07:31.037904+0000 mgr.smithi077.dxntvw (mgr.14162) 270 : cluster [DBG] pgmap v256: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:07:31.573 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:07:31.573 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:07:33.829 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:33.827+0000 7f7494aaa700 1 -- 172.21.15.155:0/4167876845 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74900f47c0 msgr2=0x7f74900f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:07:33.829 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:33.827+0000 7f7494aaa700 1 --2- 172.21.15.155:0/4167876845 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74900f47c0 0x7f74900f4be0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f7478009fb0 tx=0x7f7478009b30).stop 2024-02-10T09:07:33.829 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:33.827+0000 7f7494aaa700 1 -- 172.21.15.155:0/4167876845 shutdown_connections 2024-02-10T09:07:33.829 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:33.827+0000 7f7494aaa700 1 --2- 172.21.15.155:0/4167876845 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74900f47c0 0x7f74900f4be0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:07:33.829 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:33.827+0000 7f7494aaa700 1 -- 172.21.15.155:0/4167876845 >> 172.21.15.155:0/4167876845 conn(0x7f74900f02e0 msgr2=0x7f74900f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:07:33.829 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:33.828+0000 7f7494aaa700 1 -- 172.21.15.155:0/4167876845 shutdown_connections 2024-02-10T09:07:33.829 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:33.828+0000 7f7494aaa700 1 -- 172.21.15.155:0/4167876845 wait complete. 2024-02-10T09:07:33.831 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:33.828+0000 7f7494aaa700 1 Processor -- start 2024-02-10T09:07:33.831 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:33.828+0000 7f7494aaa700 1 -- start start 2024-02-10T09:07:33.831 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:33.828+0000 7f7494aaa700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74900f47c0 0x7f74901089e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:07:33.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:33.828+0000 7f7494aaa700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7490108ed0 con 0x7f74900f47c0 2024-02-10T09:07:33.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:33.828+0000 7f748e59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74900f47c0 0x7f74901089e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:07:33.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:33.828+0000 7f748e59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74900f47c0 0x7f74901089e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:55258/0 (socket says 172.21.15.155:55258) 2024-02-10T09:07:33.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:33.828+0000 7f748e59c700 1 -- 172.21.15.155:0/527762164 learned_addr learned my addr 172.21.15.155:0/527762164 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:07:33.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:33.829+0000 7f748e59c700 1 -- 172.21.15.155:0/527762164 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7478005040 con 0x7f74900f47c0 2024-02-10T09:07:33.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:33.829+0000 7f748e59c700 1 --2- 172.21.15.155:0/527762164 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74900f47c0 0x7f74901089e0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f7478005010 tx=0x7f747800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:07:33.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:33.829+0000 7f7486ffd700 1 -- 172.21.15.155:0/527762164 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7478009320 con 0x7f74900f47c0 2024-02-10T09:07:33.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:33.829+0000 7f7486ffd700 1 -- 172.21.15.155:0/527762164 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7478013380 con 0x7f74900f47c0 2024-02-10T09:07:33.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:33.829+0000 7f7494aaa700 1 -- 172.21.15.155:0/527762164 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74901090d0 con 0x7f74900f47c0 2024-02-10T09:07:33.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:33.829+0000 7f7486ffd700 1 -- 172.21.15.155:0/527762164 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7478014e50 con 0x7f74900f47c0 2024-02-10T09:07:33.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:33.829+0000 7f7494aaa700 1 -- 172.21.15.155:0/527762164 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7490109560 con 0x7f74900f47c0 2024-02-10T09:07:33.833 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:33.830+0000 7f7486ffd700 1 -- 172.21.15.155:0/527762164 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7478014420 con 0x7f74900f47c0 2024-02-10T09:07:33.833 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:33.830+0000 7f7494aaa700 1 -- 172.21.15.155:0/527762164 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7490045ad0 con 0x7f74900f47c0 2024-02-10T09:07:33.833 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:33.830+0000 7f7486ffd700 1 --2- 172.21.15.155:0/527762164 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f747c02ff10 0x7f747c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:07:33.833 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:33.830+0000 7f7486ffd700 1 -- 172.21.15.155:0/527762164 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7478040f40 con 0x7f74900f47c0 2024-02-10T09:07:33.833 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:33.830+0000 7f748dd9b700 1 --2- 172.21.15.155:0/527762164 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f747c02ff10 0x7f747c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:07:33.833 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:33.831+0000 7f748dd9b700 1 --2- 172.21.15.155:0/527762164 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f747c02ff10 0x7f747c0323d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f74800096a0 tx=0x7f7480006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:07:33.835 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:33.834+0000 7f7486ffd700 1 -- 172.21.15.155:0/527762164 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f74780095f0 con 0x7f74900f47c0 2024-02-10T09:07:34.039 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:34.038+0000 7f7494aaa700 1 -- 172.21.15.155:0/527762164 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f749010a410 con 0x7f74900f47c0 2024-02-10T09:07:34.040 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:34.039+0000 7f7486ffd700 1 -- 172.21.15.155:0/527762164 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f747801db70 con 0x7f74900f47c0 2024-02-10T09:07:34.040 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:07:34.040 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:07:34.042 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:34.040+0000 7f7494aaa700 1 -- 172.21.15.155:0/527762164 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f747c02ff10 msgr2=0x7f747c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:07:34.042 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:34.040+0000 7f7494aaa700 1 --2- 172.21.15.155:0/527762164 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f747c02ff10 0x7f747c0323d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f74800096a0 tx=0x7f7480006b40).stop 2024-02-10T09:07:34.042 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:34.040+0000 7f7494aaa700 1 -- 172.21.15.155:0/527762164 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74900f47c0 msgr2=0x7f74901089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:07:34.042 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:34.040+0000 7f7494aaa700 1 --2- 172.21.15.155:0/527762164 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74900f47c0 0x7f74901089e0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f7478005010 tx=0x7f747800b040).stop 2024-02-10T09:07:34.042 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:34.040+0000 7f7494aaa700 1 -- 172.21.15.155:0/527762164 shutdown_connections 2024-02-10T09:07:34.042 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:34.040+0000 7f7494aaa700 1 --2- 172.21.15.155:0/527762164 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f747c02ff10 0x7f747c0323d0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:07:34.042 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:34.040+0000 7f7494aaa700 1 --2- 172.21.15.155:0/527762164 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f74900f47c0 0x7f74901089e0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:07:34.042 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:34.040+0000 7f7494aaa700 1 -- 172.21.15.155:0/527762164 >> 172.21.15.155:0/527762164 conn(0x7f74900f02e0 msgr2=0x7f74900f0d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:07:34.042 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:34.040+0000 7f7494aaa700 1 -- 172.21.15.155:0/527762164 shutdown_connections 2024-02-10T09:07:34.042 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:34.040+0000 7f7494aaa700 1 -- 172.21.15.155:0/527762164 wait complete. 2024-02-10T09:07:34.043 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:07:34.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:07:33 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:07:33.038292+0000 mgr.smithi077.dxntvw (mgr.14162) 271 : cluster [DBG] pgmap v257: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:07:35.152 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:07:34 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:07:34.040522+0000 mon.smithi077 (mon.0) 263 : audit [DBG] from='client.? 172.21.15.155:0/527762164' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:07:35.859 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:07:35.859 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:07:36.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:07:35 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:07:35.038626+0000 mgr.smithi077.dxntvw (mgr.14162) 272 : cluster [DBG] pgmap v258: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:07:37.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:07:37 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:07:37.038949+0000 mgr.smithi077.dxntvw (mgr.14162) 273 : cluster [DBG] pgmap v259: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:07:38.098 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:38.095+0000 7f656ff60700 1 -- 172.21.15.155:0/3903727790 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6568060220 msgr2=0x7f65680f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:07:38.098 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:38.095+0000 7f656ff60700 1 --2- 172.21.15.155:0/3903727790 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6568060220 0x7f65680f9150 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f6558009fb0 tx=0x7f6558009b30).stop 2024-02-10T09:07:38.099 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:38.095+0000 7f656ff60700 1 -- 172.21.15.155:0/3903727790 shutdown_connections 2024-02-10T09:07:38.099 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:38.095+0000 7f656ff60700 1 --2- 172.21.15.155:0/3903727790 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6568060220 0x7f65680f9150 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:07:38.099 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:38.095+0000 7f656ff60700 1 -- 172.21.15.155:0/3903727790 >> 172.21.15.155:0/3903727790 conn(0x7f65680f0b70 msgr2=0x7f65680f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:07:38.099 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:38.095+0000 7f656ff60700 1 -- 172.21.15.155:0/3903727790 shutdown_connections 2024-02-10T09:07:38.099 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:38.095+0000 7f656ff60700 1 -- 172.21.15.155:0/3903727790 wait complete. 2024-02-10T09:07:38.099 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:38.096+0000 7f656ff60700 1 Processor -- start 2024-02-10T09:07:38.099 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:38.096+0000 7f656ff60700 1 -- start start 2024-02-10T09:07:38.099 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:38.096+0000 7f656ff60700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6568060220 0x7f65680f6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:07:38.099 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:38.096+0000 7f656ff60700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65680f5050 con 0x7f6568060220 2024-02-10T09:07:38.099 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:38.096+0000 7f656dcfc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6568060220 0x7f65680f6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:07:38.100 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:38.096+0000 7f656dcfc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6568060220 0x7f65680f6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:55284/0 (socket says 172.21.15.155:55284) 2024-02-10T09:07:38.100 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:38.096+0000 7f656dcfc700 1 -- 172.21.15.155:0/2653039430 learned_addr learned my addr 172.21.15.155:0/2653039430 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:07:38.100 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:38.097+0000 7f656dcfc700 1 -- 172.21.15.155:0/2653039430 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6558005040 con 0x7f6568060220 2024-02-10T09:07:38.100 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:38.097+0000 7f656dcfc700 1 --2- 172.21.15.155:0/2653039430 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6568060220 0x7f65680f6bd0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f65580089f0 tx=0x7f655800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:07:38.100 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:38.097+0000 7f655e7fc700 1 -- 172.21.15.155:0/2653039430 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6558009320 con 0x7f6568060220 2024-02-10T09:07:38.100 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:38.097+0000 7f656ff60700 1 -- 172.21.15.155:0/2653039430 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65680f51f0 con 0x7f6568060220 2024-02-10T09:07:38.100 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:38.097+0000 7f655e7fc700 1 -- 172.21.15.155:0/2653039430 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6558013380 con 0x7f6568060220 2024-02-10T09:07:38.100 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:38.097+0000 7f655e7fc700 1 -- 172.21.15.155:0/2653039430 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6558014e20 con 0x7f6568060220 2024-02-10T09:07:38.100 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:38.097+0000 7f656ff60700 1 -- 172.21.15.155:0/2653039430 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65680f5680 con 0x7f6568060220 2024-02-10T09:07:38.101 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:38.098+0000 7f656ff60700 1 -- 172.21.15.155:0/2653039430 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6568059100 con 0x7f6568060220 2024-02-10T09:07:38.102 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:38.098+0000 7f655e7fc700 1 -- 172.21.15.155:0/2653039430 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f65580095f0 con 0x7f6568060220 2024-02-10T09:07:38.102 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:38.098+0000 7f655e7fc700 1 --2- 172.21.15.155:0/2653039430 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f655402ff10 0x7f65540323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:07:38.102 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:38.098+0000 7f655e7fc700 1 -- 172.21.15.155:0/2653039430 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6558040e90 con 0x7f6568060220 2024-02-10T09:07:38.102 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:38.098+0000 7f656d4fb700 1 --2- 172.21.15.155:0/2653039430 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f655402ff10 0x7f65540323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:07:38.102 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:38.099+0000 7f656d4fb700 1 --2- 172.21.15.155:0/2653039430 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f655402ff10 0x7f65540323d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f65640096a0 tx=0x7f6564006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:07:38.104 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:38.103+0000 7f655e7fc700 1 -- 172.21.15.155:0/2653039430 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6558014420 con 0x7f6568060220 2024-02-10T09:07:38.311 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:38.309+0000 7f656ff60700 1 -- 172.21.15.155:0/2653039430 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f65680fc090 con 0x7f6568060220 2024-02-10T09:07:38.311 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:38.309+0000 7f655e7fc700 1 -- 172.21.15.155:0/2653039430 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f655801d980 con 0x7f6568060220 2024-02-10T09:07:38.312 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:07:38.312 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:07:38.313 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:38.311+0000 7f656ff60700 1 -- 172.21.15.155:0/2653039430 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f655402ff10 msgr2=0x7f65540323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:07:38.314 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:38.311+0000 7f656ff60700 1 --2- 172.21.15.155:0/2653039430 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f655402ff10 0x7f65540323d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f65640096a0 tx=0x7f6564006b40).stop 2024-02-10T09:07:38.314 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:38.311+0000 7f656ff60700 1 -- 172.21.15.155:0/2653039430 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6568060220 msgr2=0x7f65680f6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:07:38.314 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:38.311+0000 7f656ff60700 1 --2- 172.21.15.155:0/2653039430 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6568060220 0x7f65680f6bd0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f65580089f0 tx=0x7f655800b040).stop 2024-02-10T09:07:38.314 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:38.311+0000 7f656ff60700 1 -- 172.21.15.155:0/2653039430 shutdown_connections 2024-02-10T09:07:38.314 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:38.311+0000 7f656ff60700 1 --2- 172.21.15.155:0/2653039430 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f655402ff10 0x7f65540323d0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:07:38.314 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:38.311+0000 7f656ff60700 1 --2- 172.21.15.155:0/2653039430 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6568060220 0x7f65680f6bd0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:07:38.314 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:38.311+0000 7f656ff60700 1 -- 172.21.15.155:0/2653039430 >> 172.21.15.155:0/2653039430 conn(0x7f65680f0b70 msgr2=0x7f65680f1620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:07:38.315 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:38.311+0000 7f656ff60700 1 -- 172.21.15.155:0/2653039430 shutdown_connections 2024-02-10T09:07:38.315 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:38.311+0000 7f656ff60700 1 -- 172.21.15.155:0/2653039430 wait complete. 2024-02-10T09:07:38.315 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:07:39.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:07:38 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:07:38.311171+0000 mon.smithi077 (mon.0) 264 : audit [DBG] from='client.? 172.21.15.155:0/2653039430' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:07:40.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:07:39 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:07:39.039296+0000 mgr.smithi077.dxntvw (mgr.14162) 274 : cluster [DBG] pgmap v260: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:07:40.208 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:07:40.208 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:07:41.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:07:41 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:07:41.039652+0000 mgr.smithi077.dxntvw (mgr.14162) 275 : cluster [DBG] pgmap v261: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:07:42.481 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:42.480+0000 7fe4e9de7700 1 -- 172.21.15.155:0/3898979240 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4e406b8a0 msgr2=0x7fe4e4069e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:07:42.482 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:42.480+0000 7fe4e9de7700 1 --2- 172.21.15.155:0/3898979240 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4e406b8a0 0x7fe4e4069e40 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7fe4d8009fb0 tx=0x7fe4d8009b30).stop 2024-02-10T09:07:42.482 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:42.480+0000 7fe4e9de7700 1 -- 172.21.15.155:0/3898979240 shutdown_connections 2024-02-10T09:07:42.482 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:42.480+0000 7fe4e9de7700 1 --2- 172.21.15.155:0/3898979240 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4e406b8a0 0x7fe4e4069e40 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:07:42.482 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:42.480+0000 7fe4e9de7700 1 -- 172.21.15.155:0/3898979240 >> 172.21.15.155:0/3898979240 conn(0x7fe4e40f1aa0 msgr2=0x7fe4e40f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:07:42.482 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:42.480+0000 7fe4e9de7700 1 -- 172.21.15.155:0/3898979240 shutdown_connections 2024-02-10T09:07:42.482 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:42.480+0000 7fe4e9de7700 1 -- 172.21.15.155:0/3898979240 wait complete. 2024-02-10T09:07:42.482 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:42.480+0000 7fe4e9de7700 1 Processor -- start 2024-02-10T09:07:42.482 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:42.480+0000 7fe4e9de7700 1 -- start start 2024-02-10T09:07:42.484 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:42.481+0000 7fe4e9de7700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4e406b8a0 0x7fe4e410cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:07:42.484 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:42.481+0000 7fe4e9de7700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4e410d0d0 con 0x7fe4e406b8a0 2024-02-10T09:07:42.484 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:42.481+0000 7fe4e8de5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4e406b8a0 0x7fe4e410cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:07:42.484 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:42.481+0000 7fe4e8de5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4e406b8a0 0x7fe4e410cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:48736/0 (socket says 172.21.15.155:48736) 2024-02-10T09:07:42.484 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:42.481+0000 7fe4e8de5700 1 -- 172.21.15.155:0/924900926 learned_addr learned my addr 172.21.15.155:0/924900926 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:07:42.484 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:42.481+0000 7fe4e8de5700 1 -- 172.21.15.155:0/924900926 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4d8005040 con 0x7fe4e406b8a0 2024-02-10T09:07:42.485 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:42.482+0000 7fe4e8de5700 1 --2- 172.21.15.155:0/924900926 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4e406b8a0 0x7fe4e410cbe0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7fe4d8005010 tx=0x7fe4d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:07:42.485 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:42.482+0000 7fe4e17fa700 1 -- 172.21.15.155:0/924900926 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe4d8009320 con 0x7fe4e406b8a0 2024-02-10T09:07:42.485 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:42.482+0000 7fe4e9de7700 1 -- 172.21.15.155:0/924900926 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4e410d2d0 con 0x7fe4e406b8a0 2024-02-10T09:07:42.485 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:42.482+0000 7fe4e17fa700 1 -- 172.21.15.155:0/924900926 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe4d8013380 con 0x7fe4e406b8a0 2024-02-10T09:07:42.485 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:42.482+0000 7fe4e17fa700 1 -- 172.21.15.155:0/924900926 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe4d8014e50 con 0x7fe4e406b8a0 2024-02-10T09:07:42.485 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:42.482+0000 7fe4e9de7700 1 -- 172.21.15.155:0/924900926 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4e410d760 con 0x7fe4e406b8a0 2024-02-10T09:07:42.485 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:42.483+0000 7fe4e17fa700 1 -- 172.21.15.155:0/924900926 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe4d80095f0 con 0x7fe4e406b8a0 2024-02-10T09:07:42.485 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:42.483+0000 7fe4e17fa700 1 --2- 172.21.15.155:0/924900926 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fe4d402ff10 0x7fe4d40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:07:42.485 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:42.483+0000 7fe4e9de7700 1 -- 172.21.15.155:0/924900926 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4e4045ad0 con 0x7fe4e406b8a0 2024-02-10T09:07:42.485 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:42.483+0000 7fe4e17fa700 1 -- 172.21.15.155:0/924900926 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe4d8040e90 con 0x7fe4e406b8a0 2024-02-10T09:07:42.485 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:42.483+0000 7fe4e3fff700 1 --2- 172.21.15.155:0/924900926 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fe4d402ff10 0x7fe4d40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:07:42.486 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:42.484+0000 7fe4e3fff700 1 --2- 172.21.15.155:0/924900926 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fe4d402ff10 0x7fe4d40323d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fe4d00096a0 tx=0x7fe4d0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:07:42.489 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:42.488+0000 7fe4e17fa700 1 -- 172.21.15.155:0/924900926 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe4d801d990 con 0x7fe4e406b8a0 2024-02-10T09:07:42.693 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:42.692+0000 7fe4e9de7700 1 -- 172.21.15.155:0/924900926 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe4e40f71f0 con 0x7fe4e406b8a0 2024-02-10T09:07:42.694 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:42.693+0000 7fe4e17fa700 1 -- 172.21.15.155:0/924900926 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe4d80077b0 con 0x7fe4e406b8a0 2024-02-10T09:07:42.694 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:07:42.694 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:07:42.696 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:42.695+0000 7fe4e9de7700 1 -- 172.21.15.155:0/924900926 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fe4d402ff10 msgr2=0x7fe4d40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:07:42.696 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:42.695+0000 7fe4e9de7700 1 --2- 172.21.15.155:0/924900926 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fe4d402ff10 0x7fe4d40323d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fe4d00096a0 tx=0x7fe4d0006b40).stop 2024-02-10T09:07:42.696 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:42.695+0000 7fe4e9de7700 1 -- 172.21.15.155:0/924900926 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4e406b8a0 msgr2=0x7fe4e410cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:07:42.696 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:42.695+0000 7fe4e9de7700 1 --2- 172.21.15.155:0/924900926 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4e406b8a0 0x7fe4e410cbe0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7fe4d8005010 tx=0x7fe4d800b040).stop 2024-02-10T09:07:42.696 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:42.695+0000 7fe4e9de7700 1 -- 172.21.15.155:0/924900926 shutdown_connections 2024-02-10T09:07:42.696 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:42.695+0000 7fe4e9de7700 1 --2- 172.21.15.155:0/924900926 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fe4d402ff10 0x7fe4d40323d0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:07:42.696 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:42.695+0000 7fe4e9de7700 1 --2- 172.21.15.155:0/924900926 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe4e406b8a0 0x7fe4e410cbe0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:07:42.696 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:42.695+0000 7fe4e9de7700 1 -- 172.21.15.155:0/924900926 >> 172.21.15.155:0/924900926 conn(0x7fe4e40f1aa0 msgr2=0x7fe4e40f82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:07:42.696 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:42.695+0000 7fe4e9de7700 1 -- 172.21.15.155:0/924900926 shutdown_connections 2024-02-10T09:07:42.697 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:42.695+0000 7fe4e9de7700 1 -- 172.21.15.155:0/924900926 wait complete. 2024-02-10T09:07:42.697 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:07:43.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:07:42 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:07:42.694405+0000 mon.smithi077 (mon.0) 265 : audit [DBG] from='client.? 172.21.15.155:0/924900926' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:07:44.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:07:43 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:07:43.040030+0000 mgr.smithi077.dxntvw (mgr.14162) 276 : cluster [DBG] pgmap v262: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:07:44.580 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:07:44.581 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:07:45.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:07:45 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:07:45.040381+0000 mgr.smithi077.dxntvw (mgr.14162) 277 : cluster [DBG] pgmap v263: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:07:46.472 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:46.468+0000 7ff2087b0700 1 -- 172.21.15.155:0/1306051015 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2000f5410 msgr2=0x7ff2000f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:07:46.472 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:46.468+0000 7ff2087b0700 1 --2- 172.21.15.155:0/1306051015 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2000f5410 0x7ff2000f5830 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7ff1f0009fb0 tx=0x7ff1f0009b30).stop 2024-02-10T09:07:46.472 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:46.468+0000 7ff2087b0700 1 -- 172.21.15.155:0/1306051015 shutdown_connections 2024-02-10T09:07:46.472 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:46.468+0000 7ff2087b0700 1 --2- 172.21.15.155:0/1306051015 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2000f5410 0x7ff2000f5830 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:07:46.472 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:46.468+0000 7ff2087b0700 1 -- 172.21.15.155:0/1306051015 >> 172.21.15.155:0/1306051015 conn(0x7ff2000f0b70 msgr2=0x7ff2000f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:07:46.472 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:46.468+0000 7ff2087b0700 1 -- 172.21.15.155:0/1306051015 shutdown_connections 2024-02-10T09:07:46.472 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:46.468+0000 7ff2087b0700 1 -- 172.21.15.155:0/1306051015 wait complete. 2024-02-10T09:07:46.472 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:46.468+0000 7ff2087b0700 1 Processor -- start 2024-02-10T09:07:46.473 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:46.469+0000 7ff2087b0700 1 -- start start 2024-02-10T09:07:46.473 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:46.469+0000 7ff2087b0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2000f5410 0x7ff2000fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:07:46.473 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:46.469+0000 7ff2087b0700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2000fb960 con 0x7ff2000f5410 2024-02-10T09:07:46.473 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:46.469+0000 7ff20654c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2000f5410 0x7ff2000fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:07:46.473 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:46.469+0000 7ff20654c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2000f5410 0x7ff2000fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:48756/0 (socket says 172.21.15.155:48756) 2024-02-10T09:07:46.473 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:46.469+0000 7ff20654c700 1 -- 172.21.15.155:0/3132305145 learned_addr learned my addr 172.21.15.155:0/3132305145 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:07:46.473 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:46.469+0000 7ff20654c700 1 -- 172.21.15.155:0/3132305145 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1f0005040 con 0x7ff2000f5410 2024-02-10T09:07:46.473 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:46.469+0000 7ff20654c700 1 --2- 172.21.15.155:0/3132305145 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2000f5410 0x7ff2000fd4e0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7ff1f0005010 tx=0x7ff1f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:07:46.473 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:46.470+0000 7ff1f6ffd700 1 -- 172.21.15.155:0/3132305145 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff1f0009320 con 0x7ff2000f5410 2024-02-10T09:07:46.473 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:46.470+0000 7ff2087b0700 1 -- 172.21.15.155:0/3132305145 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2000fbb00 con 0x7ff2000f5410 2024-02-10T09:07:46.473 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:46.470+0000 7ff1f6ffd700 1 -- 172.21.15.155:0/3132305145 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff1f0013380 con 0x7ff2000f5410 2024-02-10T09:07:46.473 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:46.470+0000 7ff1f6ffd700 1 -- 172.21.15.155:0/3132305145 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff1f0014e50 con 0x7ff2000f5410 2024-02-10T09:07:46.473 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:46.470+0000 7ff2087b0700 1 -- 172.21.15.155:0/3132305145 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2000fbf90 con 0x7ff2000f5410 2024-02-10T09:07:46.473 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:46.470+0000 7ff1f6ffd700 1 -- 172.21.15.155:0/3132305145 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff1f00263a0 con 0x7ff2000f5410 2024-02-10T09:07:46.474 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:46.471+0000 7ff1f6ffd700 1 --2- 172.21.15.155:0/3132305145 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7ff1ec02ff10 0x7ff1ec0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:07:46.474 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:46.471+0000 7ff2087b0700 1 -- 172.21.15.155:0/3132305145 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff200059100 con 0x7ff2000f5410 2024-02-10T09:07:46.474 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:46.471+0000 7ff1f6ffd700 1 -- 172.21.15.155:0/3132305145 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff1f0040ec0 con 0x7ff2000f5410 2024-02-10T09:07:46.474 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:46.471+0000 7ff205d4b700 1 --2- 172.21.15.155:0/3132305145 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7ff1ec02ff10 0x7ff1ec0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:07:46.474 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:46.471+0000 7ff205d4b700 1 --2- 172.21.15.155:0/3132305145 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7ff1ec02ff10 0x7ff1ec0323d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7ff1fc0096a0 tx=0x7ff1fc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:07:46.476 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:46.476+0000 7ff1f6ffd700 1 -- 172.21.15.155:0/3132305145 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff1f0012920 con 0x7ff2000f5410 2024-02-10T09:07:46.684 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:46.683+0000 7ff2087b0700 1 -- 172.21.15.155:0/3132305145 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff2000fafa0 con 0x7ff2000f5410 2024-02-10T09:07:46.684 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:46.683+0000 7ff1f6ffd700 1 -- 172.21.15.155:0/3132305145 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff1f0012330 con 0x7ff2000f5410 2024-02-10T09:07:46.685 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:07:46.685 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:07:46.686 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:46.685+0000 7ff2087b0700 1 -- 172.21.15.155:0/3132305145 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7ff1ec02ff10 msgr2=0x7ff1ec0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:07:46.686 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:46.685+0000 7ff2087b0700 1 --2- 172.21.15.155:0/3132305145 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7ff1ec02ff10 0x7ff1ec0323d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7ff1fc0096a0 tx=0x7ff1fc006b40).stop 2024-02-10T09:07:46.686 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:46.685+0000 7ff2087b0700 1 -- 172.21.15.155:0/3132305145 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2000f5410 msgr2=0x7ff2000fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:07:46.686 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:46.685+0000 7ff2087b0700 1 --2- 172.21.15.155:0/3132305145 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2000f5410 0x7ff2000fd4e0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7ff1f0005010 tx=0x7ff1f000b040).stop 2024-02-10T09:07:46.686 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:46.685+0000 7ff2087b0700 1 -- 172.21.15.155:0/3132305145 shutdown_connections 2024-02-10T09:07:46.686 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:46.685+0000 7ff2087b0700 1 --2- 172.21.15.155:0/3132305145 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7ff1ec02ff10 0x7ff1ec0323d0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:07:46.686 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:46.685+0000 7ff2087b0700 1 --2- 172.21.15.155:0/3132305145 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff2000f5410 0x7ff2000fd4e0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:07:46.686 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:46.685+0000 7ff2087b0700 1 -- 172.21.15.155:0/3132305145 >> 172.21.15.155:0/3132305145 conn(0x7ff2000f0b70 msgr2=0x7ff2000f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:07:46.686 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:46.685+0000 7ff2087b0700 1 -- 172.21.15.155:0/3132305145 shutdown_connections 2024-02-10T09:07:46.686 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:46.685+0000 7ff2087b0700 1 -- 172.21.15.155:0/3132305145 wait complete. 2024-02-10T09:07:46.687 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:07:47.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:07:46 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:07:46.684845+0000 mon.smithi077 (mon.0) 266 : audit [DBG] from='client.? 172.21.15.155:0/3132305145' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:07:48.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:07:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:07:47.040642+0000 mgr.smithi077.dxntvw (mgr.14162) 278 : cluster [DBG] pgmap v264: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:07:48.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:07:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:07:47.097326+0000 mon.smithi077 (mon.0) 267 : audit [INF] from='mgr.14162 172.21.15.77:0/2029106299' entity='mgr.smithi077.dxntvw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi077.dxntvw/mirror_snapshot_schedule"}]: dispatch 2024-02-10T09:07:48.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:07:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:07:47.103083+0000 mon.smithi077 (mon.0) 268 : audit [INF] from='mgr.14162 172.21.15.77:0/2029106299' entity='mgr.smithi077.dxntvw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi077.dxntvw/trash_purge_schedule"}]: dispatch 2024-02-10T09:07:48.530 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:07:48.531 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:07:49.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:07:49 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:07:49.041007+0000 mgr.smithi077.dxntvw (mgr.14162) 279 : cluster [DBG] pgmap v265: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:07:50.825 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:50.822+0000 7fb8cc284700 1 -- 172.21.15.155:0/2025251375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8c40f5410 msgr2=0x7fb8c40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:07:50.826 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:50.822+0000 7fb8cc284700 1 --2- 172.21.15.155:0/2025251375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8c40f5410 0x7fb8c40f5830 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7fb8b4009fb0 tx=0x7fb8b4009b30).stop 2024-02-10T09:07:50.826 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:50.822+0000 7fb8cc284700 1 -- 172.21.15.155:0/2025251375 shutdown_connections 2024-02-10T09:07:50.826 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:50.822+0000 7fb8cc284700 1 --2- 172.21.15.155:0/2025251375 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8c40f5410 0x7fb8c40f5830 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:07:50.826 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:50.822+0000 7fb8cc284700 1 -- 172.21.15.155:0/2025251375 >> 172.21.15.155:0/2025251375 conn(0x7fb8c40f0b70 msgr2=0x7fb8c40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:07:50.826 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:50.823+0000 7fb8cc284700 1 -- 172.21.15.155:0/2025251375 shutdown_connections 2024-02-10T09:07:50.826 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:50.823+0000 7fb8cc284700 1 -- 172.21.15.155:0/2025251375 wait complete. 2024-02-10T09:07:50.826 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:50.823+0000 7fb8cc284700 1 Processor -- start 2024-02-10T09:07:50.826 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:50.823+0000 7fb8cc284700 1 -- start start 2024-02-10T09:07:50.826 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:50.823+0000 7fb8cc284700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8c40f5410 0x7fb8c41094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:07:50.826 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:50.823+0000 7fb8cc284700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c4060240 con 0x7fb8c40f5410 2024-02-10T09:07:50.826 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:50.823+0000 7fb8ca020700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8c40f5410 0x7fb8c41094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:07:50.826 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:50.823+0000 7fb8ca020700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8c40f5410 0x7fb8c41094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:48778/0 (socket says 172.21.15.155:48778) 2024-02-10T09:07:50.826 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:50.823+0000 7fb8ca020700 1 -- 172.21.15.155:0/3051897503 learned_addr learned my addr 172.21.15.155:0/3051897503 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:07:50.827 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:50.824+0000 7fb8ca020700 1 -- 172.21.15.155:0/3051897503 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8b4005040 con 0x7fb8c40f5410 2024-02-10T09:07:50.827 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:50.824+0000 7fb8ca020700 1 --2- 172.21.15.155:0/3051897503 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8c40f5410 0x7fb8c41094e0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7fb8b4005010 tx=0x7fb8b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:07:50.827 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:50.824+0000 7fb8baffd700 1 -- 172.21.15.155:0/3051897503 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8b4009320 con 0x7fb8c40f5410 2024-02-10T09:07:50.827 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:50.824+0000 7fb8cc284700 1 -- 172.21.15.155:0/3051897503 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8c4060440 con 0x7fb8c40f5410 2024-02-10T09:07:50.827 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:50.824+0000 7fb8baffd700 1 -- 172.21.15.155:0/3051897503 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb8b4013380 con 0x7fb8c40f5410 2024-02-10T09:07:50.827 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:50.824+0000 7fb8baffd700 1 -- 172.21.15.155:0/3051897503 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8b4014e50 con 0x7fb8c40f5410 2024-02-10T09:07:50.827 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:50.824+0000 7fb8cc284700 1 -- 172.21.15.155:0/3051897503 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8c4109b90 con 0x7fb8c40f5410 2024-02-10T09:07:50.828 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:50.825+0000 7fb8baffd700 1 -- 172.21.15.155:0/3051897503 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb8b40095f0 con 0x7fb8c40f5410 2024-02-10T09:07:50.828 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:50.825+0000 7fb8cc284700 1 -- 172.21.15.155:0/3051897503 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8c4045ad0 con 0x7fb8c40f5410 2024-02-10T09:07:50.828 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:50.825+0000 7fb8baffd700 1 --2- 172.21.15.155:0/3051897503 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb8b002ff10 0x7fb8b00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:07:50.828 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:50.825+0000 7fb8baffd700 1 -- 172.21.15.155:0/3051897503 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb8b4040fa0 con 0x7fb8c40f5410 2024-02-10T09:07:50.828 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:50.825+0000 7fb8c981f700 1 --2- 172.21.15.155:0/3051897503 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb8b002ff10 0x7fb8b00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:07:50.828 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:50.826+0000 7fb8c981f700 1 --2- 172.21.15.155:0/3051897503 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb8b002ff10 0x7fb8b00323d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fb8c00096a0 tx=0x7fb8c0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:07:50.831 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:50.830+0000 7fb8baffd700 1 -- 172.21.15.155:0/3051897503 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb8b4014420 con 0x7fb8c40f5410 2024-02-10T09:07:51.034 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:51.033+0000 7fb8cc284700 1 -- 172.21.15.155:0/3051897503 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb8c410a3b0 con 0x7fb8c40f5410 2024-02-10T09:07:51.035 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:51.035+0000 7fb8baffd700 1 -- 172.21.15.155:0/3051897503 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb8b4010230 con 0x7fb8c40f5410 2024-02-10T09:07:51.035 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:07:51.035 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:07:51.037 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:51.035+0000 7fb8cc284700 1 -- 172.21.15.155:0/3051897503 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb8b002ff10 msgr2=0x7fb8b00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:07:51.037 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:51.035+0000 7fb8cc284700 1 --2- 172.21.15.155:0/3051897503 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb8b002ff10 0x7fb8b00323d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fb8c00096a0 tx=0x7fb8c0006b40).stop 2024-02-10T09:07:51.037 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:51.036+0000 7fb8cc284700 1 -- 172.21.15.155:0/3051897503 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8c40f5410 msgr2=0x7fb8c41094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:07:51.037 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:51.036+0000 7fb8cc284700 1 --2- 172.21.15.155:0/3051897503 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8c40f5410 0x7fb8c41094e0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7fb8b4005010 tx=0x7fb8b400b040).stop 2024-02-10T09:07:51.037 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:51.036+0000 7fb8cc284700 1 -- 172.21.15.155:0/3051897503 shutdown_connections 2024-02-10T09:07:51.037 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:51.036+0000 7fb8cc284700 1 --2- 172.21.15.155:0/3051897503 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb8b002ff10 0x7fb8b00323d0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:07:51.037 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:51.036+0000 7fb8cc284700 1 --2- 172.21.15.155:0/3051897503 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb8c40f5410 0x7fb8c41094e0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:07:51.037 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:51.036+0000 7fb8cc284700 1 -- 172.21.15.155:0/3051897503 >> 172.21.15.155:0/3051897503 conn(0x7fb8c40f0b70 msgr2=0x7fb8c40f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:07:51.037 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:51.036+0000 7fb8cc284700 1 -- 172.21.15.155:0/3051897503 shutdown_connections 2024-02-10T09:07:51.037 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:51.036+0000 7fb8cc284700 1 -- 172.21.15.155:0/3051897503 wait complete. 2024-02-10T09:07:51.038 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:07:51.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:07:51 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:07:51.035458+0000 mon.smithi077 (mon.0) 269 : audit [DBG] from='client.? 172.21.15.155:0/3051897503' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:07:51.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:07:51 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:07:51.041404+0000 mgr.smithi077.dxntvw (mgr.14162) 280 : cluster [DBG] pgmap v266: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:07:52.949 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:07:52.949 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:07:54.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:07:53 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:07:53.041733+0000 mgr.smithi077.dxntvw (mgr.14162) 281 : cluster [DBG] pgmap v267: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:07:55.223 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:55.221+0000 7f2f690f9700 1 -- 172.21.15.155:0/929307554 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f640f5410 msgr2=0x7f2f640f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:07:55.224 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:55.221+0000 7f2f690f9700 1 --2- 172.21.15.155:0/929307554 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f640f5410 0x7f2f640f5830 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7f2f4c009fb0 tx=0x7f2f4c009b30).stop 2024-02-10T09:07:55.224 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:55.221+0000 7f2f690f9700 1 -- 172.21.15.155:0/929307554 shutdown_connections 2024-02-10T09:07:55.224 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:55.221+0000 7f2f690f9700 1 --2- 172.21.15.155:0/929307554 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f640f5410 0x7f2f640f5830 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:07:55.224 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:55.221+0000 7f2f690f9700 1 -- 172.21.15.155:0/929307554 >> 172.21.15.155:0/929307554 conn(0x7f2f640f0b70 msgr2=0x7f2f640f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:07:55.224 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:55.221+0000 7f2f690f9700 1 -- 172.21.15.155:0/929307554 shutdown_connections 2024-02-10T09:07:55.224 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:55.221+0000 7f2f690f9700 1 -- 172.21.15.155:0/929307554 wait complete. 2024-02-10T09:07:55.224 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:55.222+0000 7f2f690f9700 1 Processor -- start 2024-02-10T09:07:55.224 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:55.222+0000 7f2f690f9700 1 -- start start 2024-02-10T09:07:55.224 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:55.222+0000 7f2f690f9700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f640f5410 0x7f2f641094a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:07:55.224 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:55.222+0000 7f2f690f9700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f64060280 con 0x7f2f640f5410 2024-02-10T09:07:55.224 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:55.223+0000 7f2f62d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f640f5410 0x7f2f641094a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:07:55.224 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:55.223+0000 7f2f62d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f640f5410 0x7f2f641094a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:60622/0 (socket says 172.21.15.155:60622) 2024-02-10T09:07:55.225 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:55.223+0000 7f2f62d9d700 1 -- 172.21.15.155:0/3203255927 learned_addr learned my addr 172.21.15.155:0/3203255927 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:07:55.226 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:55.223+0000 7f2f62d9d700 1 -- 172.21.15.155:0/3203255927 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f4c005040 con 0x7f2f640f5410 2024-02-10T09:07:55.226 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:55.223+0000 7f2f62d9d700 1 --2- 172.21.15.155:0/3203255927 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f640f5410 0x7f2f641094a0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f2f4c005010 tx=0x7f2f4c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:07:55.226 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:55.223+0000 7f2f5b7fe700 1 -- 172.21.15.155:0/3203255927 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f4c009320 con 0x7f2f640f5410 2024-02-10T09:07:55.226 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:55.223+0000 7f2f690f9700 1 -- 172.21.15.155:0/3203255927 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f64060480 con 0x7f2f640f5410 2024-02-10T09:07:55.226 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:55.223+0000 7f2f5b7fe700 1 -- 172.21.15.155:0/3203255927 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2f4c013380 con 0x7f2f640f5410 2024-02-10T09:07:55.226 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:55.223+0000 7f2f5b7fe700 1 -- 172.21.15.155:0/3203255927 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f4c014e50 con 0x7f2f640f5410 2024-02-10T09:07:55.226 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:55.223+0000 7f2f690f9700 1 -- 172.21.15.155:0/3203255927 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f64109bd0 con 0x7f2f640f5410 2024-02-10T09:07:55.226 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:55.224+0000 7f2f5b7fe700 1 -- 172.21.15.155:0/3203255927 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2f4c0095f0 con 0x7f2f640f5410 2024-02-10T09:07:55.226 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:55.224+0000 7f2f690f9700 1 -- 172.21.15.155:0/3203255927 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f64045ad0 con 0x7f2f640f5410 2024-02-10T09:07:55.227 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:55.224+0000 7f2f5b7fe700 1 --2- 172.21.15.155:0/3203255927 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f2f5002ff10 0x7f2f500323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:07:55.227 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:55.224+0000 7f2f5b7fe700 1 -- 172.21.15.155:0/3203255927 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2f4c040fa0 con 0x7f2f640f5410 2024-02-10T09:07:55.227 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:55.224+0000 7f2f6259c700 1 --2- 172.21.15.155:0/3203255927 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f2f5002ff10 0x7f2f500323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:07:55.227 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:55.225+0000 7f2f6259c700 1 --2- 172.21.15.155:0/3203255927 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f2f5002ff10 0x7f2f500323d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f2f540096a0 tx=0x7f2f54006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:07:55.230 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:55.229+0000 7f2f5b7fe700 1 -- 172.21.15.155:0/3203255927 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2f4c014420 con 0x7f2f640f5410 2024-02-10T09:07:55.398 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:07:55 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:07:55.042099+0000 mgr.smithi077.dxntvw (mgr.14162) 282 : cluster [DBG] pgmap v268: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:07:55.437 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:55.436+0000 7f2f690f9700 1 -- 172.21.15.155:0/3203255927 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2f640fafa0 con 0x7f2f640f5410 2024-02-10T09:07:55.437 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:55.436+0000 7f2f5b7fe700 1 -- 172.21.15.155:0/3203255927 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2f4c010230 con 0x7f2f640f5410 2024-02-10T09:07:55.437 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:07:55.438 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:07:55.439 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:55.438+0000 7f2f690f9700 1 -- 172.21.15.155:0/3203255927 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f2f5002ff10 msgr2=0x7f2f500323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:07:55.439 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:55.438+0000 7f2f690f9700 1 --2- 172.21.15.155:0/3203255927 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f2f5002ff10 0x7f2f500323d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f2f540096a0 tx=0x7f2f54006b40).stop 2024-02-10T09:07:55.439 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:55.438+0000 7f2f690f9700 1 -- 172.21.15.155:0/3203255927 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f640f5410 msgr2=0x7f2f641094a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:07:55.440 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:55.438+0000 7f2f690f9700 1 --2- 172.21.15.155:0/3203255927 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f640f5410 0x7f2f641094a0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f2f4c005010 tx=0x7f2f4c00b040).stop 2024-02-10T09:07:55.440 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:55.438+0000 7f2f690f9700 1 -- 172.21.15.155:0/3203255927 shutdown_connections 2024-02-10T09:07:55.440 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:55.438+0000 7f2f690f9700 1 --2- 172.21.15.155:0/3203255927 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f2f5002ff10 0x7f2f500323d0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:07:55.440 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:55.438+0000 7f2f690f9700 1 --2- 172.21.15.155:0/3203255927 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f640f5410 0x7f2f641094a0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:07:55.440 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:55.438+0000 7f2f690f9700 1 -- 172.21.15.155:0/3203255927 >> 172.21.15.155:0/3203255927 conn(0x7f2f640f0b70 msgr2=0x7f2f640f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:07:55.440 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:55.438+0000 7f2f690f9700 1 -- 172.21.15.155:0/3203255927 shutdown_connections 2024-02-10T09:07:55.440 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:55.438+0000 7f2f690f9700 1 -- 172.21.15.155:0/3203255927 wait complete. 2024-02-10T09:07:55.440 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:07:56.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:07:56 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:07:55.437994+0000 mon.smithi077 (mon.0) 270 : audit [DBG] from='client.? 172.21.15.155:0/3203255927' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:07:57.315 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:07:57.315 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:07:57.398 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:07:57 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:07:57.042382+0000 mgr.smithi077.dxntvw (mgr.14162) 283 : cluster [DBG] pgmap v269: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:07:59.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:07:59 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:07:59.042657+0000 mgr.smithi077.dxntvw (mgr.14162) 284 : cluster [DBG] pgmap v270: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:07:59.536 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:59.486+0000 7f48c2849700 1 -- 172.21.15.155:0/4168129043 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48bc0f5410 msgr2=0x7f48bc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:07:59.536 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:59.486+0000 7f48c2849700 1 --2- 172.21.15.155:0/4168129043 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48bc0f5410 0x7f48bc0f5830 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f48b0009fb0 tx=0x7f48b0009b30).stop 2024-02-10T09:07:59.537 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:59.486+0000 7f48c2849700 1 -- 172.21.15.155:0/4168129043 shutdown_connections 2024-02-10T09:07:59.537 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:59.486+0000 7f48c2849700 1 --2- 172.21.15.155:0/4168129043 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48bc0f5410 0x7f48bc0f5830 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:07:59.537 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:59.486+0000 7f48c2849700 1 -- 172.21.15.155:0/4168129043 >> 172.21.15.155:0/4168129043 conn(0x7f48bc0f0b70 msgr2=0x7f48bc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:07:59.537 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:59.486+0000 7f48c2849700 1 -- 172.21.15.155:0/4168129043 shutdown_connections 2024-02-10T09:07:59.537 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:59.486+0000 7f48c2849700 1 -- 172.21.15.155:0/4168129043 wait complete. 2024-02-10T09:07:59.537 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:59.487+0000 7f48c2849700 1 Processor -- start 2024-02-10T09:07:59.537 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:59.487+0000 7f48c2849700 1 -- start start 2024-02-10T09:07:59.537 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:59.487+0000 7f48c2849700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48bc0f5410 0x7f48bc0fb580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:07:59.538 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:59.487+0000 7f48c2849700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48bc0f9a00 con 0x7f48bc0f5410 2024-02-10T09:07:59.538 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:59.487+0000 7f48bbfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48bc0f5410 0x7f48bc0fb580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:07:59.538 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:59.487+0000 7f48bbfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48bc0f5410 0x7f48bc0fb580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:60638/0 (socket says 172.21.15.155:60638) 2024-02-10T09:07:59.538 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:59.487+0000 7f48bbfff700 1 -- 172.21.15.155:0/375448786 learned_addr learned my addr 172.21.15.155:0/375448786 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:07:59.538 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:59.488+0000 7f48bbfff700 1 -- 172.21.15.155:0/375448786 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48b0005040 con 0x7f48bc0f5410 2024-02-10T09:07:59.538 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:59.488+0000 7f48bbfff700 1 --2- 172.21.15.155:0/375448786 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48bc0f5410 0x7f48bc0fb580 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f48b0005010 tx=0x7f48b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:07:59.538 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:59.488+0000 7f48b8ff9700 1 -- 172.21.15.155:0/375448786 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f48b0009320 con 0x7f48bc0f5410 2024-02-10T09:07:59.539 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:59.488+0000 7f48c2849700 1 -- 172.21.15.155:0/375448786 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48bc0f9ba0 con 0x7f48bc0f5410 2024-02-10T09:07:59.539 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:59.488+0000 7f48b8ff9700 1 -- 172.21.15.155:0/375448786 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f48b0013380 con 0x7f48bc0f5410 2024-02-10T09:07:59.539 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:59.488+0000 7f48b8ff9700 1 -- 172.21.15.155:0/375448786 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f48b0014e50 con 0x7f48bc0f5410 2024-02-10T09:07:59.539 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:59.488+0000 7f48c2849700 1 -- 172.21.15.155:0/375448786 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48bc0fa030 con 0x7f48bc0f5410 2024-02-10T09:07:59.539 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:59.489+0000 7f48b8ff9700 1 -- 172.21.15.155:0/375448786 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f48b00095f0 con 0x7f48bc0f5410 2024-02-10T09:07:59.539 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:59.489+0000 7f48c2849700 1 -- 172.21.15.155:0/375448786 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48bc059100 con 0x7f48bc0f5410 2024-02-10T09:07:59.539 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:59.489+0000 7f48b8ff9700 1 --2- 172.21.15.155:0/375448786 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f48a402ff10 0x7f48a40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:07:59.539 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:59.489+0000 7f48b8ff9700 1 -- 172.21.15.155:0/375448786 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f48b0040fa0 con 0x7f48bc0f5410 2024-02-10T09:07:59.539 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:59.489+0000 7f48bb7fe700 1 --2- 172.21.15.155:0/375448786 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f48a402ff10 0x7f48a40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:07:59.539 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:59.490+0000 7f48bb7fe700 1 --2- 172.21.15.155:0/375448786 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f48a402ff10 0x7f48a40323d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f48ac0096a0 tx=0x7f48ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:07:59.539 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:59.494+0000 7f48b8ff9700 1 -- 172.21.15.155:0/375448786 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f48b0014420 con 0x7f48bc0f5410 2024-02-10T09:07:59.702 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:59.702+0000 7f48c2849700 1 -- 172.21.15.155:0/375448786 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f48bc0fc0d0 con 0x7f48bc0f5410 2024-02-10T09:07:59.703 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:59.703+0000 7f48b8ff9700 1 -- 172.21.15.155:0/375448786 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f48b0010230 con 0x7f48bc0f5410 2024-02-10T09:07:59.703 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:07:59.703 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:07:59.704 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:59.704+0000 7f48c2849700 1 -- 172.21.15.155:0/375448786 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f48a402ff10 msgr2=0x7f48a40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:07:59.704 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:59.704+0000 7f48c2849700 1 --2- 172.21.15.155:0/375448786 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f48a402ff10 0x7f48a40323d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f48ac0096a0 tx=0x7f48ac006b40).stop 2024-02-10T09:07:59.704 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:59.704+0000 7f48c2849700 1 -- 172.21.15.155:0/375448786 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48bc0f5410 msgr2=0x7f48bc0fb580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:07:59.705 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:59.704+0000 7f48c2849700 1 --2- 172.21.15.155:0/375448786 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48bc0f5410 0x7f48bc0fb580 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f48b0005010 tx=0x7f48b000b040).stop 2024-02-10T09:07:59.705 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:59.704+0000 7f48c2849700 1 -- 172.21.15.155:0/375448786 shutdown_connections 2024-02-10T09:07:59.705 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:59.704+0000 7f48c2849700 1 --2- 172.21.15.155:0/375448786 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f48a402ff10 0x7f48a40323d0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:07:59.705 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:59.704+0000 7f48c2849700 1 --2- 172.21.15.155:0/375448786 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48bc0f5410 0x7f48bc0fb580 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:07:59.705 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:59.704+0000 7f48c2849700 1 -- 172.21.15.155:0/375448786 >> 172.21.15.155:0/375448786 conn(0x7f48bc0f0b70 msgr2=0x7f48bc0f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:07:59.705 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:59.704+0000 7f48c2849700 1 -- 172.21.15.155:0/375448786 shutdown_connections 2024-02-10T09:07:59.705 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:07:59.704+0000 7f48c2849700 1 -- 172.21.15.155:0/375448786 wait complete. 2024-02-10T09:07:59.705 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:08:00.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:00 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:07:59.703357+0000 mon.smithi077 (mon.0) 271 : audit [DBG] from='client.? 172.21.15.155:0/375448786' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:08:01.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:01 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:08:01.042941+0000 mgr.smithi077.dxntvw (mgr.14162) 285 : cluster [DBG] pgmap v271: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:08:01.534 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:08:01.534 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:08:03.833 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:03.831+0000 7f337ffff700 1 -- 172.21.15.155:0/998251690 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33800601e0 msgr2=0x7f33800f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:08:03.833 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:03.831+0000 7f337ffff700 1 --2- 172.21.15.155:0/998251690 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33800601e0 0x7f33800f8f20 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f3374009fb0 tx=0x7f3374009b30).stop 2024-02-10T09:08:03.833 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:03.831+0000 7f337ffff700 1 -- 172.21.15.155:0/998251690 shutdown_connections 2024-02-10T09:08:03.833 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:03.831+0000 7f337ffff700 1 --2- 172.21.15.155:0/998251690 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33800601e0 0x7f33800f8f20 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:08:03.833 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:03.831+0000 7f337ffff700 1 -- 172.21.15.155:0/998251690 >> 172.21.15.155:0/998251690 conn(0x7f33800f09a0 msgr2=0x7f33800f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:08:03.833 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:03.831+0000 7f337ffff700 1 -- 172.21.15.155:0/998251690 shutdown_connections 2024-02-10T09:08:03.833 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:03.831+0000 7f337ffff700 1 -- 172.21.15.155:0/998251690 wait complete. 2024-02-10T09:08:03.833 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:03.831+0000 7f337ffff700 1 Processor -- start 2024-02-10T09:08:03.834 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:03.831+0000 7f337ffff700 1 -- start start 2024-02-10T09:08:03.834 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:03.832+0000 7f337ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33800601e0 0x7f3380106600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:08:03.834 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:03.832+0000 7f337ffff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3380106af0 con 0x7f33800601e0 2024-02-10T09:08:03.834 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:03.832+0000 7f337effd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33800601e0 0x7f3380106600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:08:03.834 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:03.832+0000 7f337effd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33800601e0 0x7f3380106600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:34706/0 (socket says 172.21.15.155:34706) 2024-02-10T09:08:03.834 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:03.832+0000 7f337effd700 1 -- 172.21.15.155:0/2835869195 learned_addr learned my addr 172.21.15.155:0/2835869195 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:08:03.834 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:03.832+0000 7f337effd700 1 -- 172.21.15.155:0/2835869195 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3374005040 con 0x7f33800601e0 2024-02-10T09:08:03.834 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:03.832+0000 7f337effd700 1 --2- 172.21.15.155:0/2835869195 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33800601e0 0x7f3380106600 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f3374005010 tx=0x7f337400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:08:03.834 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:03.832+0000 7f33677fe700 1 -- 172.21.15.155:0/2835869195 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3374009320 con 0x7f33800601e0 2024-02-10T09:08:03.834 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:03.833+0000 7f337ffff700 1 -- 172.21.15.155:0/2835869195 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3380106cf0 con 0x7f33800601e0 2024-02-10T09:08:03.834 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:03.833+0000 7f33677fe700 1 -- 172.21.15.155:0/2835869195 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3374013380 con 0x7f33800601e0 2024-02-10T09:08:03.834 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:03.833+0000 7f33677fe700 1 -- 172.21.15.155:0/2835869195 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3374014e50 con 0x7f33800601e0 2024-02-10T09:08:03.834 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:03.833+0000 7f337ffff700 1 -- 172.21.15.155:0/2835869195 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3380107180 con 0x7f33800601e0 2024-02-10T09:08:03.835 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:03.833+0000 7f33677fe700 1 -- 172.21.15.155:0/2835869195 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f33740263a0 con 0x7f33800601e0 2024-02-10T09:08:03.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:03.833+0000 7f33677fe700 1 --2- 172.21.15.155:0/2835869195 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f336802ff10 0x7f33680323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:08:03.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:03.833+0000 7f33677fe700 1 -- 172.21.15.155:0/2835869195 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3374040ef0 con 0x7f33800601e0 2024-02-10T09:08:03.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:03.834+0000 7f337ffff700 1 -- 172.21.15.155:0/2835869195 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3380045ad0 con 0x7f33800601e0 2024-02-10T09:08:03.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:03.834+0000 7f337e7fc700 1 --2- 172.21.15.155:0/2835869195 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f336802ff10 0x7f33680323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:08:03.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:03.834+0000 7f337e7fc700 1 --2- 172.21.15.155:0/2835869195 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f336802ff10 0x7f33680323d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f33700096a0 tx=0x7f3370006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:08:03.838 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:03.838+0000 7f33677fe700 1 -- 172.21.15.155:0/2835869195 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3374012920 con 0x7f33800601e0 2024-02-10T09:08:04.044 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:04.044+0000 7f337ffff700 1 -- 172.21.15.155:0/2835869195 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3380107ea0 con 0x7f33800601e0 2024-02-10T09:08:04.045 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:04.045+0000 7f33677fe700 1 -- 172.21.15.155:0/2835869195 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f337402b020 con 0x7f33800601e0 2024-02-10T09:08:04.045 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:08:04.045 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:08:04.047 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:04.046+0000 7f337ffff700 1 -- 172.21.15.155:0/2835869195 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f336802ff10 msgr2=0x7f33680323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:08:04.047 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:04.046+0000 7f337ffff700 1 --2- 172.21.15.155:0/2835869195 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f336802ff10 0x7f33680323d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f33700096a0 tx=0x7f3370006b40).stop 2024-02-10T09:08:04.047 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:04.046+0000 7f337ffff700 1 -- 172.21.15.155:0/2835869195 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33800601e0 msgr2=0x7f3380106600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:08:04.047 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:04.046+0000 7f337ffff700 1 --2- 172.21.15.155:0/2835869195 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33800601e0 0x7f3380106600 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f3374005010 tx=0x7f337400b040).stop 2024-02-10T09:08:04.047 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:04.046+0000 7f337ffff700 1 -- 172.21.15.155:0/2835869195 shutdown_connections 2024-02-10T09:08:04.047 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:04.046+0000 7f337ffff700 1 --2- 172.21.15.155:0/2835869195 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f336802ff10 0x7f33680323d0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:08:04.047 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:04.046+0000 7f337ffff700 1 --2- 172.21.15.155:0/2835869195 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f33800601e0 0x7f3380106600 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:08:04.047 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:04.046+0000 7f337ffff700 1 -- 172.21.15.155:0/2835869195 >> 172.21.15.155:0/2835869195 conn(0x7f33800f09a0 msgr2=0x7f33800f13d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:08:04.048 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:04.046+0000 7f337ffff700 1 -- 172.21.15.155:0/2835869195 shutdown_connections 2024-02-10T09:08:04.048 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:04.046+0000 7f337ffff700 1 -- 172.21.15.155:0/2835869195 wait complete. 2024-02-10T09:08:04.048 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:08:04.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:03 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:08:03.043295+0000 mgr.smithi077.dxntvw (mgr.14162) 286 : cluster [DBG] pgmap v272: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:08:05.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:04 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:08:04.045194+0000 mon.smithi077 (mon.0) 272 : audit [DBG] from='client.? 172.21.15.155:0/2835869195' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:08:05.989 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:08:05.989 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:08:06.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:05 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:08:05.043669+0000 mgr.smithi077.dxntvw (mgr.14162) 287 : cluster [DBG] pgmap v273: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:08:07.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:07 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:08:07.044033+0000 mgr.smithi077.dxntvw (mgr.14162) 288 : cluster [DBG] pgmap v274: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:08:08.226 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:08.225+0000 7f279e705700 1 -- 172.21.15.155:0/2208941419 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27980f6510 msgr2=0x7f27980f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:08:08.226 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:08.225+0000 7f279e705700 1 --2- 172.21.15.155:0/2208941419 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27980f6510 0x7f27980f6930 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7f2780009fb0 tx=0x7f2780009b30).stop 2024-02-10T09:08:08.226 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:08.225+0000 7f279e705700 1 -- 172.21.15.155:0/2208941419 shutdown_connections 2024-02-10T09:08:08.226 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:08.225+0000 7f279e705700 1 --2- 172.21.15.155:0/2208941419 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27980f6510 0x7f27980f6930 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:08:08.226 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:08.225+0000 7f279e705700 1 -- 172.21.15.155:0/2208941419 >> 172.21.15.155:0/2208941419 conn(0x7f27980f1c70 msgr2=0x7f27980f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:08:08.226 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:08.225+0000 7f279e705700 1 -- 172.21.15.155:0/2208941419 shutdown_connections 2024-02-10T09:08:08.226 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:08.225+0000 7f279e705700 1 -- 172.21.15.155:0/2208941419 wait complete. 2024-02-10T09:08:08.227 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:08.225+0000 7f279e705700 1 Processor -- start 2024-02-10T09:08:08.227 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:08.226+0000 7f279e705700 1 -- start start 2024-02-10T09:08:08.227 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:08.226+0000 7f279e705700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27980f6510 0x7f279806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:08:08.227 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:08.226+0000 7f279e705700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2798069e20 con 0x7f27980f6510 2024-02-10T09:08:08.229 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:08.226+0000 7f2797fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27980f6510 0x7f279806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:08:08.229 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:08.226+0000 7f2797fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27980f6510 0x7f279806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:34732/0 (socket says 172.21.15.155:34732) 2024-02-10T09:08:08.229 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:08.226+0000 7f2797fff700 1 -- 172.21.15.155:0/696239197 learned_addr learned my addr 172.21.15.155:0/696239197 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:08:08.229 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:08.226+0000 7f2797fff700 1 -- 172.21.15.155:0/696239197 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2780005040 con 0x7f27980f6510 2024-02-10T09:08:08.229 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:08.227+0000 7f2797fff700 1 --2- 172.21.15.155:0/696239197 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27980f6510 0x7f279806b9a0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f2780005010 tx=0x7f278000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:08:08.229 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:08.227+0000 7f2794ff9700 1 -- 172.21.15.155:0/696239197 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2780009320 con 0x7f27980f6510 2024-02-10T09:08:08.229 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:08.227+0000 7f279e705700 1 -- 172.21.15.155:0/696239197 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2798069fc0 con 0x7f27980f6510 2024-02-10T09:08:08.229 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:08.227+0000 7f2794ff9700 1 -- 172.21.15.155:0/696239197 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2780013380 con 0x7f27980f6510 2024-02-10T09:08:08.229 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:08.227+0000 7f2794ff9700 1 -- 172.21.15.155:0/696239197 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2780014e50 con 0x7f27980f6510 2024-02-10T09:08:08.229 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:08.227+0000 7f279e705700 1 -- 172.21.15.155:0/696239197 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f279806a450 con 0x7f27980f6510 2024-02-10T09:08:08.229 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:08.228+0000 7f2794ff9700 1 -- 172.21.15.155:0/696239197 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2780014420 con 0x7f27980f6510 2024-02-10T09:08:08.229 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:08.228+0000 7f279e705700 1 -- 172.21.15.155:0/696239197 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2798059100 con 0x7f27980f6510 2024-02-10T09:08:08.230 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:08.228+0000 7f2794ff9700 1 --2- 172.21.15.155:0/696239197 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f278402ff10 0x7f27840323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:08:08.230 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:08.228+0000 7f2794ff9700 1 -- 172.21.15.155:0/696239197 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2780040f40 con 0x7f27980f6510 2024-02-10T09:08:08.230 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:08.228+0000 7f27977fe700 1 --2- 172.21.15.155:0/696239197 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f278402ff10 0x7f27840323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:08:08.230 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:08.228+0000 7f27977fe700 1 --2- 172.21.15.155:0/696239197 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f278402ff10 0x7f27840323d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f27880096a0 tx=0x7f2788006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:08:08.232 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:08.232+0000 7f2794ff9700 1 -- 172.21.15.155:0/696239197 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f27800095f0 con 0x7f27980f6510 2024-02-10T09:08:08.437 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:08.437+0000 7f279e705700 1 -- 172.21.15.155:0/696239197 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f27980fc0a0 con 0x7f27980f6510 2024-02-10T09:08:08.438 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:08.438+0000 7f2794ff9700 1 -- 172.21.15.155:0/696239197 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f278001db70 con 0x7f27980f6510 2024-02-10T09:08:08.438 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:08:08.438 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:08:08.440 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:08.440+0000 7f279e705700 1 -- 172.21.15.155:0/696239197 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f278402ff10 msgr2=0x7f27840323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:08:08.441 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:08.440+0000 7f279e705700 1 --2- 172.21.15.155:0/696239197 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f278402ff10 0x7f27840323d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f27880096a0 tx=0x7f2788006b40).stop 2024-02-10T09:08:08.441 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:08.440+0000 7f279e705700 1 -- 172.21.15.155:0/696239197 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27980f6510 msgr2=0x7f279806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:08:08.441 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:08.440+0000 7f279e705700 1 --2- 172.21.15.155:0/696239197 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27980f6510 0x7f279806b9a0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f2780005010 tx=0x7f278000b040).stop 2024-02-10T09:08:08.441 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:08.440+0000 7f279e705700 1 -- 172.21.15.155:0/696239197 shutdown_connections 2024-02-10T09:08:08.441 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:08.440+0000 7f279e705700 1 --2- 172.21.15.155:0/696239197 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f278402ff10 0x7f27840323d0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:08:08.441 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:08.440+0000 7f279e705700 1 --2- 172.21.15.155:0/696239197 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27980f6510 0x7f279806b9a0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:08:08.441 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:08.440+0000 7f279e705700 1 -- 172.21.15.155:0/696239197 >> 172.21.15.155:0/696239197 conn(0x7f27980f1c70 msgr2=0x7f27980f26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:08:08.441 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:08.440+0000 7f279e705700 1 -- 172.21.15.155:0/696239197 shutdown_connections 2024-02-10T09:08:08.441 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:08.440+0000 7f279e705700 1 -- 172.21.15.155:0/696239197 wait complete. 2024-02-10T09:08:08.441 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:08:09.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:08 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:08:08.438613+0000 mon.smithi077 (mon.0) 273 : audit [DBG] from='client.? 172.21.15.155:0/696239197' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:08:10.079 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:08:10.080 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:08:10.148 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:09 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:08:09.044392+0000 mgr.smithi077.dxntvw (mgr.14162) 289 : cluster [DBG] pgmap v275: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:08:11.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:11 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:08:11.044776+0000 mgr.smithi077.dxntvw (mgr.14162) 290 : cluster [DBG] pgmap v276: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:08:12.080 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:12.080+0000 7f1980d08700 1 -- 172.21.15.155:0/4073643350 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f197c0f51a0 msgr2=0x7f197c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:08:12.081 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:12.080+0000 7f1980d08700 1 --2- 172.21.15.155:0/4073643350 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f197c0f51a0 0x7f197c0f55c0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7f196c009fb0 tx=0x7f196c009b30).stop 2024-02-10T09:08:12.081 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:12.080+0000 7f1980d08700 1 -- 172.21.15.155:0/4073643350 shutdown_connections 2024-02-10T09:08:12.081 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:12.080+0000 7f1980d08700 1 --2- 172.21.15.155:0/4073643350 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f197c0f51a0 0x7f197c0f55c0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:08:12.081 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:12.080+0000 7f1980d08700 1 -- 172.21.15.155:0/4073643350 >> 172.21.15.155:0/4073643350 conn(0x7f197c0f09a0 msgr2=0x7f197c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:08:12.081 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:12.080+0000 7f1980d08700 1 -- 172.21.15.155:0/4073643350 shutdown_connections 2024-02-10T09:08:12.081 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:12.080+0000 7f1980d08700 1 -- 172.21.15.155:0/4073643350 wait complete. 2024-02-10T09:08:12.081 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:12.080+0000 7f1980d08700 1 Processor -- start 2024-02-10T09:08:12.081 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:12.081+0000 7f1980d08700 1 -- start start 2024-02-10T09:08:12.081 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:12.081+0000 7f1980d08700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f197c0f51a0 0x7f197c1054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:08:12.081 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:12.081+0000 7f1980d08700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f197c1059c0 con 0x7f197c0f51a0 2024-02-10T09:08:12.083 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:12.081+0000 7f197b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f197c0f51a0 0x7f197c1054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:08:12.083 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:12.081+0000 7f197b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f197c0f51a0 0x7f197c1054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:34750/0 (socket says 172.21.15.155:34750) 2024-02-10T09:08:12.083 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:12.081+0000 7f197b7fe700 1 -- 172.21.15.155:0/1492860315 learned_addr learned my addr 172.21.15.155:0/1492860315 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:08:12.084 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:12.081+0000 7f197b7fe700 1 -- 172.21.15.155:0/1492860315 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f196c005040 con 0x7f197c0f51a0 2024-02-10T09:08:12.084 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:12.081+0000 7f197b7fe700 1 --2- 172.21.15.155:0/1492860315 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f197c0f51a0 0x7f197c1054d0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f196c0089f0 tx=0x7f196c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:08:12.084 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:12.082+0000 7f1963fff700 1 -- 172.21.15.155:0/1492860315 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f196c009320 con 0x7f197c0f51a0 2024-02-10T09:08:12.084 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:12.082+0000 7f1980d08700 1 -- 172.21.15.155:0/1492860315 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f197c105bc0 con 0x7f197c0f51a0 2024-02-10T09:08:12.084 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:12.082+0000 7f1963fff700 1 -- 172.21.15.155:0/1492860315 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f196c013380 con 0x7f197c0f51a0 2024-02-10T09:08:12.084 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:12.082+0000 7f1963fff700 1 -- 172.21.15.155:0/1492860315 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f196c014e20 con 0x7f197c0f51a0 2024-02-10T09:08:12.084 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:12.082+0000 7f1980d08700 1 -- 172.21.15.155:0/1492860315 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f197c106050 con 0x7f197c0f51a0 2024-02-10T09:08:12.084 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:12.082+0000 7f1963fff700 1 -- 172.21.15.155:0/1492860315 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f196c0095f0 con 0x7f197c0f51a0 2024-02-10T09:08:12.084 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:12.082+0000 7f1980d08700 1 -- 172.21.15.155:0/1492860315 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f197c045ad0 con 0x7f197c0f51a0 2024-02-10T09:08:12.084 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:12.083+0000 7f1963fff700 1 --2- 172.21.15.155:0/1492860315 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f196402ff10 0x7f19640323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:08:12.084 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:12.083+0000 7f1963fff700 1 -- 172.21.15.155:0/1492860315 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f196c040e90 con 0x7f197c0f51a0 2024-02-10T09:08:12.085 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:12.083+0000 7f197affd700 1 --2- 172.21.15.155:0/1492860315 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f196402ff10 0x7f19640323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:08:12.085 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:12.083+0000 7f197affd700 1 --2- 172.21.15.155:0/1492860315 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f196402ff10 0x7f19640323d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f19700096a0 tx=0x7f1970006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:08:12.087 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:12.087+0000 7f1963fff700 1 -- 172.21.15.155:0/1492860315 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f196c014420 con 0x7f197c0f51a0 2024-02-10T09:08:12.297 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:12.296+0000 7f1980d08700 1 -- 172.21.15.155:0/1492860315 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f197c106f00 con 0x7f197c0f51a0 2024-02-10T09:08:12.297 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:12.297+0000 7f1963fff700 1 -- 172.21.15.155:0/1492860315 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f196c01d9d0 con 0x7f197c0f51a0 2024-02-10T09:08:12.298 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:08:12.298 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:08:12.299 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:12.299+0000 7f1980d08700 1 -- 172.21.15.155:0/1492860315 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f196402ff10 msgr2=0x7f19640323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:08:12.299 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:12.299+0000 7f1980d08700 1 --2- 172.21.15.155:0/1492860315 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f196402ff10 0x7f19640323d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f19700096a0 tx=0x7f1970006b40).stop 2024-02-10T09:08:12.299 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:12.299+0000 7f1980d08700 1 -- 172.21.15.155:0/1492860315 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f197c0f51a0 msgr2=0x7f197c1054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:08:12.299 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:12.299+0000 7f1980d08700 1 --2- 172.21.15.155:0/1492860315 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f197c0f51a0 0x7f197c1054d0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f196c0089f0 tx=0x7f196c00b040).stop 2024-02-10T09:08:12.299 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:12.299+0000 7f1980d08700 1 -- 172.21.15.155:0/1492860315 shutdown_connections 2024-02-10T09:08:12.299 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:12.299+0000 7f1980d08700 1 --2- 172.21.15.155:0/1492860315 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f196402ff10 0x7f19640323d0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:08:12.300 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:12.299+0000 7f1980d08700 1 --2- 172.21.15.155:0/1492860315 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f197c0f51a0 0x7f197c1054d0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:08:12.300 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:12.299+0000 7f1980d08700 1 -- 172.21.15.155:0/1492860315 >> 172.21.15.155:0/1492860315 conn(0x7f197c0f09a0 msgr2=0x7f197c0f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:08:12.300 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:12.299+0000 7f1980d08700 1 -- 172.21.15.155:0/1492860315 shutdown_connections 2024-02-10T09:08:12.300 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:12.299+0000 7f1980d08700 1 -- 172.21.15.155:0/1492860315 wait complete. 2024-02-10T09:08:12.300 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:08:12.649 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:12 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:08:12.297794+0000 mon.smithi077 (mon.0) 274 : audit [DBG] from='client.? 172.21.15.155:0/1492860315' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:08:14.129 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:08:14.130 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:08:14.148 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:13 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:08:13.045138+0000 mgr.smithi077.dxntvw (mgr.14162) 291 : cluster [DBG] pgmap v277: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:08:15.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:15 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:08:15.045449+0000 mgr.smithi077.dxntvw (mgr.14162) 292 : cluster [DBG] pgmap v278: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:08:16.220 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:16.203+0000 7f2bf91a1700 1 -- 172.21.15.155:0/1281217799 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2bf40f62a0 msgr2=0x7f2bf40f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:08:16.221 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:16.203+0000 7f2bf91a1700 1 --2- 172.21.15.155:0/1281217799 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2bf40f62a0 0x7f2bf40f66c0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7f2be4009fb0 tx=0x7f2be4009b30).stop 2024-02-10T09:08:16.221 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:16.203+0000 7f2bf91a1700 1 -- 172.21.15.155:0/1281217799 shutdown_connections 2024-02-10T09:08:16.221 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:16.203+0000 7f2bf91a1700 1 --2- 172.21.15.155:0/1281217799 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2bf40f62a0 0x7f2bf40f66c0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:08:16.221 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:16.203+0000 7f2bf91a1700 1 -- 172.21.15.155:0/1281217799 >> 172.21.15.155:0/1281217799 conn(0x7f2bf40f1aa0 msgr2=0x7f2bf40f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:08:16.221 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:16.203+0000 7f2bf91a1700 1 -- 172.21.15.155:0/1281217799 shutdown_connections 2024-02-10T09:08:16.221 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:16.203+0000 7f2bf91a1700 1 -- 172.21.15.155:0/1281217799 wait complete. 2024-02-10T09:08:16.221 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:16.204+0000 7f2bf91a1700 1 Processor -- start 2024-02-10T09:08:16.221 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:16.204+0000 7f2bf91a1700 1 -- start start 2024-02-10T09:08:16.221 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:16.204+0000 7f2bf91a1700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2bf40f62a0 0x7f2bf41065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:08:16.221 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:16.204+0000 7f2bf91a1700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf4106ac0 con 0x7f2bf40f62a0 2024-02-10T09:08:16.221 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:16.204+0000 7f2bf3fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2bf40f62a0 0x7f2bf41065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:08:16.222 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:16.204+0000 7f2bf3fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2bf40f62a0 0x7f2bf41065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:45604/0 (socket says 172.21.15.155:45604) 2024-02-10T09:08:16.222 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:16.204+0000 7f2bf3fff700 1 -- 172.21.15.155:0/1081790117 learned_addr learned my addr 172.21.15.155:0/1081790117 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:08:16.222 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:16.204+0000 7f2bf3fff700 1 -- 172.21.15.155:0/1081790117 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2be4005040 con 0x7f2bf40f62a0 2024-02-10T09:08:16.222 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:16.205+0000 7f2bf3fff700 1 --2- 172.21.15.155:0/1081790117 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2bf40f62a0 0x7f2bf41065d0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f2be40089f0 tx=0x7f2be400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:08:16.222 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:16.205+0000 7f2bf0ff9700 1 -- 172.21.15.155:0/1081790117 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2be4009320 con 0x7f2bf40f62a0 2024-02-10T09:08:16.222 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:16.205+0000 7f2bf91a1700 1 -- 172.21.15.155:0/1081790117 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bf4106cc0 con 0x7f2bf40f62a0 2024-02-10T09:08:16.222 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:16.205+0000 7f2bf0ff9700 1 -- 172.21.15.155:0/1081790117 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2be4013380 con 0x7f2bf40f62a0 2024-02-10T09:08:16.222 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:16.205+0000 7f2bf0ff9700 1 -- 172.21.15.155:0/1081790117 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2be4014e20 con 0x7f2bf40f62a0 2024-02-10T09:08:16.222 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:16.205+0000 7f2bf91a1700 1 -- 172.21.15.155:0/1081790117 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bf4107150 con 0x7f2bf40f62a0 2024-02-10T09:08:16.222 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:16.206+0000 7f2bf0ff9700 1 -- 172.21.15.155:0/1081790117 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2be40095f0 con 0x7f2bf40f62a0 2024-02-10T09:08:16.222 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:16.206+0000 7f2bf91a1700 1 -- 172.21.15.155:0/1081790117 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bf4045ad0 con 0x7f2bf40f62a0 2024-02-10T09:08:16.222 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:16.206+0000 7f2bf0ff9700 1 --2- 172.21.15.155:0/1081790117 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f2bdc02ff10 0x7f2bdc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:08:16.223 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:16.206+0000 7f2bf0ff9700 1 -- 172.21.15.155:0/1081790117 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2be4040f50 con 0x7f2bf40f62a0 2024-02-10T09:08:16.223 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:16.206+0000 7f2bf37fe700 1 --2- 172.21.15.155:0/1081790117 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f2bdc02ff10 0x7f2bdc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:08:16.223 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:16.206+0000 7f2bf37fe700 1 --2- 172.21.15.155:0/1081790117 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f2bdc02ff10 0x7f2bdc0323d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f2be80096a0 tx=0x7f2be8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:08:16.223 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:16.211+0000 7f2bf0ff9700 1 -- 172.21.15.155:0/1081790117 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2be40268c0 con 0x7f2bf40f62a0 2024-02-10T09:08:16.416 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:16.415+0000 7f2bf91a1700 1 -- 172.21.15.155:0/1081790117 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2bf4107f90 con 0x7f2bf40f62a0 2024-02-10T09:08:16.418 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:16.416+0000 7f2bf0ff9700 1 -- 172.21.15.155:0/1081790117 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2be401d980 con 0x7f2bf40f62a0 2024-02-10T09:08:16.418 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:08:16.418 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:08:16.419 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:16.418+0000 7f2bf91a1700 1 -- 172.21.15.155:0/1081790117 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f2bdc02ff10 msgr2=0x7f2bdc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:08:16.419 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:16.418+0000 7f2bf91a1700 1 --2- 172.21.15.155:0/1081790117 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f2bdc02ff10 0x7f2bdc0323d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f2be80096a0 tx=0x7f2be8006b40).stop 2024-02-10T09:08:16.420 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:16.418+0000 7f2bf91a1700 1 -- 172.21.15.155:0/1081790117 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2bf40f62a0 msgr2=0x7f2bf41065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:08:16.420 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:16.418+0000 7f2bf91a1700 1 --2- 172.21.15.155:0/1081790117 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2bf40f62a0 0x7f2bf41065d0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f2be40089f0 tx=0x7f2be400b040).stop 2024-02-10T09:08:16.420 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:16.418+0000 7f2bf91a1700 1 -- 172.21.15.155:0/1081790117 shutdown_connections 2024-02-10T09:08:16.420 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:16.418+0000 7f2bf91a1700 1 --2- 172.21.15.155:0/1081790117 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f2bdc02ff10 0x7f2bdc0323d0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:08:16.420 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:16.418+0000 7f2bf91a1700 1 --2- 172.21.15.155:0/1081790117 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2bf40f62a0 0x7f2bf41065d0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:08:16.420 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:16.418+0000 7f2bf91a1700 1 -- 172.21.15.155:0/1081790117 >> 172.21.15.155:0/1081790117 conn(0x7f2bf40f1aa0 msgr2=0x7f2bf40f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:08:16.420 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:16.418+0000 7f2bf91a1700 1 -- 172.21.15.155:0/1081790117 shutdown_connections 2024-02-10T09:08:16.420 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:16.418+0000 7f2bf91a1700 1 -- 172.21.15.155:0/1081790117 wait complete. 2024-02-10T09:08:16.420 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:08:16.899 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:16 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:08:16.417491+0000 mon.smithi077 (mon.0) 275 : audit [DBG] from='client.? 172.21.15.155:0/1081790117' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:08:17.899 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:17 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:08:17.045826+0000 mgr.smithi077.dxntvw (mgr.14162) 293 : cluster [DBG] pgmap v279: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:08:18.107 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:08:18.108 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:08:19.398 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:19 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:08:19.046092+0000 mgr.smithi077.dxntvw (mgr.14162) 294 : cluster [DBG] pgmap v280: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:08:20.139 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:20.138+0000 7fc1d15d2700 1 -- 172.21.15.155:0/3504205004 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc1cc0f47c0 msgr2=0x7fc1cc0f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:08:20.139 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:20.138+0000 7fc1d15d2700 1 --2- 172.21.15.155:0/3504205004 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc1cc0f47c0 0x7fc1cc0f4be0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7fc1c0009fb0 tx=0x7fc1c0009b30).stop 2024-02-10T09:08:20.140 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:20.138+0000 7fc1d15d2700 1 -- 172.21.15.155:0/3504205004 shutdown_connections 2024-02-10T09:08:20.140 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:20.138+0000 7fc1d15d2700 1 --2- 172.21.15.155:0/3504205004 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc1cc0f47c0 0x7fc1cc0f4be0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:08:20.140 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:20.138+0000 7fc1d15d2700 1 -- 172.21.15.155:0/3504205004 >> 172.21.15.155:0/3504205004 conn(0x7fc1cc0f02e0 msgr2=0x7fc1cc0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:08:20.140 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:20.138+0000 7fc1d15d2700 1 -- 172.21.15.155:0/3504205004 shutdown_connections 2024-02-10T09:08:20.140 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:20.138+0000 7fc1d15d2700 1 -- 172.21.15.155:0/3504205004 wait complete. 2024-02-10T09:08:20.141 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:20.139+0000 7fc1d15d2700 1 Processor -- start 2024-02-10T09:08:20.141 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:20.139+0000 7fc1d15d2700 1 -- start start 2024-02-10T09:08:20.141 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:20.139+0000 7fc1d15d2700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc1cc0f47c0 0x7fc1cc1067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:08:20.141 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:20.139+0000 7fc1d15d2700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1cc106cc0 con 0x7fc1cc0f47c0 2024-02-10T09:08:20.142 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:20.139+0000 7fc1caffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc1cc0f47c0 0x7fc1cc1067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:08:20.142 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:20.139+0000 7fc1caffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc1cc0f47c0 0x7fc1cc1067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:45632/0 (socket says 172.21.15.155:45632) 2024-02-10T09:08:20.142 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:20.139+0000 7fc1caffd700 1 -- 172.21.15.155:0/2185287595 learned_addr learned my addr 172.21.15.155:0/2185287595 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:08:20.142 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:20.140+0000 7fc1caffd700 1 -- 172.21.15.155:0/2185287595 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1c0005040 con 0x7fc1cc0f47c0 2024-02-10T09:08:20.142 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:20.140+0000 7fc1caffd700 1 --2- 172.21.15.155:0/2185287595 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc1cc0f47c0 0x7fc1cc1067d0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7fc1c0005010 tx=0x7fc1c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:08:20.142 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:20.140+0000 7fc1b37fe700 1 -- 172.21.15.155:0/2185287595 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc1c0009320 con 0x7fc1cc0f47c0 2024-02-10T09:08:20.142 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:20.140+0000 7fc1d15d2700 1 -- 172.21.15.155:0/2185287595 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1cc106ec0 con 0x7fc1cc0f47c0 2024-02-10T09:08:20.142 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:20.140+0000 7fc1b37fe700 1 -- 172.21.15.155:0/2185287595 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc1c0013380 con 0x7fc1cc0f47c0 2024-02-10T09:08:20.142 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:20.140+0000 7fc1b37fe700 1 -- 172.21.15.155:0/2185287595 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc1c0014e50 con 0x7fc1cc0f47c0 2024-02-10T09:08:20.142 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:20.140+0000 7fc1d15d2700 1 -- 172.21.15.155:0/2185287595 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1cc107350 con 0x7fc1cc0f47c0 2024-02-10T09:08:20.143 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:20.141+0000 7fc1b37fe700 1 -- 172.21.15.155:0/2185287595 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc1c00263a0 con 0x7fc1cc0f47c0 2024-02-10T09:08:20.143 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:20.141+0000 7fc1b37fe700 1 --2- 172.21.15.155:0/2185287595 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fc1b402ff10 0x7fc1b40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:08:20.143 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:20.141+0000 7fc1b37fe700 1 -- 172.21.15.155:0/2185287595 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc1c0040fc0 con 0x7fc1cc0f47c0 2024-02-10T09:08:20.143 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:20.141+0000 7fc1d15d2700 1 -- 172.21.15.155:0/2185287595 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1cc045ad0 con 0x7fc1cc0f47c0 2024-02-10T09:08:20.143 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:20.141+0000 7fc1ca7fc700 1 --2- 172.21.15.155:0/2185287595 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fc1b402ff10 0x7fc1b40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:08:20.143 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:20.142+0000 7fc1ca7fc700 1 --2- 172.21.15.155:0/2185287595 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fc1b402ff10 0x7fc1b40323d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fc1bc0096a0 tx=0x7fc1bc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:08:20.147 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:20.146+0000 7fc1b37fe700 1 -- 172.21.15.155:0/2185287595 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc1c0012920 con 0x7fc1cc0f47c0 2024-02-10T09:08:20.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:20.356+0000 7fc1d15d2700 1 -- 172.21.15.155:0/2185287595 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc1cc0fc090 con 0x7fc1cc0f47c0 2024-02-10T09:08:20.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:20.356+0000 7fc1b37fe700 1 -- 172.21.15.155:0/2185287595 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc1c0012330 con 0x7fc1cc0f47c0 2024-02-10T09:08:20.357 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:08:20.358 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:08:20.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:20.358+0000 7fc1d15d2700 1 -- 172.21.15.155:0/2185287595 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fc1b402ff10 msgr2=0x7fc1b40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:08:20.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:20.358+0000 7fc1d15d2700 1 --2- 172.21.15.155:0/2185287595 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fc1b402ff10 0x7fc1b40323d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fc1bc0096a0 tx=0x7fc1bc006b40).stop 2024-02-10T09:08:20.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:20.358+0000 7fc1d15d2700 1 -- 172.21.15.155:0/2185287595 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc1cc0f47c0 msgr2=0x7fc1cc1067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:08:20.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:20.358+0000 7fc1d15d2700 1 --2- 172.21.15.155:0/2185287595 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc1cc0f47c0 0x7fc1cc1067d0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7fc1c0005010 tx=0x7fc1c000b040).stop 2024-02-10T09:08:20.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:20.358+0000 7fc1d15d2700 1 -- 172.21.15.155:0/2185287595 shutdown_connections 2024-02-10T09:08:20.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:20.358+0000 7fc1d15d2700 1 --2- 172.21.15.155:0/2185287595 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fc1b402ff10 0x7fc1b40323d0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:08:20.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:20.358+0000 7fc1d15d2700 1 --2- 172.21.15.155:0/2185287595 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc1cc0f47c0 0x7fc1cc1067d0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:08:20.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:20.358+0000 7fc1d15d2700 1 -- 172.21.15.155:0/2185287595 >> 172.21.15.155:0/2185287595 conn(0x7fc1cc0f02e0 msgr2=0x7fc1cc0f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:08:20.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:20.358+0000 7fc1d15d2700 1 -- 172.21.15.155:0/2185287595 shutdown_connections 2024-02-10T09:08:20.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:20.358+0000 7fc1d15d2700 1 -- 172.21.15.155:0/2185287595 wait complete. 2024-02-10T09:08:20.360 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:08:20.899 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:20 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:08:20.357935+0000 mon.smithi077 (mon.0) 276 : audit [DBG] from='client.? 172.21.15.155:0/2185287595' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:08:21.846 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:21 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:08:21.046402+0000 mgr.smithi077.dxntvw (mgr.14162) 295 : cluster [DBG] pgmap v281: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:08:22.322 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:08:22.322 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:08:24.151 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:23 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:08:23.046753+0000 mgr.smithi077.dxntvw (mgr.14162) 296 : cluster [DBG] pgmap v282: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:08:24.538 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:24.534+0000 7f2cf0bb2700 1 -- 172.21.15.155:0/171308077 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2cec0f6510 msgr2=0x7f2cec0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:08:24.538 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:24.534+0000 7f2cf0bb2700 1 --2- 172.21.15.155:0/171308077 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2cec0f6510 0x7f2cec0f6930 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f2cdc009fb0 tx=0x7f2cdc009b30).stop 2024-02-10T09:08:24.538 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:24.534+0000 7f2cf0bb2700 1 -- 172.21.15.155:0/171308077 shutdown_connections 2024-02-10T09:08:24.538 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:24.534+0000 7f2cf0bb2700 1 --2- 172.21.15.155:0/171308077 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2cec0f6510 0x7f2cec0f6930 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:08:24.538 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:24.534+0000 7f2cf0bb2700 1 -- 172.21.15.155:0/171308077 >> 172.21.15.155:0/171308077 conn(0x7f2cec0f1c70 msgr2=0x7f2cec0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:08:24.538 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:24.534+0000 7f2cf0bb2700 1 -- 172.21.15.155:0/171308077 shutdown_connections 2024-02-10T09:08:24.538 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:24.534+0000 7f2cf0bb2700 1 -- 172.21.15.155:0/171308077 wait complete. 2024-02-10T09:08:24.539 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:24.535+0000 7f2cf0bb2700 1 Processor -- start 2024-02-10T09:08:24.540 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:24.535+0000 7f2cf0bb2700 1 -- start start 2024-02-10T09:08:24.540 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:24.535+0000 7f2cf0bb2700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2cec0f6510 0x7f2cec1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:08:24.540 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:24.535+0000 7f2cf0bb2700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cec106cd0 con 0x7f2cec0f6510 2024-02-10T09:08:24.540 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:24.535+0000 7f2cea59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2cec0f6510 0x7f2cec1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:08:24.540 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:24.535+0000 7f2cea59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2cec0f6510 0x7f2cec1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:57308/0 (socket says 172.21.15.155:57308) 2024-02-10T09:08:24.540 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:24.535+0000 7f2cea59c700 1 -- 172.21.15.155:0/4162188280 learned_addr learned my addr 172.21.15.155:0/4162188280 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:08:24.540 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:24.536+0000 7f2cea59c700 1 -- 172.21.15.155:0/4162188280 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cdc005040 con 0x7f2cec0f6510 2024-02-10T09:08:24.540 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:24.536+0000 7f2cea59c700 1 --2- 172.21.15.155:0/4162188280 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2cec0f6510 0x7f2cec1067e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f2cdc005010 tx=0x7f2cdc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:08:24.540 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:24.536+0000 7f2cdaffd700 1 -- 172.21.15.155:0/4162188280 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2cdc009320 con 0x7f2cec0f6510 2024-02-10T09:08:24.540 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:24.536+0000 7f2cf0bb2700 1 -- 172.21.15.155:0/4162188280 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cec106ed0 con 0x7f2cec0f6510 2024-02-10T09:08:24.540 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:24.536+0000 7f2cdaffd700 1 -- 172.21.15.155:0/4162188280 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2cdc013380 con 0x7f2cec0f6510 2024-02-10T09:08:24.540 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:24.536+0000 7f2cdaffd700 1 -- 172.21.15.155:0/4162188280 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2cdc014e50 con 0x7f2cec0f6510 2024-02-10T09:08:24.541 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:24.536+0000 7f2cf0bb2700 1 -- 172.21.15.155:0/4162188280 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cec107360 con 0x7f2cec0f6510 2024-02-10T09:08:24.541 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:24.537+0000 7f2cdaffd700 1 -- 172.21.15.155:0/4162188280 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2cdc0263a0 con 0x7f2cec0f6510 2024-02-10T09:08:24.541 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:24.537+0000 7f2cf0bb2700 1 -- 172.21.15.155:0/4162188280 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cec045ad0 con 0x7f2cec0f6510 2024-02-10T09:08:24.541 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:24.537+0000 7f2cdaffd700 1 --2- 172.21.15.155:0/4162188280 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f2cd402ff10 0x7f2cd40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:08:24.541 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:24.537+0000 7f2cdaffd700 1 -- 172.21.15.155:0/4162188280 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2cdc040ee0 con 0x7f2cec0f6510 2024-02-10T09:08:24.541 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:24.537+0000 7f2ce9d9b700 1 --2- 172.21.15.155:0/4162188280 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f2cd402ff10 0x7f2cd40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:08:24.541 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:24.537+0000 7f2ce9d9b700 1 --2- 172.21.15.155:0/4162188280 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f2cd402ff10 0x7f2cd40323d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f2ce00096a0 tx=0x7f2ce0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:08:24.543 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:24.542+0000 7f2cdaffd700 1 -- 172.21.15.155:0/4162188280 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2cdc01dd70 con 0x7f2cec0f6510 2024-02-10T09:08:24.748 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:24.747+0000 7f2cf0bb2700 1 -- 172.21.15.155:0/4162188280 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2cec108210 con 0x7f2cec0f6510 2024-02-10T09:08:24.749 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:24.747+0000 7f2cdaffd700 1 -- 172.21.15.155:0/4162188280 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2cdc01d980 con 0x7f2cec0f6510 2024-02-10T09:08:24.750 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:08:24.750 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:08:24.750 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:24.749+0000 7f2cf0bb2700 1 -- 172.21.15.155:0/4162188280 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f2cd402ff10 msgr2=0x7f2cd40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:08:24.751 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:24.749+0000 7f2cf0bb2700 1 --2- 172.21.15.155:0/4162188280 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f2cd402ff10 0x7f2cd40323d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f2ce00096a0 tx=0x7f2ce0006b40).stop 2024-02-10T09:08:24.751 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:24.749+0000 7f2cf0bb2700 1 -- 172.21.15.155:0/4162188280 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2cec0f6510 msgr2=0x7f2cec1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:08:24.751 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:24.749+0000 7f2cf0bb2700 1 --2- 172.21.15.155:0/4162188280 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2cec0f6510 0x7f2cec1067e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f2cdc005010 tx=0x7f2cdc00b040).stop 2024-02-10T09:08:24.751 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:24.749+0000 7f2cf0bb2700 1 -- 172.21.15.155:0/4162188280 shutdown_connections 2024-02-10T09:08:24.751 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:24.749+0000 7f2cf0bb2700 1 --2- 172.21.15.155:0/4162188280 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f2cd402ff10 0x7f2cd40323d0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:08:24.751 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:24.749+0000 7f2cf0bb2700 1 --2- 172.21.15.155:0/4162188280 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2cec0f6510 0x7f2cec1067e0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:08:24.751 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:24.749+0000 7f2cf0bb2700 1 -- 172.21.15.155:0/4162188280 >> 172.21.15.155:0/4162188280 conn(0x7f2cec0f1c70 msgr2=0x7f2cec0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:08:24.751 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:24.749+0000 7f2cf0bb2700 1 -- 172.21.15.155:0/4162188280 shutdown_connections 2024-02-10T09:08:24.751 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:24.749+0000 7f2cf0bb2700 1 -- 172.21.15.155:0/4162188280 wait complete. 2024-02-10T09:08:24.751 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:08:25.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:24 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:08:24.749040+0000 mon.smithi077 (mon.0) 277 : audit [DBG] from='client.? 172.21.15.155:0/4162188280' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:08:26.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:25 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:08:25.047031+0000 mgr.smithi077.dxntvw (mgr.14162) 297 : cluster [DBG] pgmap v283: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:08:26.634 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:08:26.634 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:08:27.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:27 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:08:27.047384+0000 mgr.smithi077.dxntvw (mgr.14162) 298 : cluster [DBG] pgmap v284: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:08:28.778 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:28.775+0000 7fad846b7700 1 -- 172.21.15.155:0/3418680956 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad7c0f5410 msgr2=0x7fad7c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:08:28.778 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:28.775+0000 7fad846b7700 1 --2- 172.21.15.155:0/3418680956 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad7c0f5410 0x7fad7c0f5830 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7fad78009fb0 tx=0x7fad78009b30).stop 2024-02-10T09:08:28.778 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:28.775+0000 7fad846b7700 1 -- 172.21.15.155:0/3418680956 shutdown_connections 2024-02-10T09:08:28.778 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:28.775+0000 7fad846b7700 1 --2- 172.21.15.155:0/3418680956 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad7c0f5410 0x7fad7c0f5830 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:08:28.778 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:28.775+0000 7fad846b7700 1 -- 172.21.15.155:0/3418680956 >> 172.21.15.155:0/3418680956 conn(0x7fad7c0f0b70 msgr2=0x7fad7c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:08:28.778 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:28.775+0000 7fad846b7700 1 -- 172.21.15.155:0/3418680956 shutdown_connections 2024-02-10T09:08:28.778 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:28.775+0000 7fad846b7700 1 -- 172.21.15.155:0/3418680956 wait complete. 2024-02-10T09:08:28.778 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:28.775+0000 7fad846b7700 1 Processor -- start 2024-02-10T09:08:28.778 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:28.775+0000 7fad846b7700 1 -- start start 2024-02-10T09:08:28.778 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:28.776+0000 7fad846b7700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad7c0f5410 0x7fad7c105720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:08:28.778 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:28.776+0000 7fad846b7700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad7c105c10 con 0x7fad7c0f5410 2024-02-10T09:08:28.779 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:28.776+0000 7fad82453700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad7c0f5410 0x7fad7c105720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:08:28.779 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:28.776+0000 7fad82453700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad7c0f5410 0x7fad7c105720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:57324/0 (socket says 172.21.15.155:57324) 2024-02-10T09:08:28.779 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:28.776+0000 7fad82453700 1 -- 172.21.15.155:0/323666710 learned_addr learned my addr 172.21.15.155:0/323666710 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:08:28.779 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:28.776+0000 7fad82453700 1 -- 172.21.15.155:0/323666710 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad78005040 con 0x7fad7c0f5410 2024-02-10T09:08:28.779 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:28.776+0000 7fad82453700 1 --2- 172.21.15.155:0/323666710 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad7c0f5410 0x7fad7c105720 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7fad78005010 tx=0x7fad7800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:08:28.779 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:28.777+0000 7fad72ffd700 1 -- 172.21.15.155:0/323666710 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fad78009320 con 0x7fad7c0f5410 2024-02-10T09:08:28.779 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:28.777+0000 7fad72ffd700 1 -- 172.21.15.155:0/323666710 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fad78013380 con 0x7fad7c0f5410 2024-02-10T09:08:28.779 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:28.777+0000 7fad846b7700 1 -- 172.21.15.155:0/323666710 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad7c105e10 con 0x7fad7c0f5410 2024-02-10T09:08:28.779 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:28.777+0000 7fad72ffd700 1 -- 172.21.15.155:0/323666710 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fad78014e50 con 0x7fad7c0f5410 2024-02-10T09:08:28.779 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:28.777+0000 7fad846b7700 1 -- 172.21.15.155:0/323666710 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad7c1062a0 con 0x7fad7c0f5410 2024-02-10T09:08:28.780 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:28.777+0000 7fad72ffd700 1 -- 172.21.15.155:0/323666710 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fad780095f0 con 0x7fad7c0f5410 2024-02-10T09:08:28.780 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:28.778+0000 7fad846b7700 1 -- 172.21.15.155:0/323666710 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad7c045ad0 con 0x7fad7c0f5410 2024-02-10T09:08:28.780 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:28.778+0000 7fad72ffd700 1 --2- 172.21.15.155:0/323666710 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fad6802ff10 0x7fad680323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:08:28.780 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:28.778+0000 7fad72ffd700 1 -- 172.21.15.155:0/323666710 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fad78040f70 con 0x7fad7c0f5410 2024-02-10T09:08:28.780 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:28.778+0000 7fad81c52700 1 --2- 172.21.15.155:0/323666710 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fad6802ff10 0x7fad680323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:08:28.780 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:28.778+0000 7fad81c52700 1 --2- 172.21.15.155:0/323666710 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fad6802ff10 0x7fad680323d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fad6c0096a0 tx=0x7fad6c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:08:28.783 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:28.782+0000 7fad72ffd700 1 -- 172.21.15.155:0/323666710 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fad78014420 con 0x7fad7c0f5410 2024-02-10T09:08:28.989 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:28.988+0000 7fad846b7700 1 -- 172.21.15.155:0/323666710 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fad7c107150 con 0x7fad7c0f5410 2024-02-10T09:08:28.990 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:28.988+0000 7fad72ffd700 1 -- 172.21.15.155:0/323666710 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fad7801d980 con 0x7fad7c0f5410 2024-02-10T09:08:28.991 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:08:28.991 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:08:28.992 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:28.990+0000 7fad846b7700 1 -- 172.21.15.155:0/323666710 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fad6802ff10 msgr2=0x7fad680323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:08:28.992 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:28.990+0000 7fad846b7700 1 --2- 172.21.15.155:0/323666710 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fad6802ff10 0x7fad680323d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fad6c0096a0 tx=0x7fad6c006b40).stop 2024-02-10T09:08:28.992 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:28.990+0000 7fad846b7700 1 -- 172.21.15.155:0/323666710 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad7c0f5410 msgr2=0x7fad7c105720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:08:28.992 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:28.990+0000 7fad846b7700 1 --2- 172.21.15.155:0/323666710 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad7c0f5410 0x7fad7c105720 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7fad78005010 tx=0x7fad7800b040).stop 2024-02-10T09:08:28.992 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:28.990+0000 7fad846b7700 1 -- 172.21.15.155:0/323666710 shutdown_connections 2024-02-10T09:08:28.992 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:28.990+0000 7fad846b7700 1 --2- 172.21.15.155:0/323666710 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fad6802ff10 0x7fad680323d0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:08:28.992 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:28.990+0000 7fad846b7700 1 --2- 172.21.15.155:0/323666710 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fad7c0f5410 0x7fad7c105720 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:08:28.992 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:28.990+0000 7fad846b7700 1 -- 172.21.15.155:0/323666710 >> 172.21.15.155:0/323666710 conn(0x7fad7c0f0b70 msgr2=0x7fad7c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:08:28.992 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:28.990+0000 7fad846b7700 1 -- 172.21.15.155:0/323666710 shutdown_connections 2024-02-10T09:08:28.992 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:28.990+0000 7fad846b7700 1 -- 172.21.15.155:0/323666710 wait complete. 2024-02-10T09:08:28.992 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:08:29.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:29 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:08:28.989983+0000 mon.smithi077 (mon.0) 278 : audit [DBG] from='client.? 172.21.15.155:0/323666710' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:08:30.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:30 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:08:29.047631+0000 mgr.smithi077.dxntvw (mgr.14162) 299 : cluster [DBG] pgmap v285: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:08:30.827 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:08:30.827 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:08:31.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:31 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:08:31.047932+0000 mgr.smithi077.dxntvw (mgr.14162) 300 : cluster [DBG] pgmap v286: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:08:33.011 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:32.994+0000 7f7c9176b700 1 -- 172.21.15.155:0/1437274844 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c8c0f6510 msgr2=0x7f7c8c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:08:33.011 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:32.994+0000 7f7c9176b700 1 --2- 172.21.15.155:0/1437274844 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c8c0f6510 0x7f7c8c0f6930 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f7c7c009fb0 tx=0x7f7c7c009b30).stop 2024-02-10T09:08:33.011 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:32.994+0000 7f7c9176b700 1 -- 172.21.15.155:0/1437274844 shutdown_connections 2024-02-10T09:08:33.011 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:32.994+0000 7f7c9176b700 1 --2- 172.21.15.155:0/1437274844 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c8c0f6510 0x7f7c8c0f6930 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:08:33.011 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:32.994+0000 7f7c9176b700 1 -- 172.21.15.155:0/1437274844 >> 172.21.15.155:0/1437274844 conn(0x7f7c8c0f1c70 msgr2=0x7f7c8c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:08:33.011 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:32.994+0000 7f7c9176b700 1 -- 172.21.15.155:0/1437274844 shutdown_connections 2024-02-10T09:08:33.012 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:32.994+0000 7f7c9176b700 1 -- 172.21.15.155:0/1437274844 wait complete. 2024-02-10T09:08:33.012 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:32.994+0000 7f7c9176b700 1 Processor -- start 2024-02-10T09:08:33.012 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:32.994+0000 7f7c9176b700 1 -- start start 2024-02-10T09:08:33.012 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:32.995+0000 7f7c9176b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c8c0f6510 0x7f7c8c1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:08:33.012 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:32.995+0000 7f7c9176b700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c8c106cd0 con 0x7f7c8c0f6510 2024-02-10T09:08:33.012 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:32.995+0000 7f7c8affd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c8c0f6510 0x7f7c8c1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:08:33.012 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:32.995+0000 7f7c8affd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c8c0f6510 0x7f7c8c1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:42138/0 (socket says 172.21.15.155:42138) 2024-02-10T09:08:33.012 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:32.995+0000 7f7c8affd700 1 -- 172.21.15.155:0/3877087403 learned_addr learned my addr 172.21.15.155:0/3877087403 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:08:33.012 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:32.995+0000 7f7c8affd700 1 -- 172.21.15.155:0/3877087403 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c7c005040 con 0x7f7c8c0f6510 2024-02-10T09:08:33.012 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:32.995+0000 7f7c8affd700 1 --2- 172.21.15.155:0/3877087403 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c8c0f6510 0x7f7c8c1067e0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f7c7c005010 tx=0x7f7c7c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:08:33.012 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:32.995+0000 7f7c737fe700 1 -- 172.21.15.155:0/3877087403 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c7c009320 con 0x7f7c8c0f6510 2024-02-10T09:08:33.012 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:32.995+0000 7f7c9176b700 1 -- 172.21.15.155:0/3877087403 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c8c106ed0 con 0x7f7c8c0f6510 2024-02-10T09:08:33.012 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:32.995+0000 7f7c737fe700 1 -- 172.21.15.155:0/3877087403 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7c7c013380 con 0x7f7c8c0f6510 2024-02-10T09:08:33.013 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:32.995+0000 7f7c737fe700 1 -- 172.21.15.155:0/3877087403 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c7c014e50 con 0x7f7c8c0f6510 2024-02-10T09:08:33.013 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:32.996+0000 7f7c9176b700 1 -- 172.21.15.155:0/3877087403 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c8c107360 con 0x7f7c8c0f6510 2024-02-10T09:08:33.013 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:32.996+0000 7f7c737fe700 1 -- 172.21.15.155:0/3877087403 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7c7c0095f0 con 0x7f7c8c0f6510 2024-02-10T09:08:33.013 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:32.996+0000 7f7c9176b700 1 -- 172.21.15.155:0/3877087403 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c8c045ad0 con 0x7f7c8c0f6510 2024-02-10T09:08:33.013 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:32.996+0000 7f7c737fe700 1 --2- 172.21.15.155:0/3877087403 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f7c7402ff10 0x7f7c740323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:08:33.013 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:32.997+0000 7f7c737fe700 1 -- 172.21.15.155:0/3877087403 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7c7c040e90 con 0x7f7c8c0f6510 2024-02-10T09:08:33.013 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:32.997+0000 7f7c8a7fc700 1 --2- 172.21.15.155:0/3877087403 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f7c7402ff10 0x7f7c740323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:08:33.013 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:32.997+0000 7f7c8a7fc700 1 --2- 172.21.15.155:0/3877087403 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f7c7402ff10 0x7f7c740323d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f7c800096a0 tx=0x7f7c80006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:08:33.013 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:33.001+0000 7f7c737fe700 1 -- 172.21.15.155:0/3877087403 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7c7c014420 con 0x7f7c8c0f6510 2024-02-10T09:08:33.209 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:33.208+0000 7f7c9176b700 1 -- 172.21.15.155:0/3877087403 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7c8c108210 con 0x7f7c8c0f6510 2024-02-10T09:08:33.210 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:33.209+0000 7f7c737fe700 1 -- 172.21.15.155:0/3877087403 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7c7c010230 con 0x7f7c8c0f6510 2024-02-10T09:08:33.210 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:08:33.210 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:08:33.212 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:33.210+0000 7f7c9176b700 1 -- 172.21.15.155:0/3877087403 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f7c7402ff10 msgr2=0x7f7c740323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:08:33.212 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:33.210+0000 7f7c9176b700 1 --2- 172.21.15.155:0/3877087403 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f7c7402ff10 0x7f7c740323d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f7c800096a0 tx=0x7f7c80006b40).stop 2024-02-10T09:08:33.212 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:33.210+0000 7f7c9176b700 1 -- 172.21.15.155:0/3877087403 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c8c0f6510 msgr2=0x7f7c8c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:08:33.212 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:33.210+0000 7f7c9176b700 1 --2- 172.21.15.155:0/3877087403 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c8c0f6510 0x7f7c8c1067e0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f7c7c005010 tx=0x7f7c7c00b040).stop 2024-02-10T09:08:33.212 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:33.210+0000 7f7c9176b700 1 -- 172.21.15.155:0/3877087403 shutdown_connections 2024-02-10T09:08:33.212 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:33.210+0000 7f7c9176b700 1 --2- 172.21.15.155:0/3877087403 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f7c7402ff10 0x7f7c740323d0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:08:33.212 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:33.210+0000 7f7c9176b700 1 --2- 172.21.15.155:0/3877087403 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7c8c0f6510 0x7f7c8c1067e0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:08:33.212 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:33.210+0000 7f7c9176b700 1 -- 172.21.15.155:0/3877087403 >> 172.21.15.155:0/3877087403 conn(0x7f7c8c0f1c70 msgr2=0x7f7c8c0f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:08:33.212 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:33.210+0000 7f7c9176b700 1 -- 172.21.15.155:0/3877087403 shutdown_connections 2024-02-10T09:08:33.212 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:33.210+0000 7f7c9176b700 1 -- 172.21.15.155:0/3877087403 wait complete. 2024-02-10T09:08:33.213 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:08:34.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:33 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:08:33.048287+0000 mgr.smithi077.dxntvw (mgr.14162) 301 : cluster [DBG] pgmap v287: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:08:34.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:33 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:08:33.210050+0000 mon.smithi077 (mon.0) 279 : audit [DBG] from='client.? 172.21.15.155:0/3877087403' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:08:35.097 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:08:35.097 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:08:35.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:35 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:08:35.048566+0000 mgr.smithi077.dxntvw (mgr.14162) 302 : cluster [DBG] pgmap v288: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:08:37.329 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:37.327+0000 7ff19e632700 1 -- 172.21.15.155:0/733775207 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1980f7ae0 msgr2=0x7ff1980f7f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:08:37.330 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:37.327+0000 7ff19e632700 1 --2- 172.21.15.155:0/733775207 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1980f7ae0 0x7ff1980f7f00 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7ff180009fb0 tx=0x7ff180009b30).stop 2024-02-10T09:08:37.330 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:37.327+0000 7ff19e632700 1 -- 172.21.15.155:0/733775207 shutdown_connections 2024-02-10T09:08:37.330 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:37.327+0000 7ff19e632700 1 --2- 172.21.15.155:0/733775207 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1980f7ae0 0x7ff1980f7f00 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:08:37.330 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:37.327+0000 7ff19e632700 1 -- 172.21.15.155:0/733775207 >> 172.21.15.155:0/733775207 conn(0x7ff1980f0260 msgr2=0x7ff1980f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:08:37.330 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:37.327+0000 7ff19e632700 1 -- 172.21.15.155:0/733775207 shutdown_connections 2024-02-10T09:08:37.330 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:37.327+0000 7ff19e632700 1 -- 172.21.15.155:0/733775207 wait complete. 2024-02-10T09:08:37.330 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:37.328+0000 7ff19e632700 1 Processor -- start 2024-02-10T09:08:37.330 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:37.328+0000 7ff19e632700 1 -- start start 2024-02-10T09:08:37.330 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:37.328+0000 7ff19e632700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1980f7ae0 0x7ff1980f73b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:08:37.330 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:37.328+0000 7ff19e632700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1980f5830 con 0x7ff1980f7ae0 2024-02-10T09:08:37.330 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:37.329+0000 7ff197fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1980f7ae0 0x7ff1980f73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:08:37.330 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:37.329+0000 7ff197fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1980f7ae0 0x7ff1980f73b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:42156/0 (socket says 172.21.15.155:42156) 2024-02-10T09:08:37.331 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:37.329+0000 7ff197fff700 1 -- 172.21.15.155:0/2941799989 learned_addr learned my addr 172.21.15.155:0/2941799989 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:08:37.332 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:37.329+0000 7ff197fff700 1 -- 172.21.15.155:0/2941799989 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff180005040 con 0x7ff1980f7ae0 2024-02-10T09:08:37.332 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:37.329+0000 7ff197fff700 1 --2- 172.21.15.155:0/2941799989 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1980f7ae0 0x7ff1980f73b0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7ff180005010 tx=0x7ff18000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:08:37.332 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:37.329+0000 7ff194ff9700 1 -- 172.21.15.155:0/2941799989 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff180009320 con 0x7ff1980f7ae0 2024-02-10T09:08:37.332 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:37.329+0000 7ff194ff9700 1 -- 172.21.15.155:0/2941799989 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff180013380 con 0x7ff1980f7ae0 2024-02-10T09:08:37.332 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:37.329+0000 7ff19e632700 1 -- 172.21.15.155:0/2941799989 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1980f59d0 con 0x7ff1980f7ae0 2024-02-10T09:08:37.332 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:37.329+0000 7ff194ff9700 1 -- 172.21.15.155:0/2941799989 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff180014e50 con 0x7ff1980f7ae0 2024-02-10T09:08:37.332 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:37.329+0000 7ff19e632700 1 -- 172.21.15.155:0/2941799989 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1980f5e60 con 0x7ff1980f7ae0 2024-02-10T09:08:37.332 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:37.330+0000 7ff194ff9700 1 -- 172.21.15.155:0/2941799989 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff180014420 con 0x7ff1980f7ae0 2024-02-10T09:08:37.333 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:37.330+0000 7ff19e632700 1 -- 172.21.15.155:0/2941799989 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff198059100 con 0x7ff1980f7ae0 2024-02-10T09:08:37.333 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:37.330+0000 7ff194ff9700 1 --2- 172.21.15.155:0/2941799989 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7ff18402ff10 0x7ff1840323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:08:37.333 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:37.330+0000 7ff194ff9700 1 -- 172.21.15.155:0/2941799989 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff180040eb0 con 0x7ff1980f7ae0 2024-02-10T09:08:37.333 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:37.330+0000 7ff1977fe700 1 --2- 172.21.15.155:0/2941799989 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7ff18402ff10 0x7ff1840323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:08:37.333 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:37.331+0000 7ff1977fe700 1 --2- 172.21.15.155:0/2941799989 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7ff18402ff10 0x7ff1840323d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7ff1880096a0 tx=0x7ff188006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:08:37.336 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:37.335+0000 7ff194ff9700 1 -- 172.21.15.155:0/2941799989 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff1800095f0 con 0x7ff1980f7ae0 2024-02-10T09:08:37.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:37 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:08:37.048814+0000 mgr.smithi077.dxntvw (mgr.14162) 303 : cluster [DBG] pgmap v289: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:08:37.539 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:37.538+0000 7ff19e632700 1 -- 172.21.15.155:0/2941799989 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff1980fc100 con 0x7ff1980f7ae0 2024-02-10T09:08:37.540 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:37.539+0000 7ff194ff9700 1 -- 172.21.15.155:0/2941799989 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff18001db70 con 0x7ff1980f7ae0 2024-02-10T09:08:37.540 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:08:37.540 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:08:37.542 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:37.540+0000 7ff19e632700 1 -- 172.21.15.155:0/2941799989 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7ff18402ff10 msgr2=0x7ff1840323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:08:37.542 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:37.540+0000 7ff19e632700 1 --2- 172.21.15.155:0/2941799989 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7ff18402ff10 0x7ff1840323d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7ff1880096a0 tx=0x7ff188006b40).stop 2024-02-10T09:08:37.542 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:37.540+0000 7ff19e632700 1 -- 172.21.15.155:0/2941799989 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1980f7ae0 msgr2=0x7ff1980f73b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:08:37.542 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:37.540+0000 7ff19e632700 1 --2- 172.21.15.155:0/2941799989 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1980f7ae0 0x7ff1980f73b0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7ff180005010 tx=0x7ff18000b040).stop 2024-02-10T09:08:37.542 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:37.541+0000 7ff19e632700 1 -- 172.21.15.155:0/2941799989 shutdown_connections 2024-02-10T09:08:37.542 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:37.541+0000 7ff19e632700 1 --2- 172.21.15.155:0/2941799989 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7ff18402ff10 0x7ff1840323d0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:08:37.542 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:37.541+0000 7ff19e632700 1 --2- 172.21.15.155:0/2941799989 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff1980f7ae0 0x7ff1980f73b0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:08:37.542 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:37.541+0000 7ff19e632700 1 -- 172.21.15.155:0/2941799989 >> 172.21.15.155:0/2941799989 conn(0x7ff1980f0260 msgr2=0x7ff1980f0d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:08:37.542 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:37.541+0000 7ff19e632700 1 -- 172.21.15.155:0/2941799989 shutdown_connections 2024-02-10T09:08:37.542 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:37.541+0000 7ff19e632700 1 -- 172.21.15.155:0/2941799989 wait complete. 2024-02-10T09:08:37.543 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:08:38.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:38 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:08:37.540550+0000 mon.smithi077 (mon.0) 280 : audit [DBG] from='client.? 172.21.15.155:0/2941799989' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:08:39.363 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:08:39.363 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:08:39.398 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:39 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:08:39.049163+0000 mgr.smithi077.dxntvw (mgr.14162) 304 : cluster [DBG] pgmap v290: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:08:41.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:41 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:08:41.049458+0000 mgr.smithi077.dxntvw (mgr.14162) 305 : cluster [DBG] pgmap v291: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:08:41.506 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:41.503+0000 7f2324a28700 1 -- 172.21.15.155:0/572009306 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23200f62a0 msgr2=0x7f23200f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:08:41.506 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:41.503+0000 7f2324a28700 1 --2- 172.21.15.155:0/572009306 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23200f62a0 0x7f23200f66c0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7f2314009fb0 tx=0x7f2314009b30).stop 2024-02-10T09:08:41.506 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:41.503+0000 7f2324a28700 1 -- 172.21.15.155:0/572009306 shutdown_connections 2024-02-10T09:08:41.506 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:41.503+0000 7f2324a28700 1 --2- 172.21.15.155:0/572009306 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23200f62a0 0x7f23200f66c0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:08:41.507 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:41.503+0000 7f2324a28700 1 -- 172.21.15.155:0/572009306 >> 172.21.15.155:0/572009306 conn(0x7f23200f1aa0 msgr2=0x7f23200f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:08:41.507 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:41.503+0000 7f2324a28700 1 -- 172.21.15.155:0/572009306 shutdown_connections 2024-02-10T09:08:41.507 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:41.503+0000 7f2324a28700 1 -- 172.21.15.155:0/572009306 wait complete. 2024-02-10T09:08:41.507 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:41.504+0000 7f2324a28700 1 Processor -- start 2024-02-10T09:08:41.507 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:41.504+0000 7f2324a28700 1 -- start start 2024-02-10T09:08:41.507 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:41.504+0000 7f2324a28700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23200f62a0 0x7f23201087d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:08:41.507 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:41.504+0000 7f2324a28700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2320108cc0 con 0x7f23200f62a0 2024-02-10T09:08:41.507 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:41.504+0000 7f231f7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23200f62a0 0x7f23201087d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:08:41.507 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:41.504+0000 7f231f7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23200f62a0 0x7f23201087d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:42168/0 (socket says 172.21.15.155:42168) 2024-02-10T09:08:41.507 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:41.504+0000 7f231f7fe700 1 -- 172.21.15.155:0/3108432572 learned_addr learned my addr 172.21.15.155:0/3108432572 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:08:41.507 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:41.505+0000 7f231f7fe700 1 -- 172.21.15.155:0/3108432572 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2314005040 con 0x7f23200f62a0 2024-02-10T09:08:41.507 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:41.505+0000 7f231f7fe700 1 --2- 172.21.15.155:0/3108432572 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23200f62a0 0x7f23201087d0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f23140089f0 tx=0x7f231400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:08:41.507 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:41.505+0000 7f2307fff700 1 -- 172.21.15.155:0/3108432572 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2314009320 con 0x7f23200f62a0 2024-02-10T09:08:41.508 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:41.505+0000 7f2307fff700 1 -- 172.21.15.155:0/3108432572 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2314013380 con 0x7f23200f62a0 2024-02-10T09:08:41.508 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:41.505+0000 7f2324a28700 1 -- 172.21.15.155:0/3108432572 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2320108ec0 con 0x7f23200f62a0 2024-02-10T09:08:41.508 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:41.505+0000 7f2307fff700 1 -- 172.21.15.155:0/3108432572 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2314014e20 con 0x7f23200f62a0 2024-02-10T09:08:41.508 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:41.505+0000 7f2324a28700 1 -- 172.21.15.155:0/3108432572 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2320109350 con 0x7f23200f62a0 2024-02-10T09:08:41.509 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:41.506+0000 7f2307fff700 1 -- 172.21.15.155:0/3108432572 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2314014420 con 0x7f23200f62a0 2024-02-10T09:08:41.509 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:41.506+0000 7f2307fff700 1 --2- 172.21.15.155:0/3108432572 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f230802ff10 0x7f23080323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:08:41.509 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:41.506+0000 7f2307fff700 1 -- 172.21.15.155:0/3108432572 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2314040ff0 con 0x7f23200f62a0 2024-02-10T09:08:41.509 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:41.506+0000 7f2324a28700 1 -- 172.21.15.155:0/3108432572 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2320045ad0 con 0x7f23200f62a0 2024-02-10T09:08:41.509 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:41.506+0000 7f231effd700 1 --2- 172.21.15.155:0/3108432572 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f230802ff10 0x7f23080323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:08:41.509 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:41.506+0000 7f231effd700 1 --2- 172.21.15.155:0/3108432572 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f230802ff10 0x7f23080323d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f23100096a0 tx=0x7f2310006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:08:41.512 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:41.511+0000 7f2307fff700 1 -- 172.21.15.155:0/3108432572 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f23140095f0 con 0x7f23200f62a0 2024-02-10T09:08:41.719 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:41.718+0000 7f2324a28700 1 -- 172.21.15.155:0/3108432572 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f232010a0a0 con 0x7f23200f62a0 2024-02-10T09:08:41.719 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:41.718+0000 7f2307fff700 1 -- 172.21.15.155:0/3108432572 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f231401db70 con 0x7f23200f62a0 2024-02-10T09:08:41.720 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:08:41.720 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:08:41.721 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:41.720+0000 7f2324a28700 1 -- 172.21.15.155:0/3108432572 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f230802ff10 msgr2=0x7f23080323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:08:41.721 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:41.720+0000 7f2324a28700 1 --2- 172.21.15.155:0/3108432572 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f230802ff10 0x7f23080323d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f23100096a0 tx=0x7f2310006b40).stop 2024-02-10T09:08:41.721 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:41.720+0000 7f2324a28700 1 -- 172.21.15.155:0/3108432572 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23200f62a0 msgr2=0x7f23201087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:08:41.721 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:41.720+0000 7f2324a28700 1 --2- 172.21.15.155:0/3108432572 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23200f62a0 0x7f23201087d0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f23140089f0 tx=0x7f231400b040).stop 2024-02-10T09:08:41.721 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:41.720+0000 7f2324a28700 1 -- 172.21.15.155:0/3108432572 shutdown_connections 2024-02-10T09:08:41.722 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:41.720+0000 7f2324a28700 1 --2- 172.21.15.155:0/3108432572 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f230802ff10 0x7f23080323d0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:08:41.722 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:41.720+0000 7f2324a28700 1 --2- 172.21.15.155:0/3108432572 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f23200f62a0 0x7f23201087d0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:08:41.722 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:41.720+0000 7f2324a28700 1 -- 172.21.15.155:0/3108432572 >> 172.21.15.155:0/3108432572 conn(0x7f23200f1aa0 msgr2=0x7f23200f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:08:41.722 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:41.720+0000 7f2324a28700 1 -- 172.21.15.155:0/3108432572 shutdown_connections 2024-02-10T09:08:41.722 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:41.720+0000 7f2324a28700 1 -- 172.21.15.155:0/3108432572 wait complete. 2024-02-10T09:08:41.722 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:08:42.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:42 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:08:41.719985+0000 mon.smithi077 (mon.0) 281 : audit [DBG] from='client.? 172.21.15.155:0/3108432572' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:08:43.625 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:08:43.626 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:08:44.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:43 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:08:43.049772+0000 mgr.smithi077.dxntvw (mgr.14162) 306 : cluster [DBG] pgmap v292: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:08:45.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:45 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:08:45.050058+0000 mgr.smithi077.dxntvw (mgr.14162) 307 : cluster [DBG] pgmap v293: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:08:45.802 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:45.790+0000 7fdf4bddf700 1 -- 172.21.15.155:0/1194748932 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf440f5410 msgr2=0x7fdf440f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:08:45.803 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:45.790+0000 7fdf4bddf700 1 --2- 172.21.15.155:0/1194748932 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf440f5410 0x7fdf440f5830 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7fdf34009fb0 tx=0x7fdf34009b30).stop 2024-02-10T09:08:45.803 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:45.790+0000 7fdf4bddf700 1 -- 172.21.15.155:0/1194748932 shutdown_connections 2024-02-10T09:08:45.803 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:45.790+0000 7fdf4bddf700 1 --2- 172.21.15.155:0/1194748932 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf440f5410 0x7fdf440f5830 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:08:45.803 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:45.790+0000 7fdf4bddf700 1 -- 172.21.15.155:0/1194748932 >> 172.21.15.155:0/1194748932 conn(0x7fdf440f0b70 msgr2=0x7fdf440f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:08:45.803 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:45.790+0000 7fdf4bddf700 1 -- 172.21.15.155:0/1194748932 shutdown_connections 2024-02-10T09:08:45.803 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:45.790+0000 7fdf4bddf700 1 -- 172.21.15.155:0/1194748932 wait complete. 2024-02-10T09:08:45.803 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:45.791+0000 7fdf4bddf700 1 Processor -- start 2024-02-10T09:08:45.803 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:45.791+0000 7fdf4bddf700 1 -- start start 2024-02-10T09:08:45.803 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:45.791+0000 7fdf4bddf700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf440f5410 0x7fdf441067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:08:45.803 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:45.791+0000 7fdf4bddf700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf44106cc0 con 0x7fdf440f5410 2024-02-10T09:08:45.803 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:45.791+0000 7fdf49b7b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf440f5410 0x7fdf441067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:08:45.803 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:45.791+0000 7fdf49b7b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf440f5410 0x7fdf441067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:36028/0 (socket says 172.21.15.155:36028) 2024-02-10T09:08:45.804 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:45.791+0000 7fdf49b7b700 1 -- 172.21.15.155:0/2421410936 learned_addr learned my addr 172.21.15.155:0/2421410936 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:08:45.804 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:45.792+0000 7fdf49b7b700 1 -- 172.21.15.155:0/2421410936 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf34005040 con 0x7fdf440f5410 2024-02-10T09:08:45.804 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:45.792+0000 7fdf49b7b700 1 --2- 172.21.15.155:0/2421410936 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf440f5410 0x7fdf441067d0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7fdf34005010 tx=0x7fdf3400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:08:45.804 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:45.792+0000 7fdf3a7fc700 1 -- 172.21.15.155:0/2421410936 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf34009320 con 0x7fdf440f5410 2024-02-10T09:08:45.804 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:45.792+0000 7fdf4bddf700 1 -- 172.21.15.155:0/2421410936 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf44106ec0 con 0x7fdf440f5410 2024-02-10T09:08:45.804 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:45.792+0000 7fdf3a7fc700 1 -- 172.21.15.155:0/2421410936 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdf34013380 con 0x7fdf440f5410 2024-02-10T09:08:45.804 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:45.792+0000 7fdf3a7fc700 1 -- 172.21.15.155:0/2421410936 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf34014e50 con 0x7fdf440f5410 2024-02-10T09:08:45.804 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:45.792+0000 7fdf4bddf700 1 -- 172.21.15.155:0/2421410936 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf44107350 con 0x7fdf440f5410 2024-02-10T09:08:45.804 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:45.793+0000 7fdf3a7fc700 1 -- 172.21.15.155:0/2421410936 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdf340095f0 con 0x7fdf440f5410 2024-02-10T09:08:45.804 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:45.793+0000 7fdf4bddf700 1 -- 172.21.15.155:0/2421410936 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf44045ad0 con 0x7fdf440f5410 2024-02-10T09:08:45.804 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:45.793+0000 7fdf3a7fc700 1 --2- 172.21.15.155:0/2421410936 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fdf3002ff10 0x7fdf300323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:08:45.804 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:45.793+0000 7fdf3a7fc700 1 -- 172.21.15.155:0/2421410936 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdf34040e90 con 0x7fdf440f5410 2024-02-10T09:08:45.804 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:45.793+0000 7fdf4937a700 1 --2- 172.21.15.155:0/2421410936 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fdf3002ff10 0x7fdf300323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:08:45.804 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:45.794+0000 7fdf4937a700 1 --2- 172.21.15.155:0/2421410936 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fdf3002ff10 0x7fdf300323d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fdf400096a0 tx=0x7fdf40006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:08:45.805 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:45.798+0000 7fdf3a7fc700 1 -- 172.21.15.155:0/2421410936 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdf34014420 con 0x7fdf440f5410 2024-02-10T09:08:46.003 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:46.003+0000 7fdf4bddf700 1 -- 172.21.15.155:0/2421410936 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdf44108200 con 0x7fdf440f5410 2024-02-10T09:08:46.003 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:46.004+0000 7fdf3a7fc700 1 -- 172.21.15.155:0/2421410936 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdf34010230 con 0x7fdf440f5410 2024-02-10T09:08:46.004 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:08:46.004 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:08:46.005 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:46.005+0000 7fdf4bddf700 1 -- 172.21.15.155:0/2421410936 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fdf3002ff10 msgr2=0x7fdf300323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:08:46.005 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:46.005+0000 7fdf4bddf700 1 --2- 172.21.15.155:0/2421410936 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fdf3002ff10 0x7fdf300323d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fdf400096a0 tx=0x7fdf40006b40).stop 2024-02-10T09:08:46.005 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:46.005+0000 7fdf4bddf700 1 -- 172.21.15.155:0/2421410936 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf440f5410 msgr2=0x7fdf441067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:08:46.006 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:46.005+0000 7fdf4bddf700 1 --2- 172.21.15.155:0/2421410936 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf440f5410 0x7fdf441067d0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7fdf34005010 tx=0x7fdf3400b040).stop 2024-02-10T09:08:46.006 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:46.005+0000 7fdf4bddf700 1 -- 172.21.15.155:0/2421410936 shutdown_connections 2024-02-10T09:08:46.006 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:46.005+0000 7fdf4bddf700 1 --2- 172.21.15.155:0/2421410936 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fdf3002ff10 0x7fdf300323d0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:08:46.006 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:46.005+0000 7fdf4bddf700 1 --2- 172.21.15.155:0/2421410936 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf440f5410 0x7fdf441067d0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:08:46.006 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:46.005+0000 7fdf4bddf700 1 -- 172.21.15.155:0/2421410936 >> 172.21.15.155:0/2421410936 conn(0x7fdf440f0b70 msgr2=0x7fdf4405fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:08:46.006 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:46.005+0000 7fdf4bddf700 1 -- 172.21.15.155:0/2421410936 shutdown_connections 2024-02-10T09:08:46.006 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:46.005+0000 7fdf4bddf700 1 -- 172.21.15.155:0/2421410936 wait complete. 2024-02-10T09:08:46.006 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:08:46.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:46 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:08:46.004076+0000 mon.smithi077 (mon.0) 282 : audit [DBG] from='client.? 172.21.15.155:0/2421410936' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:08:47.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:08:47.050327+0000 mgr.smithi077.dxntvw (mgr.14162) 308 : cluster [DBG] pgmap v294: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:08:47.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:08:47.098229+0000 mon.smithi077 (mon.0) 283 : audit [INF] from='mgr.14162 172.21.15.77:0/2029106299' entity='mgr.smithi077.dxntvw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi077.dxntvw/mirror_snapshot_schedule"}]: dispatch 2024-02-10T09:08:47.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:08:47.104050+0000 mon.smithi077 (mon.0) 284 : audit [INF] from='mgr.14162 172.21.15.77:0/2029106299' entity='mgr.smithi077.dxntvw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi077.dxntvw/trash_purge_schedule"}]: dispatch 2024-02-10T09:08:47.982 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:08:47.983 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:08:49.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cephadm 2024-02-10T09:08:47.918684+0000 mgr.smithi077.dxntvw (mgr.14162) 309 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon prometheus.smithi077 ... 2024-02-10T09:08:49.150 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2024-02-10T09:08:49.150 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2024-02-10T09:08:49.150 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: stat: stderr time="2024-02-10T09:06:55Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:08:49.150 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: stat: stderr time="2024-02-10T09:07:32Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:08:49.150 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: stat: stderr time="2024-02-10T09:08:10Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:08:49.150 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-10T09:08:49.150 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: Traceback (most recent call last): 2024-02-10T09:08:49.150 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: File "/var/lib/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-10T09:08:49.150 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: main() 2024-02-10T09:08:49.150 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: File "/var/lib/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-10T09:08:49.150 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: r = ctx.func(ctx) 2024-02-10T09:08:49.150 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: File "/var/lib/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-10T09:08:49.151 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: return func(ctx) 2024-02-10T09:08:49.151 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: File "/var/lib/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-02-10T09:08:49.151 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-10T09:08:49.151 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: File "/var/lib/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4185, in extract_uid_gid_monitoring 2024-02-10T09:08:49.151 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2024-02-10T09:08:49.151 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: File "/var/lib/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-02-10T09:08:49.151 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: raise RuntimeError('uid/gid not found') 2024-02-10T09:08:49.151 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: RuntimeError: uid/gid not found 2024-02-10T09:08:49.151 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: Traceback (most recent call last): 2024-02-10T09:08:49.151 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-02-10T09:08:49.151 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: yield (conn, connr) 2024-02-10T09:08:49.151 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-02-10T09:08:49.151 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: code, '\n'.join(err))) 2024-02-10T09:08:49.151 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon prometheus.smithi077 ... 2024-02-10T09:08:49.152 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi077 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2024-02-10T09:08:49.152 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2024-02-10T09:08:49.152 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: stat: stderr time="2024-02-10T09:06:55Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:08:49.152 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: stat: stderr time="2024-02-10T09:07:32Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:08:49.152 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: stat: stderr time="2024-02-10T09:08:10Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-10T09:08:49.152 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-10T09:08:49.152 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: Traceback (most recent call last): 2024-02-10T09:08:49.152 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: File "/var/lib/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-10T09:08:49.152 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: main() 2024-02-10T09:08:49.152 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: File "/var/lib/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-10T09:08:49.152 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: r = ctx.func(ctx) 2024-02-10T09:08:49.152 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: File "/var/lib/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-10T09:08:49.152 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: return func(ctx) 2024-02-10T09:08:49.153 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: File "/var/lib/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-02-10T09:08:49.153 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-10T09:08:49.153 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: File "/var/lib/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4185, in extract_uid_gid_monitoring 2024-02-10T09:08:49.153 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2024-02-10T09:08:49.153 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: File "/var/lib/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-02-10T09:08:49.153 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: raise RuntimeError('uid/gid not found') 2024-02-10T09:08:49.153 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: RuntimeError: uid/gid not found 2024-02-10T09:08:49.153 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:08:47.919211+0000 mon.smithi077 (mon.0) 285 : audit [DBG] from='mgr.14162 172.21.15.77:0/2029106299' entity='mgr.smithi077.dxntvw' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T09:08:49.153 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:08:48.140989+0000 mon.smithi077 (mon.0) 286 : audit [INF] from='mgr.14162 172.21.15.77:0/2029106299' entity='mgr.smithi077.dxntvw' 2024-02-10T09:08:49.153 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:08:48.171611+0000 mon.smithi077 (mon.0) 287 : audit [INF] from='mgr.14162 172.21.15.77:0/2029106299' entity='mgr.smithi077.dxntvw' 2024-02-10T09:08:50.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:49 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:08:48.899989+0000 mon.smithi077 (mon.0) 288 : audit [INF] from='mgr.14162 172.21.15.77:0/2029106299' entity='mgr.smithi077.dxntvw' 2024-02-10T09:08:50.439 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:49 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:08:49.050559+0000 mgr.smithi077.dxntvw (mgr.14162) 310 : cluster [DBG] pgmap v295: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:08:50.818 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:50.233+0000 7fc91f6fd700 1 -- 172.21.15.155:0/128372598 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9180f5410 msgr2=0x7fc9180f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:08:50.819 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:50.233+0000 7fc91f6fd700 1 --2- 172.21.15.155:0/128372598 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9180f5410 0x7fc9180f5830 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7fc914009fb0 tx=0x7fc914009b30).stop 2024-02-10T09:08:50.819 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:50.233+0000 7fc91f6fd700 1 -- 172.21.15.155:0/128372598 shutdown_connections 2024-02-10T09:08:50.819 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:50.233+0000 7fc91f6fd700 1 --2- 172.21.15.155:0/128372598 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9180f5410 0x7fc9180f5830 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:08:50.819 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:50.233+0000 7fc91f6fd700 1 -- 172.21.15.155:0/128372598 >> 172.21.15.155:0/128372598 conn(0x7fc9180f0b70 msgr2=0x7fc9180f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:08:50.819 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:50.233+0000 7fc91f6fd700 1 -- 172.21.15.155:0/128372598 shutdown_connections 2024-02-10T09:08:50.819 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:50.233+0000 7fc91f6fd700 1 -- 172.21.15.155:0/128372598 wait complete. 2024-02-10T09:08:50.819 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:50.234+0000 7fc91f6fd700 1 Processor -- start 2024-02-10T09:08:50.819 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:50.234+0000 7fc91f6fd700 1 -- start start 2024-02-10T09:08:50.819 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:50.234+0000 7fc91f6fd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9180f5410 0x7fc9181056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:08:50.819 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:50.234+0000 7fc91f6fd700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc918105bd0 con 0x7fc9180f5410 2024-02-10T09:08:50.819 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:50.234+0000 7fc91d499700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9180f5410 0x7fc9181056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:08:50.820 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:50.234+0000 7fc91d499700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9180f5410 0x7fc9181056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:36046/0 (socket says 172.21.15.155:36046) 2024-02-10T09:08:50.820 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:50.234+0000 7fc91d499700 1 -- 172.21.15.155:0/3258158237 learned_addr learned my addr 172.21.15.155:0/3258158237 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:08:50.820 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:50.234+0000 7fc91d499700 1 -- 172.21.15.155:0/3258158237 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc914005040 con 0x7fc9180f5410 2024-02-10T09:08:50.820 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:50.234+0000 7fc91d499700 1 --2- 172.21.15.155:0/3258158237 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9180f5410 0x7fc9181056e0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7fc914005010 tx=0x7fc91400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:08:50.820 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:50.235+0000 7fc90dffb700 1 -- 172.21.15.155:0/3258158237 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc914009320 con 0x7fc9180f5410 2024-02-10T09:08:50.820 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:50.235+0000 7fc91f6fd700 1 -- 172.21.15.155:0/3258158237 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc918105dd0 con 0x7fc9180f5410 2024-02-10T09:08:50.820 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:50.235+0000 7fc90dffb700 1 -- 172.21.15.155:0/3258158237 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc914013380 con 0x7fc9180f5410 2024-02-10T09:08:50.820 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:50.235+0000 7fc90dffb700 1 -- 172.21.15.155:0/3258158237 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc914014e50 con 0x7fc9180f5410 2024-02-10T09:08:50.820 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:50.235+0000 7fc91f6fd700 1 -- 172.21.15.155:0/3258158237 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc918106260 con 0x7fc9180f5410 2024-02-10T09:08:50.820 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:50.236+0000 7fc90dffb700 1 -- 172.21.15.155:0/3258158237 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc9140095f0 con 0x7fc9180f5410 2024-02-10T09:08:50.820 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:50.236+0000 7fc91f6fd700 1 -- 172.21.15.155:0/3258158237 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc918045ad0 con 0x7fc9180f5410 2024-02-10T09:08:50.820 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:50.236+0000 7fc90dffb700 1 --2- 172.21.15.155:0/3258158237 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fc90402ff10 0x7fc9040323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:08:50.820 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:50.236+0000 7fc90dffb700 1 -- 172.21.15.155:0/3258158237 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc914040e60 con 0x7fc9180f5410 2024-02-10T09:08:50.820 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:50.236+0000 7fc91cc98700 1 --2- 172.21.15.155:0/3258158237 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fc90402ff10 0x7fc9040323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:08:50.821 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:50.236+0000 7fc91cc98700 1 --2- 172.21.15.155:0/3258158237 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fc90402ff10 0x7fc9040323d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fc9080096a0 tx=0x7fc908006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:08:50.821 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:50.240+0000 7fc90dffb700 1 -- 172.21.15.155:0/3258158237 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc914014420 con 0x7fc9180f5410 2024-02-10T09:08:50.821 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:08:50.821 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:08:50.821 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:50.454+0000 7fc91f6fd700 1 -- 172.21.15.155:0/3258158237 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc918107110 con 0x7fc9180f5410 2024-02-10T09:08:50.821 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:50.454+0000 7fc90dffb700 1 -- 172.21.15.155:0/3258158237 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc91401d980 con 0x7fc9180f5410 2024-02-10T09:08:50.821 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:50.456+0000 7fc91f6fd700 1 -- 172.21.15.155:0/3258158237 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fc90402ff10 msgr2=0x7fc9040323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:08:50.822 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:50.456+0000 7fc91f6fd700 1 --2- 172.21.15.155:0/3258158237 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fc90402ff10 0x7fc9040323d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fc9080096a0 tx=0x7fc908006b40).stop 2024-02-10T09:08:50.822 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:50.456+0000 7fc91f6fd700 1 -- 172.21.15.155:0/3258158237 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9180f5410 msgr2=0x7fc9181056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:08:50.822 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:50.456+0000 7fc91f6fd700 1 --2- 172.21.15.155:0/3258158237 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9180f5410 0x7fc9181056e0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7fc914005010 tx=0x7fc91400b040).stop 2024-02-10T09:08:50.822 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:50.456+0000 7fc91f6fd700 1 -- 172.21.15.155:0/3258158237 shutdown_connections 2024-02-10T09:08:50.822 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:50.456+0000 7fc91f6fd700 1 --2- 172.21.15.155:0/3258158237 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fc90402ff10 0x7fc9040323d0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:08:50.822 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:50.456+0000 7fc91f6fd700 1 --2- 172.21.15.155:0/3258158237 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9180f5410 0x7fc9181056e0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:08:50.822 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:50.456+0000 7fc91f6fd700 1 -- 172.21.15.155:0/3258158237 >> 172.21.15.155:0/3258158237 conn(0x7fc9180f0b70 msgr2=0x7fc9180f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:08:50.822 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:50.456+0000 7fc91f6fd700 1 -- 172.21.15.155:0/3258158237 shutdown_connections 2024-02-10T09:08:50.822 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:50.456+0000 7fc91f6fd700 1 -- 172.21.15.155:0/3258158237 wait complete. 2024-02-10T09:08:50.822 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:08:51.649 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:51 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:08:50.185632+0000 mon.smithi077 (mon.0) 289 : audit [INF] from='mgr.14162 172.21.15.77:0/2029106299' entity='mgr.smithi077.dxntvw' 2024-02-10T09:08:51.649 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:51 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:08:50.454856+0000 mon.smithi077 (mon.0) 290 : audit [DBG] from='client.? 172.21.15.155:0/3258158237' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:08:51.649 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:51 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:08:51.050825+0000 mgr.smithi077.dxntvw (mgr.14162) 311 : cluster [DBG] pgmap v296: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:08:52.674 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:08:52.675 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:08:54.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:53 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:08:53.051115+0000 mgr.smithi077.dxntvw (mgr.14162) 312 : cluster [DBG] pgmap v297: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:08:55.400 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:55 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:08:55.051379+0000 mgr.smithi077.dxntvw (mgr.14162) 313 : cluster [DBG] pgmap v298: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:08:55.400 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:55 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:08:55.076671+0000 mon.smithi077 (mon.0) 291 : audit [INF] from='mgr.14162 172.21.15.77:0/2029106299' entity='mgr.smithi077.dxntvw' 2024-02-10T09:08:55.564 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:55.563+0000 7f5520ce9700 1 -- 172.21.15.155:0/3365902844 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f551c0f6510 msgr2=0x7f551c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:08:55.564 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:55.563+0000 7f5520ce9700 1 --2- 172.21.15.155:0/3365902844 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f551c0f6510 0x7f551c0f6930 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f5504009fb0 tx=0x7f5504009b30).stop 2024-02-10T09:08:55.565 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:55.563+0000 7f5520ce9700 1 -- 172.21.15.155:0/3365902844 shutdown_connections 2024-02-10T09:08:55.565 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:55.563+0000 7f5520ce9700 1 --2- 172.21.15.155:0/3365902844 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f551c0f6510 0x7f551c0f6930 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:08:55.565 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:55.563+0000 7f5520ce9700 1 -- 172.21.15.155:0/3365902844 >> 172.21.15.155:0/3365902844 conn(0x7f551c0f1c70 msgr2=0x7f551c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:08:55.565 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:55.563+0000 7f5520ce9700 1 -- 172.21.15.155:0/3365902844 shutdown_connections 2024-02-10T09:08:55.565 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:55.563+0000 7f5520ce9700 1 -- 172.21.15.155:0/3365902844 wait complete. 2024-02-10T09:08:55.565 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:55.564+0000 7f5520ce9700 1 Processor -- start 2024-02-10T09:08:55.565 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:55.564+0000 7f5520ce9700 1 -- start start 2024-02-10T09:08:55.565 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:55.564+0000 7f5520ce9700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f551c0f6510 0x7f551c1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:08:55.565 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:55.564+0000 7f5520ce9700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f551c106cd0 con 0x7f551c0f6510 2024-02-10T09:08:55.565 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:55.564+0000 7f551a59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f551c0f6510 0x7f551c1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:08:55.565 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:55.564+0000 7f551a59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f551c0f6510 0x7f551c1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:51018/0 (socket says 172.21.15.155:51018) 2024-02-10T09:08:55.565 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:55.564+0000 7f551a59c700 1 -- 172.21.15.155:0/997666434 learned_addr learned my addr 172.21.15.155:0/997666434 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:08:55.565 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:55.564+0000 7f551a59c700 1 -- 172.21.15.155:0/997666434 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5504005040 con 0x7f551c0f6510 2024-02-10T09:08:55.565 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:55.565+0000 7f551a59c700 1 --2- 172.21.15.155:0/997666434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f551c0f6510 0x7f551c1067e0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f55040089f0 tx=0x7f550400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:08:55.567 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:55.565+0000 7f5512ffd700 1 -- 172.21.15.155:0/997666434 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5504009320 con 0x7f551c0f6510 2024-02-10T09:08:55.567 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:55.565+0000 7f5520ce9700 1 -- 172.21.15.155:0/997666434 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f551c106ed0 con 0x7f551c0f6510 2024-02-10T09:08:55.567 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:55.565+0000 7f5512ffd700 1 -- 172.21.15.155:0/997666434 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5504013380 con 0x7f551c0f6510 2024-02-10T09:08:55.567 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:55.565+0000 7f5512ffd700 1 -- 172.21.15.155:0/997666434 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f550401ed80 con 0x7f551c0f6510 2024-02-10T09:08:55.567 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:55.565+0000 7f5520ce9700 1 -- 172.21.15.155:0/997666434 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f551c107360 con 0x7f551c0f6510 2024-02-10T09:08:55.567 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:55.566+0000 7f5512ffd700 1 -- 172.21.15.155:0/997666434 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f550401e420 con 0x7f551c0f6510 2024-02-10T09:08:55.567 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:55.566+0000 7f5520ce9700 1 -- 172.21.15.155:0/997666434 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f551c045ad0 con 0x7f551c0f6510 2024-02-10T09:08:55.567 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:55.566+0000 7f5512ffd700 1 --2- 172.21.15.155:0/997666434 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f550802ff60 0x7f5508032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:08:55.567 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:55.566+0000 7f5512ffd700 1 -- 172.21.15.155:0/997666434 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5504040e70 con 0x7f551c0f6510 2024-02-10T09:08:55.567 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:55.566+0000 7f5519d9b700 1 --2- 172.21.15.155:0/997666434 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f550802ff60 0x7f5508032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:08:55.567 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:55.566+0000 7f5519d9b700 1 --2- 172.21.15.155:0/997666434 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f550802ff60 0x7f5508032420 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f550c0096a0 tx=0x7f550c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:08:55.570 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:55.571+0000 7f5512ffd700 1 -- 172.21.15.155:0/997666434 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f550400e2a0 con 0x7f551c0f6510 2024-02-10T09:08:55.785 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:55.785+0000 7f5520ce9700 1 -- 172.21.15.155:0/997666434 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f551c108210 con 0x7f551c0f6510 2024-02-10T09:08:55.786 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:55.786+0000 7f5512ffd700 1 -- 172.21.15.155:0/997666434 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f550400e2a0 con 0x7f551c0f6510 2024-02-10T09:08:55.786 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:08:55.786 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:08:55.787 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:55.787+0000 7f5520ce9700 1 -- 172.21.15.155:0/997666434 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f550802ff60 msgr2=0x7f5508032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:08:55.787 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:55.787+0000 7f5520ce9700 1 --2- 172.21.15.155:0/997666434 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f550802ff60 0x7f5508032420 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f550c0096a0 tx=0x7f550c006b40).stop 2024-02-10T09:08:55.788 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:55.787+0000 7f5520ce9700 1 -- 172.21.15.155:0/997666434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f551c0f6510 msgr2=0x7f551c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:08:55.788 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:55.787+0000 7f5520ce9700 1 --2- 172.21.15.155:0/997666434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f551c0f6510 0x7f551c1067e0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f55040089f0 tx=0x7f550400b040).stop 2024-02-10T09:08:55.788 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:55.787+0000 7f5520ce9700 1 -- 172.21.15.155:0/997666434 shutdown_connections 2024-02-10T09:08:55.788 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:55.787+0000 7f5520ce9700 1 --2- 172.21.15.155:0/997666434 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f550802ff60 0x7f5508032420 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:08:55.788 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:55.787+0000 7f5520ce9700 1 --2- 172.21.15.155:0/997666434 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f551c0f6510 0x7f551c1067e0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:08:55.788 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:55.787+0000 7f5520ce9700 1 -- 172.21.15.155:0/997666434 >> 172.21.15.155:0/997666434 conn(0x7f551c0f1c70 msgr2=0x7f551c0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:08:55.788 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:55.787+0000 7f5520ce9700 1 -- 172.21.15.155:0/997666434 shutdown_connections 2024-02-10T09:08:55.788 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:55.787+0000 7f5520ce9700 1 -- 172.21.15.155:0/997666434 wait complete. 2024-02-10T09:08:55.789 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:08:57.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:56 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:08:55.728938+0000 mon.smithi077 (mon.0) 292 : audit [INF] from='mgr.14162 172.21.15.77:0/2029106299' entity='mgr.smithi077.dxntvw' 2024-02-10T09:08:57.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:56 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:08:55.729758+0000 mgr.smithi077.dxntvw (mgr.14162) 314 : cluster [DBG] pgmap v299: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:08:57.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:56 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:08:55.731889+0000 mon.smithi077 (mon.0) 293 : audit [INF] from='mgr.14162 172.21.15.77:0/2029106299' entity='mgr.smithi077.dxntvw' 2024-02-10T09:08:57.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:56 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cephadm 2024-02-10T09:08:55.734007+0000 mgr.smithi077.dxntvw (mgr.14162) 315 : cephadm [INF] Deploying daemon alertmanager.smithi077 on smithi077 2024-02-10T09:08:57.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:56 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:08:55.786186+0000 mon.smithi077 (mon.0) 294 : audit [DBG] from='client.? 172.21.15.155:0/997666434' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:08:57.553 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:08:57.553 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:08:58.148 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:57 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:08:56.727332+0000 mon.smithi077 (mon.0) 295 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-02-10T09:08:59.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:08:58 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:08:57.730097+0000 mgr.smithi077.dxntvw (mgr.14162) 316 : cluster [DBG] pgmap v300: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:08:59.855 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:59.852+0000 7f8812b2d700 1 -- 172.21.15.155:0/812705833 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f880c0f6510 msgr2=0x7f880c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:08:59.856 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:59.852+0000 7f8812b2d700 1 --2- 172.21.15.155:0/812705833 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f880c0f6510 0x7f880c0f6930 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f8800009fb0 tx=0x7f8800009b30).stop 2024-02-10T09:08:59.856 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:59.852+0000 7f8812b2d700 1 -- 172.21.15.155:0/812705833 shutdown_connections 2024-02-10T09:08:59.856 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:59.852+0000 7f8812b2d700 1 --2- 172.21.15.155:0/812705833 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f880c0f6510 0x7f880c0f6930 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:08:59.856 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:59.852+0000 7f8812b2d700 1 -- 172.21.15.155:0/812705833 >> 172.21.15.155:0/812705833 conn(0x7f880c0f1c70 msgr2=0x7f880c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:08:59.856 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:59.852+0000 7f8812b2d700 1 -- 172.21.15.155:0/812705833 shutdown_connections 2024-02-10T09:08:59.856 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:59.852+0000 7f8812b2d700 1 -- 172.21.15.155:0/812705833 wait complete. 2024-02-10T09:08:59.856 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:59.853+0000 7f8812b2d700 1 Processor -- start 2024-02-10T09:08:59.856 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:59.853+0000 7f8812b2d700 1 -- start start 2024-02-10T09:08:59.856 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:59.853+0000 7f8812b2d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f880c0f6510 0x7f880c1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:08:59.856 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:59.853+0000 7f8812b2d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f880c106cd0 con 0x7f880c0f6510 2024-02-10T09:08:59.856 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:59.853+0000 7f88108c9700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f880c0f6510 0x7f880c1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:08:59.856 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:59.853+0000 7f88108c9700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f880c0f6510 0x7f880c1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:51036/0 (socket says 172.21.15.155:51036) 2024-02-10T09:08:59.857 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:59.853+0000 7f88108c9700 1 -- 172.21.15.155:0/1744464461 learned_addr learned my addr 172.21.15.155:0/1744464461 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:08:59.857 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:59.854+0000 7f88108c9700 1 -- 172.21.15.155:0/1744464461 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8800005040 con 0x7f880c0f6510 2024-02-10T09:08:59.857 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:59.854+0000 7f88108c9700 1 --2- 172.21.15.155:0/1744464461 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f880c0f6510 0x7f880c1067e0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f8800005010 tx=0x7f880000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:08:59.857 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:59.854+0000 7f88097fa700 1 -- 172.21.15.155:0/1744464461 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8800009320 con 0x7f880c0f6510 2024-02-10T09:08:59.857 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:59.854+0000 7f8812b2d700 1 -- 172.21.15.155:0/1744464461 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f880c106ed0 con 0x7f880c0f6510 2024-02-10T09:08:59.857 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:59.854+0000 7f88097fa700 1 -- 172.21.15.155:0/1744464461 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8800013380 con 0x7f880c0f6510 2024-02-10T09:08:59.857 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:59.854+0000 7f88097fa700 1 -- 172.21.15.155:0/1744464461 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8800014e50 con 0x7f880c0f6510 2024-02-10T09:08:59.857 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:59.854+0000 7f8812b2d700 1 -- 172.21.15.155:0/1744464461 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f880c107360 con 0x7f880c0f6510 2024-02-10T09:08:59.857 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:59.855+0000 7f88097fa700 1 -- 172.21.15.155:0/1744464461 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8800014420 con 0x7f880c0f6510 2024-02-10T09:08:59.857 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:59.855+0000 7f8812b2d700 1 -- 172.21.15.155:0/1744464461 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f880c045ad0 con 0x7f880c0f6510 2024-02-10T09:08:59.858 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:59.855+0000 7f88097fa700 1 --2- 172.21.15.155:0/1744464461 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f87fc02ff10 0x7f87fc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:08:59.858 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:59.855+0000 7f88097fa700 1 -- 172.21.15.155:0/1744464461 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8800040e30 con 0x7f880c0f6510 2024-02-10T09:08:59.858 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:59.855+0000 7f880bfff700 1 --2- 172.21.15.155:0/1744464461 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f87fc02ff10 0x7f87fc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:08:59.858 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:59.856+0000 7f880bfff700 1 --2- 172.21.15.155:0/1744464461 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f87fc02ff10 0x7f87fc0323d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f87f80096a0 tx=0x7f87f8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:08:59.860 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:08:59.860+0000 7f88097fa700 1 -- 172.21.15.155:0/1744464461 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f88000095f0 con 0x7f880c0f6510 2024-02-10T09:09:00.066 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:00.066+0000 7f8812b2d700 1 -- 172.21.15.155:0/1744464461 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f880c108210 con 0x7f880c0f6510 2024-02-10T09:09:00.067 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:00.067+0000 7f88097fa700 1 -- 172.21.15.155:0/1744464461 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f880001db70 con 0x7f880c0f6510 2024-02-10T09:09:00.067 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:09:00.067 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:09:00.068 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:00.068+0000 7f8812b2d700 1 -- 172.21.15.155:0/1744464461 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f87fc02ff10 msgr2=0x7f87fc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:09:00.069 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:00.068+0000 7f8812b2d700 1 --2- 172.21.15.155:0/1744464461 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f87fc02ff10 0x7f87fc0323d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f87f80096a0 tx=0x7f87f8006b40).stop 2024-02-10T09:09:00.069 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:00.068+0000 7f8812b2d700 1 -- 172.21.15.155:0/1744464461 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f880c0f6510 msgr2=0x7f880c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:09:00.069 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:00.068+0000 7f8812b2d700 1 --2- 172.21.15.155:0/1744464461 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f880c0f6510 0x7f880c1067e0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f8800005010 tx=0x7f880000b040).stop 2024-02-10T09:09:00.069 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:00.068+0000 7f8812b2d700 1 -- 172.21.15.155:0/1744464461 shutdown_connections 2024-02-10T09:09:00.069 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:00.068+0000 7f8812b2d700 1 --2- 172.21.15.155:0/1744464461 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f87fc02ff10 0x7f87fc0323d0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:09:00.069 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:00.068+0000 7f8812b2d700 1 --2- 172.21.15.155:0/1744464461 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f880c0f6510 0x7f880c1067e0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:09:00.069 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:00.068+0000 7f8812b2d700 1 -- 172.21.15.155:0/1744464461 >> 172.21.15.155:0/1744464461 conn(0x7f880c0f1c70 msgr2=0x7f880c0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:09:00.069 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:00.069+0000 7f8812b2d700 1 -- 172.21.15.155:0/1744464461 shutdown_connections 2024-02-10T09:09:00.069 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:00.069+0000 7f8812b2d700 1 -- 172.21.15.155:0/1744464461 wait complete. 2024-02-10T09:09:00.070 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:09:01.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:09:00 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:08:59.730534+0000 mgr.smithi077.dxntvw (mgr.14162) 317 : cluster [DBG] pgmap v301: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:09:01.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:09:00 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:09:00.067409+0000 mon.smithi077 (mon.0) 296 : audit [DBG] from='client.? 172.21.15.155:0/1744464461' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:09:01.879 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:09:01.880 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:09:03.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:09:02 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:09:01.730874+0000 mgr.smithi077.dxntvw (mgr.14162) 318 : cluster [DBG] pgmap v302: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:09:03.973 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:03.970+0000 7f52ac802700 1 -- 172.21.15.155:0/556896636 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52a80f51a0 msgr2=0x7f52a80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:09:03.973 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:03.970+0000 7f52ac802700 1 --2- 172.21.15.155:0/556896636 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52a80f51a0 0x7f52a80f55c0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f5290009fb0 tx=0x7f5290009b30).stop 2024-02-10T09:09:03.974 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:03.970+0000 7f52ac802700 1 -- 172.21.15.155:0/556896636 shutdown_connections 2024-02-10T09:09:03.974 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:03.970+0000 7f52ac802700 1 --2- 172.21.15.155:0/556896636 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52a80f51a0 0x7f52a80f55c0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:09:03.974 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:03.970+0000 7f52ac802700 1 -- 172.21.15.155:0/556896636 >> 172.21.15.155:0/556896636 conn(0x7f52a80f09a0 msgr2=0x7f52a80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:09:03.974 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:03.970+0000 7f52ac802700 1 -- 172.21.15.155:0/556896636 shutdown_connections 2024-02-10T09:09:03.974 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:03.970+0000 7f52ac802700 1 -- 172.21.15.155:0/556896636 wait complete. 2024-02-10T09:09:03.974 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:03.971+0000 7f52ac802700 1 Processor -- start 2024-02-10T09:09:03.974 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:03.971+0000 7f52ac802700 1 -- start start 2024-02-10T09:09:03.974 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:03.971+0000 7f52ac802700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52a80f51a0 0x7f52a81092d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:09:03.974 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:03.971+0000 7f52ac802700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a8060240 con 0x7f52a80f51a0 2024-02-10T09:09:03.974 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:03.971+0000 7f52a6d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52a80f51a0 0x7f52a81092d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:09:03.974 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:03.971+0000 7f52a6d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52a80f51a0 0x7f52a81092d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:46236/0 (socket says 172.21.15.155:46236) 2024-02-10T09:09:03.974 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:03.971+0000 7f52a6d9d700 1 -- 172.21.15.155:0/3832985347 learned_addr learned my addr 172.21.15.155:0/3832985347 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:09:03.975 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:03.972+0000 7f52a6d9d700 1 -- 172.21.15.155:0/3832985347 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5290005040 con 0x7f52a80f51a0 2024-02-10T09:09:03.975 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:03.972+0000 7f52a6d9d700 1 --2- 172.21.15.155:0/3832985347 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52a80f51a0 0x7f52a81092d0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f52900089f0 tx=0x7f529000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:09:03.975 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:03.972+0000 7f529f7fe700 1 -- 172.21.15.155:0/3832985347 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5290009320 con 0x7f52a80f51a0 2024-02-10T09:09:03.975 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:03.972+0000 7f52ac802700 1 -- 172.21.15.155:0/3832985347 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52a8060440 con 0x7f52a80f51a0 2024-02-10T09:09:03.975 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:03.972+0000 7f529f7fe700 1 -- 172.21.15.155:0/3832985347 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5290013380 con 0x7f52a80f51a0 2024-02-10T09:09:03.975 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:03.972+0000 7f529f7fe700 1 -- 172.21.15.155:0/3832985347 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5290014e20 con 0x7f52a80f51a0 2024-02-10T09:09:03.975 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:03.972+0000 7f52ac802700 1 -- 172.21.15.155:0/3832985347 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52a8109980 con 0x7f52a80f51a0 2024-02-10T09:09:03.976 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:03.973+0000 7f529f7fe700 1 -- 172.21.15.155:0/3832985347 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5290014420 con 0x7f52a80f51a0 2024-02-10T09:09:03.976 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:03.973+0000 7f52ac802700 1 -- 172.21.15.155:0/3832985347 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52a8045ad0 con 0x7f52a80f51a0 2024-02-10T09:09:03.976 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:03.973+0000 7f529f7fe700 1 --2- 172.21.15.155:0/3832985347 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f529402ff60 0x7f5294032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:09:03.976 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:03.973+0000 7f529f7fe700 1 -- 172.21.15.155:0/3832985347 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5290040e20 con 0x7f52a80f51a0 2024-02-10T09:09:03.976 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:03.973+0000 7f52a659c700 1 --2- 172.21.15.155:0/3832985347 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f529402ff60 0x7f5294032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:09:03.976 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:03.974+0000 7f52a659c700 1 --2- 172.21.15.155:0/3832985347 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f529402ff60 0x7f5294032420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f52980096a0 tx=0x7f5298006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:09:03.979 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:03.978+0000 7f529f7fe700 1 -- 172.21.15.155:0/3832985347 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5290014790 con 0x7f52a80f51a0 2024-02-10T09:09:04.185 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:04.184+0000 7f52ac802700 1 -- 172.21.15.155:0/3832985347 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f52a810a1a0 con 0x7f52a80f51a0 2024-02-10T09:09:04.186 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:04.185+0000 7f529f7fe700 1 -- 172.21.15.155:0/3832985347 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5290010230 con 0x7f52a80f51a0 2024-02-10T09:09:04.186 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:09:04.186 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:09:04.188 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:04.187+0000 7f52ac802700 1 -- 172.21.15.155:0/3832985347 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f529402ff60 msgr2=0x7f5294032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:09:04.188 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:04.187+0000 7f52ac802700 1 --2- 172.21.15.155:0/3832985347 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f529402ff60 0x7f5294032420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f52980096a0 tx=0x7f5298006b40).stop 2024-02-10T09:09:04.188 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:04.187+0000 7f52ac802700 1 -- 172.21.15.155:0/3832985347 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52a80f51a0 msgr2=0x7f52a81092d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:09:04.188 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:04.187+0000 7f52ac802700 1 --2- 172.21.15.155:0/3832985347 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52a80f51a0 0x7f52a81092d0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f52900089f0 tx=0x7f529000b040).stop 2024-02-10T09:09:04.188 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:04.187+0000 7f52ac802700 1 -- 172.21.15.155:0/3832985347 shutdown_connections 2024-02-10T09:09:04.188 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:04.187+0000 7f52ac802700 1 --2- 172.21.15.155:0/3832985347 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f529402ff60 0x7f5294032420 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:09:04.188 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:04.187+0000 7f52ac802700 1 --2- 172.21.15.155:0/3832985347 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52a80f51a0 0x7f52a81092d0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:09:04.188 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:04.187+0000 7f52ac802700 1 -- 172.21.15.155:0/3832985347 >> 172.21.15.155:0/3832985347 conn(0x7f52a80f09a0 msgr2=0x7f52a80f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:09:04.188 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:04.187+0000 7f52ac802700 1 -- 172.21.15.155:0/3832985347 shutdown_connections 2024-02-10T09:09:04.188 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:04.187+0000 7f52ac802700 1 -- 172.21.15.155:0/3832985347 wait complete. 2024-02-10T09:09:04.189 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:09:05.148 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:09:04 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:09:03.731108+0000 mgr.smithi077.dxntvw (mgr.14162) 319 : cluster [DBG] pgmap v303: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:09:05.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:09:04 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:09:04.186433+0000 mon.smithi077 (mon.0) 297 : audit [DBG] from='client.? 172.21.15.155:0/3832985347' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:09:05.952 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:09:05.953 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:09:07.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:09:06 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:09:05.731394+0000 mgr.smithi077.dxntvw (mgr.14162) 320 : cluster [DBG] pgmap v304: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:09:08.160 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:08.158+0000 7f6626c3b700 1 -- 172.21.15.155:0/4256653069 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f66200f6510 msgr2=0x7f66200f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:09:08.160 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:08.158+0000 7f6626c3b700 1 --2- 172.21.15.155:0/4256653069 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f66200f6510 0x7f66200f6930 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7f6614009fb0 tx=0x7f6614009b30).stop 2024-02-10T09:09:08.160 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:08.158+0000 7f6626c3b700 1 -- 172.21.15.155:0/4256653069 shutdown_connections 2024-02-10T09:09:08.160 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:08.158+0000 7f6626c3b700 1 --2- 172.21.15.155:0/4256653069 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f66200f6510 0x7f66200f6930 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:09:08.160 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:08.158+0000 7f6626c3b700 1 -- 172.21.15.155:0/4256653069 >> 172.21.15.155:0/4256653069 conn(0x7f66200f1c70 msgr2=0x7f66200f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:09:08.160 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:08.158+0000 7f6626c3b700 1 -- 172.21.15.155:0/4256653069 shutdown_connections 2024-02-10T09:09:08.160 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:08.158+0000 7f6626c3b700 1 -- 172.21.15.155:0/4256653069 wait complete. 2024-02-10T09:09:08.160 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:08.158+0000 7f6626c3b700 1 Processor -- start 2024-02-10T09:09:08.160 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:08.159+0000 7f6626c3b700 1 -- start start 2024-02-10T09:09:08.163 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:08.159+0000 7f6626c3b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f66200f6510 0x7f6620108c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:09:08.163 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:08.159+0000 7f6626c3b700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6620109110 con 0x7f66200f6510 2024-02-10T09:09:08.163 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:08.159+0000 7f66249d7700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f66200f6510 0x7f6620108c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:09:08.164 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:08.159+0000 7f66249d7700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f66200f6510 0x7f6620108c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:46242/0 (socket says 172.21.15.155:46242) 2024-02-10T09:09:08.164 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:08.159+0000 7f66249d7700 1 -- 172.21.15.155:0/3143467394 learned_addr learned my addr 172.21.15.155:0/3143467394 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:09:08.164 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:08.160+0000 7f66249d7700 1 -- 172.21.15.155:0/3143467394 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6614005040 con 0x7f66200f6510 2024-02-10T09:09:08.164 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:08.160+0000 7f66249d7700 1 --2- 172.21.15.155:0/3143467394 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f66200f6510 0x7f6620108c20 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f6614005010 tx=0x7f661400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:09:08.164 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:08.160+0000 7f661d7fa700 1 -- 172.21.15.155:0/3143467394 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6614009320 con 0x7f66200f6510 2024-02-10T09:09:08.164 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:08.160+0000 7f661d7fa700 1 -- 172.21.15.155:0/3143467394 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6614013380 con 0x7f66200f6510 2024-02-10T09:09:08.164 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:08.160+0000 7f6626c3b700 1 -- 172.21.15.155:0/3143467394 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6620109310 con 0x7f66200f6510 2024-02-10T09:09:08.164 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:08.160+0000 7f661d7fa700 1 -- 172.21.15.155:0/3143467394 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6614014e50 con 0x7f66200f6510 2024-02-10T09:09:08.164 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:08.160+0000 7f6626c3b700 1 -- 172.21.15.155:0/3143467394 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66201047e0 con 0x7f66200f6510 2024-02-10T09:09:08.164 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:08.161+0000 7f661d7fa700 1 -- 172.21.15.155:0/3143467394 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f66140263a0 con 0x7f66200f6510 2024-02-10T09:09:08.164 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:08.161+0000 7f6626c3b700 1 -- 172.21.15.155:0/3143467394 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6620045ad0 con 0x7f66200f6510 2024-02-10T09:09:08.164 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:08.161+0000 7f661d7fa700 1 --2- 172.21.15.155:0/3143467394 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f661002ff10 0x7f66100323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:09:08.164 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:08.161+0000 7f661d7fa700 1 -- 172.21.15.155:0/3143467394 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6614040fc0 con 0x7f66200f6510 2024-02-10T09:09:08.165 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:08.161+0000 7f661ffff700 1 --2- 172.21.15.155:0/3143467394 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f661002ff10 0x7f66100323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:09:08.165 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:08.162+0000 7f661ffff700 1 --2- 172.21.15.155:0/3143467394 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f661002ff10 0x7f66100323d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f660c0096a0 tx=0x7f660c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:09:08.167 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:08.166+0000 7f661d7fa700 1 -- 172.21.15.155:0/3143467394 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f66140126f0 con 0x7f66200f6510 2024-02-10T09:09:08.370 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:08.369+0000 7f6626c3b700 1 -- 172.21.15.155:0/3143467394 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f66200fc0a0 con 0x7f66200f6510 2024-02-10T09:09:08.370 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:08.369+0000 7f661d7fa700 1 -- 172.21.15.155:0/3143467394 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6614012330 con 0x7f66200f6510 2024-02-10T09:09:08.371 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:09:08.371 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:09:08.372 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:08.371+0000 7f6626c3b700 1 -- 172.21.15.155:0/3143467394 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f661002ff10 msgr2=0x7f66100323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:09:08.372 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:08.371+0000 7f6626c3b700 1 --2- 172.21.15.155:0/3143467394 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f661002ff10 0x7f66100323d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f660c0096a0 tx=0x7f660c006b40).stop 2024-02-10T09:09:08.372 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:08.371+0000 7f6626c3b700 1 -- 172.21.15.155:0/3143467394 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f66200f6510 msgr2=0x7f6620108c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:09:08.372 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:08.371+0000 7f6626c3b700 1 --2- 172.21.15.155:0/3143467394 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f66200f6510 0x7f6620108c20 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f6614005010 tx=0x7f661400b040).stop 2024-02-10T09:09:08.372 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:08.371+0000 7f6626c3b700 1 -- 172.21.15.155:0/3143467394 shutdown_connections 2024-02-10T09:09:08.372 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:08.371+0000 7f6626c3b700 1 --2- 172.21.15.155:0/3143467394 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f661002ff10 0x7f66100323d0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:09:08.372 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:08.371+0000 7f6626c3b700 1 --2- 172.21.15.155:0/3143467394 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f66200f6510 0x7f6620108c20 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:09:08.373 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:08.371+0000 7f6626c3b700 1 -- 172.21.15.155:0/3143467394 >> 172.21.15.155:0/3143467394 conn(0x7f66200f1c70 msgr2=0x7f66200f2650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:09:08.373 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:08.371+0000 7f6626c3b700 1 -- 172.21.15.155:0/3143467394 shutdown_connections 2024-02-10T09:09:08.373 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:08.371+0000 7f6626c3b700 1 -- 172.21.15.155:0/3143467394 wait complete. 2024-02-10T09:09:08.373 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:09:09.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:09:08 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:09:07.731719+0000 mgr.smithi077.dxntvw (mgr.14162) 321 : cluster [DBG] pgmap v305: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:09:09.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:09:08 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:09:08.371069+0000 mon.smithi077 (mon.0) 298 : audit [DBG] from='client.? 172.21.15.155:0/3143467394' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:09:10.077 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:09:10.078 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:09:11.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:09:10 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:09:09.732077+0000 mgr.smithi077.dxntvw (mgr.14162) 322 : cluster [DBG] pgmap v306: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:09:12.311 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:12.308+0000 7f10c33c8700 1 -- 172.21.15.155:0/3492581367 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10bc0f5410 msgr2=0x7f10bc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:09:12.311 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:12.308+0000 7f10c33c8700 1 --2- 172.21.15.155:0/3492581367 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10bc0f5410 0x7f10bc0f5830 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f10ac009fb0 tx=0x7f10ac009b30).stop 2024-02-10T09:09:12.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:12.308+0000 7f10c33c8700 1 -- 172.21.15.155:0/3492581367 shutdown_connections 2024-02-10T09:09:12.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:12.308+0000 7f10c33c8700 1 --2- 172.21.15.155:0/3492581367 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10bc0f5410 0x7f10bc0f5830 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:09:12.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:12.308+0000 7f10c33c8700 1 -- 172.21.15.155:0/3492581367 >> 172.21.15.155:0/3492581367 conn(0x7f10bc0f0b70 msgr2=0x7f10bc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:09:12.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:12.308+0000 7f10c33c8700 1 -- 172.21.15.155:0/3492581367 shutdown_connections 2024-02-10T09:09:12.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:12.308+0000 7f10c33c8700 1 -- 172.21.15.155:0/3492581367 wait complete. 2024-02-10T09:09:12.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:12.309+0000 7f10c33c8700 1 Processor -- start 2024-02-10T09:09:12.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:12.309+0000 7f10c33c8700 1 -- start start 2024-02-10T09:09:12.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:12.309+0000 7f10c33c8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10bc0f5410 0x7f10bc109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:09:12.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:12.309+0000 7f10c33c8700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10bc060220 con 0x7f10bc0f5410 2024-02-10T09:09:12.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:12.309+0000 7f10c1164700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10bc0f5410 0x7f10bc109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:09:12.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:12.309+0000 7f10c1164700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10bc0f5410 0x7f10bc109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:36946/0 (socket says 172.21.15.155:36946) 2024-02-10T09:09:12.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:12.309+0000 7f10c1164700 1 -- 172.21.15.155:0/1702409170 learned_addr learned my addr 172.21.15.155:0/1702409170 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:09:12.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:12.309+0000 7f10c1164700 1 -- 172.21.15.155:0/1702409170 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10ac005040 con 0x7f10bc0f5410 2024-02-10T09:09:12.313 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:12.310+0000 7f10c1164700 1 --2- 172.21.15.155:0/1702409170 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10bc0f5410 0x7f10bc109500 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f10ac005010 tx=0x7f10ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:09:12.313 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:12.310+0000 7f10b1ffb700 1 -- 172.21.15.155:0/1702409170 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f10ac009320 con 0x7f10bc0f5410 2024-02-10T09:09:12.313 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:12.310+0000 7f10c33c8700 1 -- 172.21.15.155:0/1702409170 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10bc060420 con 0x7f10bc0f5410 2024-02-10T09:09:12.313 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:12.310+0000 7f10b1ffb700 1 -- 172.21.15.155:0/1702409170 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f10ac013380 con 0x7f10bc0f5410 2024-02-10T09:09:12.313 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:12.310+0000 7f10b1ffb700 1 -- 172.21.15.155:0/1702409170 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f10ac014e50 con 0x7f10bc0f5410 2024-02-10T09:09:12.313 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:12.310+0000 7f10c33c8700 1 -- 172.21.15.155:0/1702409170 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10bc109bb0 con 0x7f10bc0f5410 2024-02-10T09:09:12.314 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:12.311+0000 7f10b1ffb700 1 -- 172.21.15.155:0/1702409170 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f10ac0095f0 con 0x7f10bc0f5410 2024-02-10T09:09:12.314 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:12.311+0000 7f10c33c8700 1 -- 172.21.15.155:0/1702409170 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10bc059100 con 0x7f10bc0f5410 2024-02-10T09:09:12.314 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:12.311+0000 7f10b1ffb700 1 --2- 172.21.15.155:0/1702409170 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f10a802ff10 0x7f10a80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:09:12.314 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:12.311+0000 7f10b1ffb700 1 -- 172.21.15.155:0/1702409170 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f10ac040e70 con 0x7f10bc0f5410 2024-02-10T09:09:12.314 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:12.311+0000 7f10c0963700 1 --2- 172.21.15.155:0/1702409170 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f10a802ff10 0x7f10a80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:09:12.314 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:12.312+0000 7f10c0963700 1 --2- 172.21.15.155:0/1702409170 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f10a802ff10 0x7f10a80323d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f10b80096a0 tx=0x7f10b8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:09:12.317 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:12.316+0000 7f10b1ffb700 1 -- 172.21.15.155:0/1702409170 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f10ac014420 con 0x7f10bc0f5410 2024-02-10T09:09:12.523 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:12.521+0000 7f10c33c8700 1 -- 172.21.15.155:0/1702409170 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f10bc045ad0 con 0x7f10bc0f5410 2024-02-10T09:09:12.523 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:12.522+0000 7f10b1ffb700 1 -- 172.21.15.155:0/1702409170 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f10ac010230 con 0x7f10bc0f5410 2024-02-10T09:09:12.524 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:09:12.524 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:09:12.525 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:12.523+0000 7f10c33c8700 1 -- 172.21.15.155:0/1702409170 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f10a802ff10 msgr2=0x7f10a80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:09:12.526 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:12.523+0000 7f10c33c8700 1 --2- 172.21.15.155:0/1702409170 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f10a802ff10 0x7f10a80323d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f10b80096a0 tx=0x7f10b8006b40).stop 2024-02-10T09:09:12.526 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:12.523+0000 7f10c33c8700 1 -- 172.21.15.155:0/1702409170 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10bc0f5410 msgr2=0x7f10bc109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:09:12.526 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:12.523+0000 7f10c33c8700 1 --2- 172.21.15.155:0/1702409170 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10bc0f5410 0x7f10bc109500 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f10ac005010 tx=0x7f10ac00b040).stop 2024-02-10T09:09:12.526 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:12.523+0000 7f10c33c8700 1 -- 172.21.15.155:0/1702409170 shutdown_connections 2024-02-10T09:09:12.526 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:12.523+0000 7f10c33c8700 1 --2- 172.21.15.155:0/1702409170 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f10a802ff10 0x7f10a80323d0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:09:12.526 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:12.523+0000 7f10c33c8700 1 --2- 172.21.15.155:0/1702409170 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f10bc0f5410 0x7f10bc109500 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:09:12.526 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:12.523+0000 7f10c33c8700 1 -- 172.21.15.155:0/1702409170 >> 172.21.15.155:0/1702409170 conn(0x7f10bc0f0b70 msgr2=0x7f10bc0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:09:12.527 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:12.524+0000 7f10c33c8700 1 -- 172.21.15.155:0/1702409170 shutdown_connections 2024-02-10T09:09:12.527 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:12.524+0000 7f10c33c8700 1 -- 172.21.15.155:0/1702409170 wait complete. 2024-02-10T09:09:12.527 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:09:13.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:09:12 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:09:11.732336+0000 mgr.smithi077.dxntvw (mgr.14162) 323 : cluster [DBG] pgmap v307: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:09:13.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:09:12 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:09:12.523593+0000 mon.smithi077 (mon.0) 299 : audit [DBG] from='client.? 172.21.15.155:0/1702409170' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:09:14.310 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:09:14.310 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:09:15.148 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:09:14 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:09:13.732693+0000 mgr.smithi077.dxntvw (mgr.14162) 324 : cluster [DBG] pgmap v308: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:09:16.467 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:16.465+0000 7f9205b46700 1 -- 172.21.15.155:0/1036958930 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92000f5410 msgr2=0x7f92000f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:09:16.467 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:16.465+0000 7f9205b46700 1 --2- 172.21.15.155:0/1036958930 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92000f5410 0x7f92000f5830 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7f91e8009fb0 tx=0x7f91e8009b30).stop 2024-02-10T09:09:16.467 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:16.465+0000 7f9205b46700 1 -- 172.21.15.155:0/1036958930 shutdown_connections 2024-02-10T09:09:16.468 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:16.465+0000 7f9205b46700 1 --2- 172.21.15.155:0/1036958930 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92000f5410 0x7f92000f5830 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:09:16.468 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:16.465+0000 7f9205b46700 1 -- 172.21.15.155:0/1036958930 >> 172.21.15.155:0/1036958930 conn(0x7f92000f0b70 msgr2=0x7f92000f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:09:16.468 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:16.465+0000 7f9205b46700 1 -- 172.21.15.155:0/1036958930 shutdown_connections 2024-02-10T09:09:16.468 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:16.465+0000 7f9205b46700 1 -- 172.21.15.155:0/1036958930 wait complete. 2024-02-10T09:09:16.468 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:16.466+0000 7f9205b46700 1 Processor -- start 2024-02-10T09:09:16.468 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:16.466+0000 7f9205b46700 1 -- start start 2024-02-10T09:09:16.468 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:16.466+0000 7f9205b46700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92000f5410 0x7f92001056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:09:16.468 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:16.466+0000 7f9205b46700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9200105bd0 con 0x7f92000f5410 2024-02-10T09:09:16.470 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:16.467+0000 7f91ff7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92000f5410 0x7f92001056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:09:16.470 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:16.467+0000 7f91ff7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92000f5410 0x7f92001056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:36966/0 (socket says 172.21.15.155:36966) 2024-02-10T09:09:16.470 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:16.467+0000 7f91ff7fe700 1 -- 172.21.15.155:0/3404919841 learned_addr learned my addr 172.21.15.155:0/3404919841 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:09:16.470 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:16.467+0000 7f91ff7fe700 1 -- 172.21.15.155:0/3404919841 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91e8005040 con 0x7f92000f5410 2024-02-10T09:09:16.470 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:16.467+0000 7f91ff7fe700 1 --2- 172.21.15.155:0/3404919841 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92000f5410 0x7f92001056e0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f91e8005010 tx=0x7f91e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:09:16.470 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:16.467+0000 7f91f7fff700 1 -- 172.21.15.155:0/3404919841 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f91e8009320 con 0x7f92000f5410 2024-02-10T09:09:16.470 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:16.467+0000 7f9205b46700 1 -- 172.21.15.155:0/3404919841 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9200105dd0 con 0x7f92000f5410 2024-02-10T09:09:16.471 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:16.467+0000 7f91f7fff700 1 -- 172.21.15.155:0/3404919841 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f91e8013380 con 0x7f92000f5410 2024-02-10T09:09:16.471 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:16.467+0000 7f91f7fff700 1 -- 172.21.15.155:0/3404919841 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f91e8014e50 con 0x7f92000f5410 2024-02-10T09:09:16.471 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:16.467+0000 7f9205b46700 1 -- 172.21.15.155:0/3404919841 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9200106260 con 0x7f92000f5410 2024-02-10T09:09:16.471 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:16.468+0000 7f91f7fff700 1 -- 172.21.15.155:0/3404919841 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f91e8014420 con 0x7f92000f5410 2024-02-10T09:09:16.471 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:16.468+0000 7f9205b46700 1 -- 172.21.15.155:0/3404919841 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9200045ad0 con 0x7f92000f5410 2024-02-10T09:09:16.471 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:16.468+0000 7f91f7fff700 1 --2- 172.21.15.155:0/3404919841 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f91ec02ff10 0x7f91ec0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:09:16.471 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:16.468+0000 7f91f7fff700 1 -- 172.21.15.155:0/3404919841 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f91e8042d10 con 0x7f92000f5410 2024-02-10T09:09:16.471 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:16.468+0000 7f91feffd700 1 --2- 172.21.15.155:0/3404919841 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f91ec02ff10 0x7f91ec0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:09:16.471 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:16.469+0000 7f91feffd700 1 --2- 172.21.15.155:0/3404919841 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f91ec02ff10 0x7f91ec0323d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f91f00096a0 tx=0x7f91f0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:09:16.474 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:16.473+0000 7f91f7fff700 1 -- 172.21.15.155:0/3404919841 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f91e8014790 con 0x7f92000f5410 2024-02-10T09:09:16.677 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:16.676+0000 7f9205b46700 1 -- 172.21.15.155:0/3404919841 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f92001070a0 con 0x7f92000f5410 2024-02-10T09:09:16.677 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:16.676+0000 7f91f7fff700 1 -- 172.21.15.155:0/3404919841 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f91e8011290 con 0x7f92000f5410 2024-02-10T09:09:16.678 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:09:16.678 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:09:16.679 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:16.678+0000 7f9205b46700 1 -- 172.21.15.155:0/3404919841 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f91ec02ff10 msgr2=0x7f91ec0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:09:16.679 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:16.678+0000 7f9205b46700 1 --2- 172.21.15.155:0/3404919841 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f91ec02ff10 0x7f91ec0323d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f91f00096a0 tx=0x7f91f0006b40).stop 2024-02-10T09:09:16.679 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:16.678+0000 7f9205b46700 1 -- 172.21.15.155:0/3404919841 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92000f5410 msgr2=0x7f92001056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:09:16.679 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:16.678+0000 7f9205b46700 1 --2- 172.21.15.155:0/3404919841 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92000f5410 0x7f92001056e0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f91e8005010 tx=0x7f91e800b040).stop 2024-02-10T09:09:16.679 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:16.678+0000 7f9205b46700 1 -- 172.21.15.155:0/3404919841 shutdown_connections 2024-02-10T09:09:16.679 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:16.678+0000 7f9205b46700 1 --2- 172.21.15.155:0/3404919841 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f91ec02ff10 0x7f91ec0323d0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:09:16.679 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:16.678+0000 7f9205b46700 1 --2- 172.21.15.155:0/3404919841 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92000f5410 0x7f92001056e0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:09:16.680 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:16.678+0000 7f9205b46700 1 -- 172.21.15.155:0/3404919841 >> 172.21.15.155:0/3404919841 conn(0x7f92000f0b70 msgr2=0x7f92000f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:09:16.680 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:16.678+0000 7f9205b46700 1 -- 172.21.15.155:0/3404919841 shutdown_connections 2024-02-10T09:09:16.680 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:16.678+0000 7f9205b46700 1 -- 172.21.15.155:0/3404919841 wait complete. 2024-02-10T09:09:16.680 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:09:17.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:09:16 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:09:15.732988+0000 mgr.smithi077.dxntvw (mgr.14162) 325 : cluster [DBG] pgmap v309: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:09:17.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:09:16 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:09:16.677993+0000 mon.smithi077 (mon.0) 300 : audit [DBG] from='client.? 172.21.15.155:0/3404919841' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:09:18.600 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:09:18.600 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:09:19.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:09:18 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:09:17.733382+0000 mgr.smithi077.dxntvw (mgr.14162) 326 : cluster [DBG] pgmap v310: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:09:20.799 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:20.789+0000 7f8c2f615700 1 -- 172.21.15.155:0/3286402805 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c280f6510 msgr2=0x7f8c280f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:09:20.799 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:20.789+0000 7f8c2f615700 1 --2- 172.21.15.155:0/3286402805 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c280f6510 0x7f8c280f6930 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7f8c1c009fb0 tx=0x7f8c1c009b30).stop 2024-02-10T09:09:20.799 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:20.789+0000 7f8c2f615700 1 -- 172.21.15.155:0/3286402805 shutdown_connections 2024-02-10T09:09:20.799 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:20.789+0000 7f8c2f615700 1 --2- 172.21.15.155:0/3286402805 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c280f6510 0x7f8c280f6930 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:09:20.800 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:20.789+0000 7f8c2f615700 1 -- 172.21.15.155:0/3286402805 >> 172.21.15.155:0/3286402805 conn(0x7f8c280f1c70 msgr2=0x7f8c280f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:09:20.800 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:20.789+0000 7f8c2f615700 1 -- 172.21.15.155:0/3286402805 shutdown_connections 2024-02-10T09:09:20.800 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:20.789+0000 7f8c2f615700 1 -- 172.21.15.155:0/3286402805 wait complete. 2024-02-10T09:09:20.800 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:20.790+0000 7f8c2f615700 1 Processor -- start 2024-02-10T09:09:20.800 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:20.790+0000 7f8c2f615700 1 -- start start 2024-02-10T09:09:20.800 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:20.790+0000 7f8c2f615700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c280f6510 0x7f8c281067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:09:20.800 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:20.790+0000 7f8c2f615700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c28106cd0 con 0x7f8c280f6510 2024-02-10T09:09:20.800 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:20.790+0000 7f8c2d3b1700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c280f6510 0x7f8c281067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:09:20.800 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:20.790+0000 7f8c2d3b1700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c280f6510 0x7f8c281067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:36998/0 (socket says 172.21.15.155:36998) 2024-02-10T09:09:20.800 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:20.790+0000 7f8c2d3b1700 1 -- 172.21.15.155:0/2245084000 learned_addr learned my addr 172.21.15.155:0/2245084000 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:09:20.800 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:20.790+0000 7f8c2d3b1700 1 -- 172.21.15.155:0/2245084000 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c1c005040 con 0x7f8c280f6510 2024-02-10T09:09:20.800 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:20.791+0000 7f8c2d3b1700 1 --2- 172.21.15.155:0/2245084000 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c280f6510 0x7f8c281067e0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f8c1c005010 tx=0x7f8c1c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:09:20.800 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:20.791+0000 7f8c19ffb700 1 -- 172.21.15.155:0/2245084000 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c1c009320 con 0x7f8c280f6510 2024-02-10T09:09:20.801 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:20.791+0000 7f8c2f615700 1 -- 172.21.15.155:0/2245084000 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c28106ed0 con 0x7f8c280f6510 2024-02-10T09:09:20.801 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:20.791+0000 7f8c19ffb700 1 -- 172.21.15.155:0/2245084000 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8c1c013380 con 0x7f8c280f6510 2024-02-10T09:09:20.801 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:20.791+0000 7f8c19ffb700 1 -- 172.21.15.155:0/2245084000 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c1c014e50 con 0x7f8c280f6510 2024-02-10T09:09:20.801 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:20.791+0000 7f8c2f615700 1 -- 172.21.15.155:0/2245084000 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c28107360 con 0x7f8c280f6510 2024-02-10T09:09:20.801 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:20.792+0000 7f8c2f615700 1 -- 172.21.15.155:0/2245084000 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c28045ad0 con 0x7f8c280f6510 2024-02-10T09:09:20.801 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:20.792+0000 7f8c19ffb700 1 -- 172.21.15.155:0/2245084000 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8c1c0095f0 con 0x7f8c280f6510 2024-02-10T09:09:20.801 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:20.792+0000 7f8c19ffb700 1 --2- 172.21.15.155:0/2245084000 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f8c1402ff10 0x7f8c140323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:09:20.801 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:20.792+0000 7f8c19ffb700 1 -- 172.21.15.155:0/2245084000 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8c1c040e90 con 0x7f8c280f6510 2024-02-10T09:09:20.801 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:20.792+0000 7f8c2cbb0700 1 --2- 172.21.15.155:0/2245084000 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f8c1402ff10 0x7f8c140323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:09:20.801 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:20.793+0000 7f8c2cbb0700 1 --2- 172.21.15.155:0/2245084000 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f8c1402ff10 0x7f8c140323d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f8c240096a0 tx=0x7f8c24006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:09:20.801 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:20.797+0000 7f8c19ffb700 1 -- 172.21.15.155:0/2245084000 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8c1c014420 con 0x7f8c280f6510 2024-02-10T09:09:21.004 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:21.002+0000 7f8c2f615700 1 -- 172.21.15.155:0/2245084000 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8c281081b0 con 0x7f8c280f6510 2024-02-10T09:09:21.004 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:21.003+0000 7f8c19ffb700 1 -- 172.21.15.155:0/2245084000 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8c1c010230 con 0x7f8c280f6510 2024-02-10T09:09:21.004 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:09:21.005 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:09:21.006 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:21.005+0000 7f8c2f615700 1 -- 172.21.15.155:0/2245084000 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f8c1402ff10 msgr2=0x7f8c140323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:09:21.006 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:21.005+0000 7f8c2f615700 1 --2- 172.21.15.155:0/2245084000 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f8c1402ff10 0x7f8c140323d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f8c240096a0 tx=0x7f8c24006b40).stop 2024-02-10T09:09:21.006 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:21.005+0000 7f8c2f615700 1 -- 172.21.15.155:0/2245084000 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c280f6510 msgr2=0x7f8c281067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:09:21.006 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:21.005+0000 7f8c2f615700 1 --2- 172.21.15.155:0/2245084000 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c280f6510 0x7f8c281067e0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f8c1c005010 tx=0x7f8c1c00b040).stop 2024-02-10T09:09:21.006 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:21.005+0000 7f8c2f615700 1 -- 172.21.15.155:0/2245084000 shutdown_connections 2024-02-10T09:09:21.006 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:21.005+0000 7f8c2f615700 1 --2- 172.21.15.155:0/2245084000 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f8c1402ff10 0x7f8c140323d0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:09:21.006 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:21.005+0000 7f8c2f615700 1 --2- 172.21.15.155:0/2245084000 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c280f6510 0x7f8c281067e0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:09:21.006 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:21.005+0000 7f8c2f615700 1 -- 172.21.15.155:0/2245084000 >> 172.21.15.155:0/2245084000 conn(0x7f8c280f1c70 msgr2=0x7f8c280f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:09:21.006 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:21.005+0000 7f8c2f615700 1 -- 172.21.15.155:0/2245084000 shutdown_connections 2024-02-10T09:09:21.006 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:21.005+0000 7f8c2f615700 1 -- 172.21.15.155:0/2245084000 wait complete. 2024-02-10T09:09:21.007 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:09:21.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:09:20 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:09:19.733721+0000 mgr.smithi077.dxntvw (mgr.14162) 327 : cluster [DBG] pgmap v311: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:09:22.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:09:21 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:09:21.004735+0000 mon.smithi077 (mon.0) 301 : audit [DBG] from='client.? 172.21.15.155:0/2245084000' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:09:22.776 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:09:22.776 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:09:23.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:09:22 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:09:21.734069+0000 mgr.smithi077.dxntvw (mgr.14162) 328 : cluster [DBG] pgmap v312: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:09:25.077 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:25.075+0000 7fe9318aa700 1 -- 172.21.15.155:0/1539893633 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe92c0f5410 msgr2=0x7fe92c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:09:25.078 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:25.075+0000 7fe9318aa700 1 --2- 172.21.15.155:0/1539893633 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe92c0f5410 0x7fe92c0f5830 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7fe914009fb0 tx=0x7fe914009b30).stop 2024-02-10T09:09:25.078 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:25.076+0000 7fe9318aa700 1 -- 172.21.15.155:0/1539893633 shutdown_connections 2024-02-10T09:09:25.078 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:25.076+0000 7fe9318aa700 1 --2- 172.21.15.155:0/1539893633 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe92c0f5410 0x7fe92c0f5830 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:09:25.078 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:25.076+0000 7fe9318aa700 1 -- 172.21.15.155:0/1539893633 >> 172.21.15.155:0/1539893633 conn(0x7fe92c0f0b70 msgr2=0x7fe92c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:09:25.078 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:25.076+0000 7fe9318aa700 1 -- 172.21.15.155:0/1539893633 shutdown_connections 2024-02-10T09:09:25.078 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:25.076+0000 7fe9318aa700 1 -- 172.21.15.155:0/1539893633 wait complete. 2024-02-10T09:09:25.078 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:25.076+0000 7fe9318aa700 1 Processor -- start 2024-02-10T09:09:25.078 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:25.076+0000 7fe9318aa700 1 -- start start 2024-02-10T09:09:25.078 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:25.076+0000 7fe9318aa700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe92c0f5410 0x7fe92c1067f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:09:25.078 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:25.076+0000 7fe9318aa700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe92c106ce0 con 0x7fe92c0f5410 2024-02-10T09:09:25.078 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:25.077+0000 7fe92affd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe92c0f5410 0x7fe92c1067f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:09:25.079 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:25.077+0000 7fe92affd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe92c0f5410 0x7fe92c1067f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:36306/0 (socket says 172.21.15.155:36306) 2024-02-10T09:09:25.079 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:25.077+0000 7fe92affd700 1 -- 172.21.15.155:0/1654615151 learned_addr learned my addr 172.21.15.155:0/1654615151 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:09:25.080 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:25.077+0000 7fe92affd700 1 -- 172.21.15.155:0/1654615151 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe914005040 con 0x7fe92c0f5410 2024-02-10T09:09:25.080 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:25.077+0000 7fe92affd700 1 --2- 172.21.15.155:0/1654615151 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe92c0f5410 0x7fe92c1067f0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7fe914005010 tx=0x7fe91400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:09:25.080 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:25.077+0000 7fe9237fe700 1 -- 172.21.15.155:0/1654615151 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe914009320 con 0x7fe92c0f5410 2024-02-10T09:09:25.081 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:25.077+0000 7fe9318aa700 1 -- 172.21.15.155:0/1654615151 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe92c106ee0 con 0x7fe92c0f5410 2024-02-10T09:09:25.081 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:25.077+0000 7fe9237fe700 1 -- 172.21.15.155:0/1654615151 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe914013380 con 0x7fe92c0f5410 2024-02-10T09:09:25.081 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:25.077+0000 7fe9237fe700 1 -- 172.21.15.155:0/1654615151 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe914014e50 con 0x7fe92c0f5410 2024-02-10T09:09:25.081 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:25.077+0000 7fe9318aa700 1 -- 172.21.15.155:0/1654615151 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe92c107370 con 0x7fe92c0f5410 2024-02-10T09:09:25.081 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:25.078+0000 7fe9237fe700 1 -- 172.21.15.155:0/1654615151 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe9140095f0 con 0x7fe92c0f5410 2024-02-10T09:09:25.081 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:25.078+0000 7fe9318aa700 1 -- 172.21.15.155:0/1654615151 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe92c045ad0 con 0x7fe92c0f5410 2024-02-10T09:09:25.081 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:25.078+0000 7fe9237fe700 1 --2- 172.21.15.155:0/1654615151 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fe91802ff10 0x7fe9180323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:09:25.081 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:25.078+0000 7fe9237fe700 1 -- 172.21.15.155:0/1654615151 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe914040e90 con 0x7fe92c0f5410 2024-02-10T09:09:25.081 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:25.079+0000 7fe92a7fc700 1 --2- 172.21.15.155:0/1654615151 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fe91802ff10 0x7fe9180323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:09:25.081 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:25.079+0000 7fe92a7fc700 1 --2- 172.21.15.155:0/1654615151 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fe91802ff10 0x7fe9180323d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7fe91c0096a0 tx=0x7fe91c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:09:25.083 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:25.083+0000 7fe9237fe700 1 -- 172.21.15.155:0/1654615151 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe914014420 con 0x7fe92c0f5410 2024-02-10T09:09:25.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:09:24 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:09:23.734487+0000 mgr.smithi077.dxntvw (mgr.14162) 329 : cluster [DBG] pgmap v313: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:09:25.288 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:25.287+0000 7fe9318aa700 1 -- 172.21.15.155:0/1654615151 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe92c108220 con 0x7fe92c0f5410 2024-02-10T09:09:25.289 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:25.288+0000 7fe9237fe700 1 -- 172.21.15.155:0/1654615151 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe914010230 con 0x7fe92c0f5410 2024-02-10T09:09:25.289 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:09:25.289 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:09:25.291 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:25.289+0000 7fe9318aa700 1 -- 172.21.15.155:0/1654615151 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fe91802ff10 msgr2=0x7fe9180323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:09:25.291 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:25.289+0000 7fe9318aa700 1 --2- 172.21.15.155:0/1654615151 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fe91802ff10 0x7fe9180323d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7fe91c0096a0 tx=0x7fe91c006b40).stop 2024-02-10T09:09:25.291 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:25.289+0000 7fe9318aa700 1 -- 172.21.15.155:0/1654615151 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe92c0f5410 msgr2=0x7fe92c1067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:09:25.291 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:25.289+0000 7fe9318aa700 1 --2- 172.21.15.155:0/1654615151 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe92c0f5410 0x7fe92c1067f0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7fe914005010 tx=0x7fe91400b040).stop 2024-02-10T09:09:25.291 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:25.289+0000 7fe9318aa700 1 -- 172.21.15.155:0/1654615151 shutdown_connections 2024-02-10T09:09:25.291 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:25.289+0000 7fe9318aa700 1 --2- 172.21.15.155:0/1654615151 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fe91802ff10 0x7fe9180323d0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:09:25.291 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:25.289+0000 7fe9318aa700 1 --2- 172.21.15.155:0/1654615151 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe92c0f5410 0x7fe92c1067f0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:09:25.291 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:25.289+0000 7fe9318aa700 1 -- 172.21.15.155:0/1654615151 >> 172.21.15.155:0/1654615151 conn(0x7fe92c0f0b70 msgr2=0x7fe92c05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:09:25.291 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:25.289+0000 7fe9318aa700 1 -- 172.21.15.155:0/1654615151 shutdown_connections 2024-02-10T09:09:25.291 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:25.289+0000 7fe9318aa700 1 -- 172.21.15.155:0/1654615151 wait complete. 2024-02-10T09:09:25.292 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:09:26.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:09:25 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:09:25.289470+0000 mon.smithi077 (mon.0) 302 : audit [DBG] from='client.? 172.21.15.155:0/1654615151' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:09:27.110 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:09:27.111 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:09:27.148 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:09:26 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:09:25.734864+0000 mgr.smithi077.dxntvw (mgr.14162) 330 : cluster [DBG] pgmap v314: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:09:29.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:09:28 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:09:27.735239+0000 mgr.smithi077.dxntvw (mgr.14162) 331 : cluster [DBG] pgmap v315: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:09:29.188 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:29.185+0000 7fe224c6e700 1 -- 172.21.15.155:0/1841334856 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2200f5410 msgr2=0x7fe2200f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:09:29.188 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:29.185+0000 7fe224c6e700 1 --2- 172.21.15.155:0/1841334856 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2200f5410 0x7fe2200f5830 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7fe208009fb0 tx=0x7fe208009b30).stop 2024-02-10T09:09:29.188 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:29.185+0000 7fe224c6e700 1 -- 172.21.15.155:0/1841334856 shutdown_connections 2024-02-10T09:09:29.189 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:29.185+0000 7fe224c6e700 1 --2- 172.21.15.155:0/1841334856 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2200f5410 0x7fe2200f5830 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:09:29.189 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:29.185+0000 7fe224c6e700 1 -- 172.21.15.155:0/1841334856 >> 172.21.15.155:0/1841334856 conn(0x7fe2200f0b70 msgr2=0x7fe2200f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:09:29.189 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:29.185+0000 7fe224c6e700 1 -- 172.21.15.155:0/1841334856 shutdown_connections 2024-02-10T09:09:29.189 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:29.185+0000 7fe224c6e700 1 -- 172.21.15.155:0/1841334856 wait complete. 2024-02-10T09:09:29.189 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:29.186+0000 7fe224c6e700 1 Processor -- start 2024-02-10T09:09:29.189 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:29.186+0000 7fe224c6e700 1 -- start start 2024-02-10T09:09:29.189 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:29.186+0000 7fe224c6e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2200f5410 0x7fe2201067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:09:29.189 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:29.186+0000 7fe224c6e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe220106cc0 con 0x7fe2200f5410 2024-02-10T09:09:29.189 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:29.186+0000 7fe21e59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2200f5410 0x7fe2201067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:09:29.189 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:29.186+0000 7fe21e59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2200f5410 0x7fe2201067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:36328/0 (socket says 172.21.15.155:36328) 2024-02-10T09:09:29.189 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:29.186+0000 7fe21e59c700 1 -- 172.21.15.155:0/2470068723 learned_addr learned my addr 172.21.15.155:0/2470068723 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:09:29.189 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:29.187+0000 7fe21e59c700 1 -- 172.21.15.155:0/2470068723 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe208005040 con 0x7fe2200f5410 2024-02-10T09:09:29.189 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:29.187+0000 7fe21e59c700 1 --2- 172.21.15.155:0/2470068723 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2200f5410 0x7fe2201067d0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7fe208005010 tx=0x7fe20800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:09:29.189 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:29.187+0000 7fe216ffd700 1 -- 172.21.15.155:0/2470068723 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe208009320 con 0x7fe2200f5410 2024-02-10T09:09:29.190 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:29.187+0000 7fe224c6e700 1 -- 172.21.15.155:0/2470068723 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe220106ec0 con 0x7fe2200f5410 2024-02-10T09:09:29.190 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:29.187+0000 7fe216ffd700 1 -- 172.21.15.155:0/2470068723 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe208013380 con 0x7fe2200f5410 2024-02-10T09:09:29.190 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:29.187+0000 7fe216ffd700 1 -- 172.21.15.155:0/2470068723 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe208014e50 con 0x7fe2200f5410 2024-02-10T09:09:29.190 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:29.187+0000 7fe224c6e700 1 -- 172.21.15.155:0/2470068723 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe220107350 con 0x7fe2200f5410 2024-02-10T09:09:29.191 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:29.188+0000 7fe216ffd700 1 -- 172.21.15.155:0/2470068723 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe2080095f0 con 0x7fe2200f5410 2024-02-10T09:09:29.191 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:29.188+0000 7fe224c6e700 1 -- 172.21.15.155:0/2470068723 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe220045ad0 con 0x7fe2200f5410 2024-02-10T09:09:29.191 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:29.188+0000 7fe216ffd700 1 --2- 172.21.15.155:0/2470068723 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fe20c02ff10 0x7fe20c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:09:29.191 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:29.188+0000 7fe216ffd700 1 -- 172.21.15.155:0/2470068723 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe208040e90 con 0x7fe2200f5410 2024-02-10T09:09:29.191 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:29.188+0000 7fe21dd9b700 1 --2- 172.21.15.155:0/2470068723 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fe20c02ff10 0x7fe20c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:09:29.191 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:29.188+0000 7fe21dd9b700 1 --2- 172.21.15.155:0/2470068723 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fe20c02ff10 0x7fe20c0323d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fe2100096a0 tx=0x7fe210006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:09:29.193 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:29.193+0000 7fe216ffd700 1 -- 172.21.15.155:0/2470068723 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe208014420 con 0x7fe2200f5410 2024-02-10T09:09:29.398 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:29.397+0000 7fe224c6e700 1 -- 172.21.15.155:0/2470068723 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe220108200 con 0x7fe2200f5410 2024-02-10T09:09:29.399 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:29.398+0000 7fe216ffd700 1 -- 172.21.15.155:0/2470068723 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe208010230 con 0x7fe2200f5410 2024-02-10T09:09:29.399 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:09:29.399 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:09:29.400 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:29.399+0000 7fe224c6e700 1 -- 172.21.15.155:0/2470068723 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fe20c02ff10 msgr2=0x7fe20c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:09:29.400 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:29.399+0000 7fe224c6e700 1 --2- 172.21.15.155:0/2470068723 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fe20c02ff10 0x7fe20c0323d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fe2100096a0 tx=0x7fe210006b40).stop 2024-02-10T09:09:29.401 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:29.399+0000 7fe224c6e700 1 -- 172.21.15.155:0/2470068723 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2200f5410 msgr2=0x7fe2201067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:09:29.401 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:29.399+0000 7fe224c6e700 1 --2- 172.21.15.155:0/2470068723 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2200f5410 0x7fe2201067d0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7fe208005010 tx=0x7fe20800b040).stop 2024-02-10T09:09:29.401 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:29.399+0000 7fe224c6e700 1 -- 172.21.15.155:0/2470068723 shutdown_connections 2024-02-10T09:09:29.401 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:29.399+0000 7fe224c6e700 1 --2- 172.21.15.155:0/2470068723 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fe20c02ff10 0x7fe20c0323d0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:09:29.401 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:29.399+0000 7fe224c6e700 1 --2- 172.21.15.155:0/2470068723 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe2200f5410 0x7fe2201067d0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:09:29.401 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:29.399+0000 7fe224c6e700 1 -- 172.21.15.155:0/2470068723 >> 172.21.15.155:0/2470068723 conn(0x7fe2200f0b70 msgr2=0x7fe22005fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:09:29.401 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:29.399+0000 7fe224c6e700 1 -- 172.21.15.155:0/2470068723 shutdown_connections 2024-02-10T09:09:29.401 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:29.399+0000 7fe224c6e700 1 -- 172.21.15.155:0/2470068723 wait complete. 2024-02-10T09:09:29.401 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:09:30.152 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:09:29 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:09:29.399244+0000 mon.smithi077 (mon.0) 303 : audit [DBG] from='client.? 172.21.15.155:0/2470068723' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:09:31.049 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:09:31.049 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:09:31.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:09:30 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:09:29.735613+0000 mgr.smithi077.dxntvw (mgr.14162) 332 : cluster [DBG] pgmap v316: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:09:33.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:09:32 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:09:31.735961+0000 mgr.smithi077.dxntvw (mgr.14162) 333 : cluster [DBG] pgmap v317: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:09:33.198 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:33.196+0000 7f999a01f700 1 -- 172.21.15.155:0/2334243103 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99940f6510 msgr2=0x7f99940f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:09:33.198 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:33.196+0000 7f999a01f700 1 --2- 172.21.15.155:0/2334243103 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99940f6510 0x7f99940f6930 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7f997c009fb0 tx=0x7f997c009b30).stop 2024-02-10T09:09:33.198 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:33.196+0000 7f999a01f700 1 -- 172.21.15.155:0/2334243103 shutdown_connections 2024-02-10T09:09:33.198 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:33.196+0000 7f999a01f700 1 --2- 172.21.15.155:0/2334243103 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99940f6510 0x7f99940f6930 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:09:33.199 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:33.196+0000 7f999a01f700 1 -- 172.21.15.155:0/2334243103 >> 172.21.15.155:0/2334243103 conn(0x7f99940f1c70 msgr2=0x7f99940f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:09:33.199 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:33.196+0000 7f999a01f700 1 -- 172.21.15.155:0/2334243103 shutdown_connections 2024-02-10T09:09:33.199 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:33.196+0000 7f999a01f700 1 -- 172.21.15.155:0/2334243103 wait complete. 2024-02-10T09:09:33.199 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:33.197+0000 7f999a01f700 1 Processor -- start 2024-02-10T09:09:33.199 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:33.197+0000 7f999a01f700 1 -- start start 2024-02-10T09:09:33.199 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:33.197+0000 7f999a01f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99940f6510 0x7f99941067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:09:33.199 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:33.197+0000 7f999a01f700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9994106cd0 con 0x7f99940f6510 2024-02-10T09:09:33.201 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:33.198+0000 7f99937fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99940f6510 0x7f99941067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:09:33.201 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:33.198+0000 7f99937fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99940f6510 0x7f99941067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:44154/0 (socket says 172.21.15.155:44154) 2024-02-10T09:09:33.201 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:33.198+0000 7f99937fe700 1 -- 172.21.15.155:0/1410967979 learned_addr learned my addr 172.21.15.155:0/1410967979 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:09:33.201 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:33.198+0000 7f99937fe700 1 -- 172.21.15.155:0/1410967979 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f997c005040 con 0x7f99940f6510 2024-02-10T09:09:33.201 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:33.198+0000 7f99937fe700 1 --2- 172.21.15.155:0/1410967979 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99940f6510 0x7f99941067e0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f997c005010 tx=0x7f997c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:09:33.202 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:33.198+0000 7f998bfff700 1 -- 172.21.15.155:0/1410967979 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f997c009320 con 0x7f99940f6510 2024-02-10T09:09:33.202 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:33.198+0000 7f999a01f700 1 -- 172.21.15.155:0/1410967979 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9994106ed0 con 0x7f99940f6510 2024-02-10T09:09:33.202 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:33.198+0000 7f998bfff700 1 -- 172.21.15.155:0/1410967979 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f997c013380 con 0x7f99940f6510 2024-02-10T09:09:33.202 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:33.198+0000 7f998bfff700 1 -- 172.21.15.155:0/1410967979 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f997c014e50 con 0x7f99940f6510 2024-02-10T09:09:33.202 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:33.198+0000 7f999a01f700 1 -- 172.21.15.155:0/1410967979 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9994107360 con 0x7f99940f6510 2024-02-10T09:09:33.202 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:33.199+0000 7f998bfff700 1 -- 172.21.15.155:0/1410967979 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f997c014420 con 0x7f99940f6510 2024-02-10T09:09:33.202 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:33.199+0000 7f999a01f700 1 -- 172.21.15.155:0/1410967979 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9994045ad0 con 0x7f99940f6510 2024-02-10T09:09:33.202 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:33.199+0000 7f998bfff700 1 --2- 172.21.15.155:0/1410967979 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f998002ff10 0x7f99800323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:09:33.202 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:33.199+0000 7f998bfff700 1 -- 172.21.15.155:0/1410967979 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f997c040f40 con 0x7f99940f6510 2024-02-10T09:09:33.202 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:33.199+0000 7f9992ffd700 1 --2- 172.21.15.155:0/1410967979 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f998002ff10 0x7f99800323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:09:33.202 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:33.200+0000 7f9992ffd700 1 --2- 172.21.15.155:0/1410967979 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f998002ff10 0x7f99800323d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f99840096a0 tx=0x7f9984006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:09:33.205 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:33.205+0000 7f998bfff700 1 -- 172.21.15.155:0/1410967979 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f997c0095f0 con 0x7f99940f6510 2024-02-10T09:09:33.413 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:33.412+0000 7f999a01f700 1 -- 172.21.15.155:0/1410967979 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9994108210 con 0x7f99940f6510 2024-02-10T09:09:33.414 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:33.414+0000 7f998bfff700 1 -- 172.21.15.155:0/1410967979 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f997c01db70 con 0x7f99940f6510 2024-02-10T09:09:33.415 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:09:33.415 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:09:33.416 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:33.414+0000 7f999a01f700 1 -- 172.21.15.155:0/1410967979 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f998002ff10 msgr2=0x7f99800323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:09:33.416 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:33.414+0000 7f999a01f700 1 --2- 172.21.15.155:0/1410967979 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f998002ff10 0x7f99800323d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f99840096a0 tx=0x7f9984006b40).stop 2024-02-10T09:09:33.416 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:33.414+0000 7f999a01f700 1 -- 172.21.15.155:0/1410967979 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99940f6510 msgr2=0x7f99941067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:09:33.416 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:33.414+0000 7f999a01f700 1 --2- 172.21.15.155:0/1410967979 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99940f6510 0x7f99941067e0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f997c005010 tx=0x7f997c00b040).stop 2024-02-10T09:09:33.416 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:33.415+0000 7f999a01f700 1 -- 172.21.15.155:0/1410967979 shutdown_connections 2024-02-10T09:09:33.416 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:33.415+0000 7f999a01f700 1 --2- 172.21.15.155:0/1410967979 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f998002ff10 0x7f99800323d0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:09:33.417 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:33.415+0000 7f999a01f700 1 --2- 172.21.15.155:0/1410967979 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99940f6510 0x7f99941067e0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:09:33.417 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:33.415+0000 7f999a01f700 1 -- 172.21.15.155:0/1410967979 >> 172.21.15.155:0/1410967979 conn(0x7f99940f1c70 msgr2=0x7f99940f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:09:33.417 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:33.415+0000 7f999a01f700 1 -- 172.21.15.155:0/1410967979 shutdown_connections 2024-02-10T09:09:33.417 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:33.415+0000 7f999a01f700 1 -- 172.21.15.155:0/1410967979 wait complete. 2024-02-10T09:09:33.417 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:09:34.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:09:33 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:09:33.414407+0000 mon.smithi077 (mon.0) 304 : audit [DBG] from='client.? 172.21.15.155:0/1410967979' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:09:35.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:09:34 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:09:33.736445+0000 mgr.smithi077.dxntvw (mgr.14162) 334 : cluster [DBG] pgmap v318: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:09:35.326 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:09:35.326 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:09:37.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:09:36 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:09:35.736768+0000 mgr.smithi077.dxntvw (mgr.14162) 335 : cluster [DBG] pgmap v319: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:09:37.592 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:37.591+0000 7f403ce05700 1 -- 172.21.15.155:0/2468545423 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40380f51a0 msgr2=0x7f40380f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:09:37.593 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:37.591+0000 7f403ce05700 1 --2- 172.21.15.155:0/2468545423 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40380f51a0 0x7f40380f55c0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f4020009fb0 tx=0x7f4020009b30).stop 2024-02-10T09:09:37.593 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:37.591+0000 7f403ce05700 1 -- 172.21.15.155:0/2468545423 shutdown_connections 2024-02-10T09:09:37.593 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:37.591+0000 7f403ce05700 1 --2- 172.21.15.155:0/2468545423 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40380f51a0 0x7f40380f55c0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:09:37.593 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:37.591+0000 7f403ce05700 1 -- 172.21.15.155:0/2468545423 >> 172.21.15.155:0/2468545423 conn(0x7f40380f09a0 msgr2=0x7f40380f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:09:37.593 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:37.591+0000 7f403ce05700 1 -- 172.21.15.155:0/2468545423 shutdown_connections 2024-02-10T09:09:37.593 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:37.591+0000 7f403ce05700 1 -- 172.21.15.155:0/2468545423 wait complete. 2024-02-10T09:09:37.593 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:37.591+0000 7f403ce05700 1 Processor -- start 2024-02-10T09:09:37.593 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:37.592+0000 7f403ce05700 1 -- start start 2024-02-10T09:09:37.593 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:37.592+0000 7f403ce05700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40380f51a0 0x7f4038109290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:09:37.593 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:37.592+0000 7f403ce05700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4038060280 con 0x7f40380f51a0 2024-02-10T09:09:37.593 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:37.592+0000 7f40377fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40380f51a0 0x7f4038109290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:09:37.594 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:37.592+0000 7f40377fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40380f51a0 0x7f4038109290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:44176/0 (socket says 172.21.15.155:44176) 2024-02-10T09:09:37.594 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:37.592+0000 7f40377fe700 1 -- 172.21.15.155:0/1356620655 learned_addr learned my addr 172.21.15.155:0/1356620655 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:09:37.594 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:37.593+0000 7f40377fe700 1 -- 172.21.15.155:0/1356620655 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4020005040 con 0x7f40380f51a0 2024-02-10T09:09:37.594 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:37.593+0000 7f40377fe700 1 --2- 172.21.15.155:0/1356620655 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40380f51a0 0x7f4038109290 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f40200089f0 tx=0x7f402000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:09:37.595 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:37.593+0000 7f402ffff700 1 -- 172.21.15.155:0/1356620655 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4020009320 con 0x7f40380f51a0 2024-02-10T09:09:37.595 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:37.593+0000 7f403ce05700 1 -- 172.21.15.155:0/1356620655 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4038060480 con 0x7f40380f51a0 2024-02-10T09:09:37.595 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:37.593+0000 7f402ffff700 1 -- 172.21.15.155:0/1356620655 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4020013380 con 0x7f40380f51a0 2024-02-10T09:09:37.595 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:37.593+0000 7f402ffff700 1 -- 172.21.15.155:0/1356620655 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4020014e20 con 0x7f40380f51a0 2024-02-10T09:09:37.596 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:37.593+0000 7f403ce05700 1 -- 172.21.15.155:0/1356620655 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40381099c0 con 0x7f40380f51a0 2024-02-10T09:09:37.596 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:37.594+0000 7f402ffff700 1 -- 172.21.15.155:0/1356620655 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f40200095f0 con 0x7f40380f51a0 2024-02-10T09:09:37.596 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:37.594+0000 7f402ffff700 1 --2- 172.21.15.155:0/1356620655 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f402402ff10 0x7f40240323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:09:37.596 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:37.594+0000 7f403ce05700 1 -- 172.21.15.155:0/1356620655 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4038045ad0 con 0x7f40380f51a0 2024-02-10T09:09:37.596 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:37.594+0000 7f402ffff700 1 -- 172.21.15.155:0/1356620655 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4020040ef0 con 0x7f40380f51a0 2024-02-10T09:09:37.596 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:37.594+0000 7f4036ffd700 1 --2- 172.21.15.155:0/1356620655 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f402402ff10 0x7f40240323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:09:37.596 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:37.594+0000 7f4036ffd700 1 --2- 172.21.15.155:0/1356620655 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f402402ff10 0x7f40240323d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f40280096a0 tx=0x7f4028006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:09:37.598 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:37.599+0000 7f402ffff700 1 -- 172.21.15.155:0/1356620655 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f402001d7f0 con 0x7f40380f51a0 2024-02-10T09:09:37.804 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:37.802+0000 7f403ce05700 1 -- 172.21.15.155:0/1356620655 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f40380fad30 con 0x7f40380f51a0 2024-02-10T09:09:37.804 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:37.803+0000 7f402ffff700 1 -- 172.21.15.155:0/1356620655 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4020010230 con 0x7f40380f51a0 2024-02-10T09:09:37.804 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:09:37.804 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:09:37.805 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:37.804+0000 7f403ce05700 1 -- 172.21.15.155:0/1356620655 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f402402ff10 msgr2=0x7f40240323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:09:37.805 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:37.804+0000 7f403ce05700 1 --2- 172.21.15.155:0/1356620655 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f402402ff10 0x7f40240323d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f40280096a0 tx=0x7f4028006b40).stop 2024-02-10T09:09:37.805 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:37.805+0000 7f403ce05700 1 -- 172.21.15.155:0/1356620655 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40380f51a0 msgr2=0x7f4038109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:09:37.805 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:37.805+0000 7f403ce05700 1 --2- 172.21.15.155:0/1356620655 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40380f51a0 0x7f4038109290 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f40200089f0 tx=0x7f402000b040).stop 2024-02-10T09:09:37.805 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:37.805+0000 7f403ce05700 1 -- 172.21.15.155:0/1356620655 shutdown_connections 2024-02-10T09:09:37.806 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:37.805+0000 7f403ce05700 1 --2- 172.21.15.155:0/1356620655 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f402402ff10 0x7f40240323d0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:09:37.806 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:37.805+0000 7f403ce05700 1 --2- 172.21.15.155:0/1356620655 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40380f51a0 0x7f4038109290 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:09:37.806 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:37.805+0000 7f403ce05700 1 -- 172.21.15.155:0/1356620655 >> 172.21.15.155:0/1356620655 conn(0x7f40380f09a0 msgr2=0x7f40380f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:09:37.806 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:37.805+0000 7f403ce05700 1 -- 172.21.15.155:0/1356620655 shutdown_connections 2024-02-10T09:09:37.806 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:37.805+0000 7f403ce05700 1 -- 172.21.15.155:0/1356620655 wait complete. 2024-02-10T09:09:37.806 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:09:39.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:09:38 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:09:37.737131+0000 mgr.smithi077.dxntvw (mgr.14162) 336 : cluster [DBG] pgmap v320: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:09:39.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:09:38 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:09:37.803717+0000 mon.smithi077 (mon.0) 305 : audit [DBG] from='client.? 172.21.15.155:0/1356620655' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:09:39.641 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:09:39.642 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:09:41.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:09:40 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:09:39.737557+0000 mgr.smithi077.dxntvw (mgr.14162) 337 : cluster [DBG] pgmap v321: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:09:41.742 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:41.741+0000 7f3714145700 1 -- 172.21.15.155:0/3123462057 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f370c0f6510 msgr2=0x7f370c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:09:41.742 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:41.741+0000 7f3714145700 1 --2- 172.21.15.155:0/3123462057 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f370c0f6510 0x7f370c0f6930 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f3700009fb0 tx=0x7f3700009b30).stop 2024-02-10T09:09:41.742 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:41.741+0000 7f3714145700 1 -- 172.21.15.155:0/3123462057 shutdown_connections 2024-02-10T09:09:41.742 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:41.741+0000 7f3714145700 1 --2- 172.21.15.155:0/3123462057 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f370c0f6510 0x7f370c0f6930 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:09:41.742 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:41.741+0000 7f3714145700 1 -- 172.21.15.155:0/3123462057 >> 172.21.15.155:0/3123462057 conn(0x7f370c0f1c70 msgr2=0x7f370c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:09:41.742 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:41.741+0000 7f3714145700 1 -- 172.21.15.155:0/3123462057 shutdown_connections 2024-02-10T09:09:41.742 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:41.741+0000 7f3714145700 1 -- 172.21.15.155:0/3123462057 wait complete. 2024-02-10T09:09:41.743 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:41.741+0000 7f3714145700 1 Processor -- start 2024-02-10T09:09:41.743 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:41.742+0000 7f3714145700 1 -- start start 2024-02-10T09:09:41.743 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:41.742+0000 7f3714145700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f370c0f6510 0x7f370c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:09:41.743 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:41.742+0000 7f3714145700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f370c069e20 con 0x7f370c0f6510 2024-02-10T09:09:41.743 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:41.742+0000 7f3711ee1700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f370c0f6510 0x7f370c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:09:41.743 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:41.742+0000 7f3711ee1700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f370c0f6510 0x7f370c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:44198/0 (socket says 172.21.15.155:44198) 2024-02-10T09:09:41.743 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:41.742+0000 7f3711ee1700 1 -- 172.21.15.155:0/3552386744 learned_addr learned my addr 172.21.15.155:0/3552386744 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:09:41.744 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:41.742+0000 7f3711ee1700 1 -- 172.21.15.155:0/3552386744 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3700005040 con 0x7f370c0f6510 2024-02-10T09:09:41.745 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:41.742+0000 7f3711ee1700 1 --2- 172.21.15.155:0/3552386744 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f370c0f6510 0x7f370c06b9a0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f3700005010 tx=0x7f370000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:09:41.745 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:41.743+0000 7f36fe7fc700 1 -- 172.21.15.155:0/3552386744 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3700009320 con 0x7f370c0f6510 2024-02-10T09:09:41.745 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:41.743+0000 7f36fe7fc700 1 -- 172.21.15.155:0/3552386744 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3700013380 con 0x7f370c0f6510 2024-02-10T09:09:41.745 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:41.743+0000 7f3714145700 1 -- 172.21.15.155:0/3552386744 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f370c069fc0 con 0x7f370c0f6510 2024-02-10T09:09:41.745 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:41.743+0000 7f36fe7fc700 1 -- 172.21.15.155:0/3552386744 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3700014e50 con 0x7f370c0f6510 2024-02-10T09:09:41.745 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:41.743+0000 7f3714145700 1 -- 172.21.15.155:0/3552386744 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f370c06a450 con 0x7f370c0f6510 2024-02-10T09:09:41.745 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:41.744+0000 7f36fe7fc700 1 -- 172.21.15.155:0/3552386744 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f37000095f0 con 0x7f370c0f6510 2024-02-10T09:09:41.745 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:41.744+0000 7f3714145700 1 -- 172.21.15.155:0/3552386744 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f370c059100 con 0x7f370c0f6510 2024-02-10T09:09:41.745 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:41.744+0000 7f36fe7fc700 1 --2- 172.21.15.155:0/3552386744 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f36f802ff10 0x7f36f80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:09:41.745 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:41.744+0000 7f36fe7fc700 1 -- 172.21.15.155:0/3552386744 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3700040e90 con 0x7f370c0f6510 2024-02-10T09:09:41.745 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:41.744+0000 7f37116e0700 1 --2- 172.21.15.155:0/3552386744 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f36f802ff10 0x7f36f80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:09:41.745 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:41.744+0000 7f37116e0700 1 --2- 172.21.15.155:0/3552386744 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f36f802ff10 0x7f36f80323d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f37080096a0 tx=0x7f3708006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:09:41.748 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:41.749+0000 7f36fe7fc700 1 -- 172.21.15.155:0/3552386744 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3700014420 con 0x7f370c0f6510 2024-02-10T09:09:41.953 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:41.953+0000 7f3714145700 1 -- 172.21.15.155:0/3552386744 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f370c0fc0a0 con 0x7f370c0f6510 2024-02-10T09:09:41.954 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:41.954+0000 7f36fe7fc700 1 -- 172.21.15.155:0/3552386744 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3700010230 con 0x7f370c0f6510 2024-02-10T09:09:41.954 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:09:41.954 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:09:41.956 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:41.955+0000 7f3714145700 1 -- 172.21.15.155:0/3552386744 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f36f802ff10 msgr2=0x7f36f80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:09:41.956 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:41.955+0000 7f3714145700 1 --2- 172.21.15.155:0/3552386744 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f36f802ff10 0x7f36f80323d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f37080096a0 tx=0x7f3708006b40).stop 2024-02-10T09:09:41.956 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:41.955+0000 7f3714145700 1 -- 172.21.15.155:0/3552386744 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f370c0f6510 msgr2=0x7f370c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:09:41.956 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:41.955+0000 7f3714145700 1 --2- 172.21.15.155:0/3552386744 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f370c0f6510 0x7f370c06b9a0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f3700005010 tx=0x7f370000b040).stop 2024-02-10T09:09:41.956 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:41.955+0000 7f3714145700 1 -- 172.21.15.155:0/3552386744 shutdown_connections 2024-02-10T09:09:41.956 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:41.955+0000 7f3714145700 1 --2- 172.21.15.155:0/3552386744 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f36f802ff10 0x7f36f80323d0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:09:41.956 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:41.955+0000 7f3714145700 1 --2- 172.21.15.155:0/3552386744 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f370c0f6510 0x7f370c06b9a0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:09:41.956 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:41.955+0000 7f3714145700 1 -- 172.21.15.155:0/3552386744 >> 172.21.15.155:0/3552386744 conn(0x7f370c0f1c70 msgr2=0x7f370c0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:09:41.956 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:41.955+0000 7f3714145700 1 -- 172.21.15.155:0/3552386744 shutdown_connections 2024-02-10T09:09:41.956 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:41.955+0000 7f3714145700 1 -- 172.21.15.155:0/3552386744 wait complete. 2024-02-10T09:09:41.957 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:09:43.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:09:42 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:09:41.737869+0000 mgr.smithi077.dxntvw (mgr.14162) 338 : cluster [DBG] pgmap v322: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:09:43.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:09:42 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:09:41.954135+0000 mon.smithi077 (mon.0) 306 : audit [DBG] from='client.? 172.21.15.155:0/3552386744' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:09:43.808 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:09:43.808 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:09:45.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:09:44 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:09:43.738195+0000 mgr.smithi077.dxntvw (mgr.14162) 339 : cluster [DBG] pgmap v323: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:09:45.899 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:45.896+0000 7fa1b6a17700 1 -- 172.21.15.155:0/168351955 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1b00f5410 msgr2=0x7fa1b00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:09:45.899 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:45.896+0000 7fa1b6a17700 1 --2- 172.21.15.155:0/168351955 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1b00f5410 0x7fa1b00f5830 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7fa1a0009fb0 tx=0x7fa1a0009b30).stop 2024-02-10T09:09:45.900 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:45.896+0000 7fa1b6a17700 1 -- 172.21.15.155:0/168351955 shutdown_connections 2024-02-10T09:09:45.900 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:45.896+0000 7fa1b6a17700 1 --2- 172.21.15.155:0/168351955 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1b00f5410 0x7fa1b00f5830 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:09:45.900 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:45.896+0000 7fa1b6a17700 1 -- 172.21.15.155:0/168351955 >> 172.21.15.155:0/168351955 conn(0x7fa1b00f0b70 msgr2=0x7fa1b00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:09:45.900 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:45.896+0000 7fa1b6a17700 1 -- 172.21.15.155:0/168351955 shutdown_connections 2024-02-10T09:09:45.900 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:45.896+0000 7fa1b6a17700 1 -- 172.21.15.155:0/168351955 wait complete. 2024-02-10T09:09:45.900 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:45.896+0000 7fa1b6a17700 1 Processor -- start 2024-02-10T09:09:45.900 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:45.897+0000 7fa1b6a17700 1 -- start start 2024-02-10T09:09:45.900 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:45.897+0000 7fa1b6a17700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1b00f5410 0x7fa1b01094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:09:45.900 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:45.897+0000 7fa1b6a17700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1b0060240 con 0x7fa1b00f5410 2024-02-10T09:09:45.900 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:45.897+0000 7fa1affff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1b00f5410 0x7fa1b01094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:09:45.901 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:45.897+0000 7fa1affff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1b00f5410 0x7fa1b01094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:39508/0 (socket says 172.21.15.155:39508) 2024-02-10T09:09:45.901 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:45.897+0000 7fa1affff700 1 -- 172.21.15.155:0/3101682388 learned_addr learned my addr 172.21.15.155:0/3101682388 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:09:45.901 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:45.898+0000 7fa1affff700 1 -- 172.21.15.155:0/3101682388 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1a0005040 con 0x7fa1b00f5410 2024-02-10T09:09:45.901 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:45.898+0000 7fa1affff700 1 --2- 172.21.15.155:0/3101682388 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1b00f5410 0x7fa1b01094e0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7fa1a0005010 tx=0x7fa1a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:09:45.901 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:45.898+0000 7fa1acff9700 1 -- 172.21.15.155:0/3101682388 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa1a0009320 con 0x7fa1b00f5410 2024-02-10T09:09:45.901 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:45.898+0000 7fa1acff9700 1 -- 172.21.15.155:0/3101682388 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa1a0013380 con 0x7fa1b00f5410 2024-02-10T09:09:45.901 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:45.898+0000 7fa1b6a17700 1 -- 172.21.15.155:0/3101682388 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1b0060440 con 0x7fa1b00f5410 2024-02-10T09:09:45.901 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:45.898+0000 7fa1acff9700 1 -- 172.21.15.155:0/3101682388 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa1a0014e50 con 0x7fa1b00f5410 2024-02-10T09:09:45.901 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:45.898+0000 7fa1b6a17700 1 -- 172.21.15.155:0/3101682388 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1b0109b90 con 0x7fa1b00f5410 2024-02-10T09:09:45.901 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:45.899+0000 7fa1acff9700 1 -- 172.21.15.155:0/3101682388 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa1a00095f0 con 0x7fa1b00f5410 2024-02-10T09:09:45.901 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:45.899+0000 7fa1b6a17700 1 -- 172.21.15.155:0/3101682388 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1b0045ad0 con 0x7fa1b00f5410 2024-02-10T09:09:45.902 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:45.899+0000 7fa1acff9700 1 --2- 172.21.15.155:0/3101682388 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fa19802ff10 0x7fa1980323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:09:45.902 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:45.899+0000 7fa1acff9700 1 -- 172.21.15.155:0/3101682388 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa1a0040e90 con 0x7fa1b00f5410 2024-02-10T09:09:45.902 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:45.900+0000 7fa1af7fe700 1 --2- 172.21.15.155:0/3101682388 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fa19802ff10 0x7fa1980323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:09:45.902 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:45.900+0000 7fa1af7fe700 1 --2- 172.21.15.155:0/3101682388 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fa19802ff10 0x7fa1980323d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fa1a40096a0 tx=0x7fa1a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:09:45.904 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:45.904+0000 7fa1acff9700 1 -- 172.21.15.155:0/3101682388 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa1a0014420 con 0x7fa1b00f5410 2024-02-10T09:09:46.113 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:46.113+0000 7fa1b6a17700 1 -- 172.21.15.155:0/3101682388 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa1b010a3b0 con 0x7fa1b00f5410 2024-02-10T09:09:46.113 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:46.113+0000 7fa1acff9700 1 -- 172.21.15.155:0/3101682388 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa1a0010230 con 0x7fa1b00f5410 2024-02-10T09:09:46.114 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:09:46.114 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:09:46.115 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:46.115+0000 7fa1b6a17700 1 -- 172.21.15.155:0/3101682388 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fa19802ff10 msgr2=0x7fa1980323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:09:46.115 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:46.115+0000 7fa1b6a17700 1 --2- 172.21.15.155:0/3101682388 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fa19802ff10 0x7fa1980323d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fa1a40096a0 tx=0x7fa1a4006b40).stop 2024-02-10T09:09:46.116 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:46.115+0000 7fa1b6a17700 1 -- 172.21.15.155:0/3101682388 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1b00f5410 msgr2=0x7fa1b01094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:09:46.116 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:46.115+0000 7fa1b6a17700 1 --2- 172.21.15.155:0/3101682388 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1b00f5410 0x7fa1b01094e0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7fa1a0005010 tx=0x7fa1a000b040).stop 2024-02-10T09:09:46.116 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:46.115+0000 7fa1b6a17700 1 -- 172.21.15.155:0/3101682388 shutdown_connections 2024-02-10T09:09:46.116 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:46.115+0000 7fa1b6a17700 1 --2- 172.21.15.155:0/3101682388 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fa19802ff10 0x7fa1980323d0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:09:46.116 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:46.115+0000 7fa1b6a17700 1 --2- 172.21.15.155:0/3101682388 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa1b00f5410 0x7fa1b01094e0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:09:46.116 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:46.115+0000 7fa1b6a17700 1 -- 172.21.15.155:0/3101682388 >> 172.21.15.155:0/3101682388 conn(0x7fa1b00f0b70 msgr2=0x7fa1b00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:09:46.116 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:46.115+0000 7fa1b6a17700 1 -- 172.21.15.155:0/3101682388 shutdown_connections 2024-02-10T09:09:46.116 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:46.115+0000 7fa1b6a17700 1 -- 172.21.15.155:0/3101682388 wait complete. 2024-02-10T09:09:46.117 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:09:47.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:09:46 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:09:45.738494+0000 mgr.smithi077.dxntvw (mgr.14162) 340 : cluster [DBG] pgmap v324: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:09:47.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:09:46 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:09:46.113937+0000 mon.smithi077 (mon.0) 307 : audit [DBG] from='client.? 172.21.15.155:0/3101682388' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:09:47.902 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:09:47.903 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:09:48.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:09:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:09:47.099634+0000 mon.smithi077 (mon.0) 308 : audit [INF] from='mgr.14162 172.21.15.77:0/2029106299' entity='mgr.smithi077.dxntvw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi077.dxntvw/mirror_snapshot_schedule"}]: dispatch 2024-02-10T09:09:48.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:09:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:09:47.104966+0000 mon.smithi077 (mon.0) 309 : audit [INF] from='mgr.14162 172.21.15.77:0/2029106299' entity='mgr.smithi077.dxntvw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi077.dxntvw/trash_purge_schedule"}]: dispatch 2024-02-10T09:09:49.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:09:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:09:47.738826+0000 mgr.smithi077.dxntvw (mgr.14162) 341 : cluster [DBG] pgmap v325: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:09:50.062 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:50.060+0000 7f4018c87700 1 -- 172.21.15.155:0/1992040453 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40140601e0 msgr2=0x7f40140f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:09:50.062 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:50.060+0000 7f4018c87700 1 --2- 172.21.15.155:0/1992040453 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40140601e0 0x7f40140f8f20 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f4004009fb0 tx=0x7f4004009b30).stop 2024-02-10T09:09:50.062 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:50.060+0000 7f4018c87700 1 -- 172.21.15.155:0/1992040453 shutdown_connections 2024-02-10T09:09:50.062 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:50.060+0000 7f4018c87700 1 --2- 172.21.15.155:0/1992040453 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40140601e0 0x7f40140f8f20 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:09:50.062 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:50.060+0000 7f4018c87700 1 -- 172.21.15.155:0/1992040453 >> 172.21.15.155:0/1992040453 conn(0x7f40140f09a0 msgr2=0x7f40140f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:09:50.062 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:50.060+0000 7f4018c87700 1 -- 172.21.15.155:0/1992040453 shutdown_connections 2024-02-10T09:09:50.062 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:50.060+0000 7f4018c87700 1 -- 172.21.15.155:0/1992040453 wait complete. 2024-02-10T09:09:50.062 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:50.060+0000 7f4018c87700 1 Processor -- start 2024-02-10T09:09:50.063 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:50.061+0000 7f4018c87700 1 -- start start 2024-02-10T09:09:50.063 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:50.061+0000 7f4018c87700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40140601e0 0x7f4014106600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:09:50.063 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:50.061+0000 7f4018c87700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4014106af0 con 0x7f40140601e0 2024-02-10T09:09:50.063 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:50.061+0000 7f40137fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40140601e0 0x7f4014106600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:09:50.063 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:50.061+0000 7f40137fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40140601e0 0x7f4014106600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:39516/0 (socket says 172.21.15.155:39516) 2024-02-10T09:09:50.063 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:50.061+0000 7f40137fe700 1 -- 172.21.15.155:0/3203478147 learned_addr learned my addr 172.21.15.155:0/3203478147 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:09:50.063 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:50.061+0000 7f40137fe700 1 -- 172.21.15.155:0/3203478147 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4004005040 con 0x7f40140601e0 2024-02-10T09:09:50.063 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:50.062+0000 7f40137fe700 1 --2- 172.21.15.155:0/3203478147 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40140601e0 0x7f4014106600 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f4004005010 tx=0x7f400400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:09:50.063 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:50.062+0000 7f3ffbfff700 1 -- 172.21.15.155:0/3203478147 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4004009320 con 0x7f40140601e0 2024-02-10T09:09:50.063 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:50.062+0000 7f4018c87700 1 -- 172.21.15.155:0/3203478147 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4014106cf0 con 0x7f40140601e0 2024-02-10T09:09:50.063 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:50.062+0000 7f3ffbfff700 1 -- 172.21.15.155:0/3203478147 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4004013380 con 0x7f40140601e0 2024-02-10T09:09:50.063 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:50.062+0000 7f3ffbfff700 1 -- 172.21.15.155:0/3203478147 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4004014e50 con 0x7f40140601e0 2024-02-10T09:09:50.063 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:50.062+0000 7f4018c87700 1 -- 172.21.15.155:0/3203478147 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4014107180 con 0x7f40140601e0 2024-02-10T09:09:50.064 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:50.063+0000 7f3ffbfff700 1 -- 172.21.15.155:0/3203478147 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f40040095f0 con 0x7f40140601e0 2024-02-10T09:09:50.064 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:50.063+0000 7f4018c87700 1 -- 172.21.15.155:0/3203478147 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4014045ad0 con 0x7f40140601e0 2024-02-10T09:09:50.065 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:50.063+0000 7f3ffbfff700 1 --2- 172.21.15.155:0/3203478147 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f3ffc02ff10 0x7f3ffc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:09:50.065 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:50.063+0000 7f3ffbfff700 1 -- 172.21.15.155:0/3203478147 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4004040e90 con 0x7f40140601e0 2024-02-10T09:09:50.065 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:50.063+0000 7f4012ffd700 1 --2- 172.21.15.155:0/3203478147 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f3ffc02ff10 0x7f3ffc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:09:50.065 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:50.063+0000 7f4012ffd700 1 --2- 172.21.15.155:0/3203478147 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f3ffc02ff10 0x7f3ffc0323d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f40080096a0 tx=0x7f4008006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:09:50.068 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:50.068+0000 7f3ffbfff700 1 -- 172.21.15.155:0/3203478147 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4004014420 con 0x7f40140601e0 2024-02-10T09:09:50.279 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:50.279+0000 7f4018c87700 1 -- 172.21.15.155:0/3203478147 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4014108030 con 0x7f40140601e0 2024-02-10T09:09:50.280 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:50.280+0000 7f3ffbfff700 1 -- 172.21.15.155:0/3203478147 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4004010230 con 0x7f40140601e0 2024-02-10T09:09:50.280 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:09:50.280 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:09:50.281 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:50.281+0000 7f4018c87700 1 -- 172.21.15.155:0/3203478147 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f3ffc02ff10 msgr2=0x7f3ffc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:09:50.281 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:50.281+0000 7f4018c87700 1 --2- 172.21.15.155:0/3203478147 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f3ffc02ff10 0x7f3ffc0323d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f40080096a0 tx=0x7f4008006b40).stop 2024-02-10T09:09:50.282 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:50.281+0000 7f4018c87700 1 -- 172.21.15.155:0/3203478147 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40140601e0 msgr2=0x7f4014106600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:09:50.282 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:50.281+0000 7f4018c87700 1 --2- 172.21.15.155:0/3203478147 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40140601e0 0x7f4014106600 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f4004005010 tx=0x7f400400b040).stop 2024-02-10T09:09:50.282 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:50.281+0000 7f4018c87700 1 -- 172.21.15.155:0/3203478147 shutdown_connections 2024-02-10T09:09:50.282 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:50.281+0000 7f4018c87700 1 --2- 172.21.15.155:0/3203478147 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f3ffc02ff10 0x7f3ffc0323d0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:09:50.282 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:50.281+0000 7f4018c87700 1 --2- 172.21.15.155:0/3203478147 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f40140601e0 0x7f4014106600 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:09:50.282 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:50.281+0000 7f4018c87700 1 -- 172.21.15.155:0/3203478147 >> 172.21.15.155:0/3203478147 conn(0x7f40140f09a0 msgr2=0x7f40140f13d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:09:50.282 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:50.281+0000 7f4018c87700 1 -- 172.21.15.155:0/3203478147 shutdown_connections 2024-02-10T09:09:50.282 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:50.281+0000 7f4018c87700 1 -- 172.21.15.155:0/3203478147 wait complete. 2024-02-10T09:09:50.282 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:09:51.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:09:50 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:09:49.739106+0000 mgr.smithi077.dxntvw (mgr.14162) 342 : cluster [DBG] pgmap v326: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:09:51.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:09:50 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:09:50.280099+0000 mon.smithi077 (mon.0) 310 : audit [DBG] from='client.? 172.21.15.155:0/3203478147' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:09:52.089 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:09:52.090 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:09:53.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:09:52 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:09:51.739468+0000 mgr.smithi077.dxntvw (mgr.14162) 343 : cluster [DBG] pgmap v327: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:09:54.159 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:54.157+0000 7fed6d184700 1 -- 172.21.15.155:0/2083793894 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed680f62a0 msgr2=0x7fed680f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:09:54.159 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:54.157+0000 7fed6d184700 1 --2- 172.21.15.155:0/2083793894 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed680f62a0 0x7fed680f66c0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7fed58009fb0 tx=0x7fed58009b30).stop 2024-02-10T09:09:54.159 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:54.157+0000 7fed6d184700 1 -- 172.21.15.155:0/2083793894 shutdown_connections 2024-02-10T09:09:54.160 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:54.157+0000 7fed6d184700 1 --2- 172.21.15.155:0/2083793894 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed680f62a0 0x7fed680f66c0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:09:54.160 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:54.157+0000 7fed6d184700 1 -- 172.21.15.155:0/2083793894 >> 172.21.15.155:0/2083793894 conn(0x7fed680f1aa0 msgr2=0x7fed680f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:09:54.160 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:54.158+0000 7fed6d184700 1 -- 172.21.15.155:0/2083793894 shutdown_connections 2024-02-10T09:09:54.160 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:54.158+0000 7fed6d184700 1 -- 172.21.15.155:0/2083793894 wait complete. 2024-02-10T09:09:54.160 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:54.158+0000 7fed6d184700 1 Processor -- start 2024-02-10T09:09:54.160 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:54.158+0000 7fed6d184700 1 -- start start 2024-02-10T09:09:54.160 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:54.158+0000 7fed6d184700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed680f62a0 0x7fed681087e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:09:54.160 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:54.158+0000 7fed6d184700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed68108cd0 con 0x7fed680f62a0 2024-02-10T09:09:54.162 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:54.159+0000 7fed67fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed680f62a0 0x7fed681087e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:09:54.162 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:54.159+0000 7fed67fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed680f62a0 0x7fed681087e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:39394/0 (socket says 172.21.15.155:39394) 2024-02-10T09:09:54.162 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:54.159+0000 7fed67fff700 1 -- 172.21.15.155:0/99048742 learned_addr learned my addr 172.21.15.155:0/99048742 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:09:54.162 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:54.159+0000 7fed67fff700 1 -- 172.21.15.155:0/99048742 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed58005040 con 0x7fed680f62a0 2024-02-10T09:09:54.162 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:54.159+0000 7fed67fff700 1 --2- 172.21.15.155:0/99048742 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed680f62a0 0x7fed681087e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7fed580089f0 tx=0x7fed5800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:09:54.162 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:54.159+0000 7fed64ff9700 1 -- 172.21.15.155:0/99048742 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed58009320 con 0x7fed680f62a0 2024-02-10T09:09:54.163 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:54.159+0000 7fed6d184700 1 -- 172.21.15.155:0/99048742 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed68108ed0 con 0x7fed680f62a0 2024-02-10T09:09:54.163 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:54.159+0000 7fed64ff9700 1 -- 172.21.15.155:0/99048742 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fed58013380 con 0x7fed680f62a0 2024-02-10T09:09:54.163 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:54.159+0000 7fed64ff9700 1 -- 172.21.15.155:0/99048742 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed58014e20 con 0x7fed680f62a0 2024-02-10T09:09:54.163 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:54.159+0000 7fed6d184700 1 -- 172.21.15.155:0/99048742 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed68109360 con 0x7fed680f62a0 2024-02-10T09:09:54.163 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:54.160+0000 7fed64ff9700 1 -- 172.21.15.155:0/99048742 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fed580263a0 con 0x7fed680f62a0 2024-02-10T09:09:54.163 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:54.160+0000 7fed64ff9700 1 --2- 172.21.15.155:0/99048742 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fed5002ff10 0x7fed500323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:09:54.163 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:54.160+0000 7fed64ff9700 1 -- 172.21.15.155:0/99048742 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fed58040f10 con 0x7fed680f62a0 2024-02-10T09:09:54.163 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:54.160+0000 7fed6d184700 1 -- 172.21.15.155:0/99048742 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed68045ad0 con 0x7fed680f62a0 2024-02-10T09:09:54.163 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:54.161+0000 7fed677fe700 1 --2- 172.21.15.155:0/99048742 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fed5002ff10 0x7fed500323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:09:54.163 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:54.161+0000 7fed677fe700 1 --2- 172.21.15.155:0/99048742 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fed5002ff10 0x7fed500323d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fed5c0096a0 tx=0x7fed5c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:09:54.166 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:54.165+0000 7fed64ff9700 1 -- 172.21.15.155:0/99048742 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fed58012920 con 0x7fed680f62a0 2024-02-10T09:09:54.373 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:54.372+0000 7fed6d184700 1 -- 172.21.15.155:0/99048742 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fed680fbf20 con 0x7fed680f62a0 2024-02-10T09:09:54.374 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:54.373+0000 7fed64ff9700 1 -- 172.21.15.155:0/99048742 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fed5802b020 con 0x7fed680f62a0 2024-02-10T09:09:54.374 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:09:54.375 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:09:54.376 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:54.374+0000 7fed6d184700 1 -- 172.21.15.155:0/99048742 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fed5002ff10 msgr2=0x7fed500323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:09:54.376 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:54.374+0000 7fed6d184700 1 --2- 172.21.15.155:0/99048742 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fed5002ff10 0x7fed500323d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fed5c0096a0 tx=0x7fed5c006b40).stop 2024-02-10T09:09:54.376 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:54.374+0000 7fed6d184700 1 -- 172.21.15.155:0/99048742 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed680f62a0 msgr2=0x7fed681087e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:09:54.376 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:54.374+0000 7fed6d184700 1 --2- 172.21.15.155:0/99048742 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed680f62a0 0x7fed681087e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7fed580089f0 tx=0x7fed5800b040).stop 2024-02-10T09:09:54.376 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:54.375+0000 7fed6d184700 1 -- 172.21.15.155:0/99048742 shutdown_connections 2024-02-10T09:09:54.376 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:54.375+0000 7fed6d184700 1 --2- 172.21.15.155:0/99048742 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fed5002ff10 0x7fed500323d0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:09:54.376 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:54.375+0000 7fed6d184700 1 --2- 172.21.15.155:0/99048742 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fed680f62a0 0x7fed681087e0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:09:54.376 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:54.375+0000 7fed6d184700 1 -- 172.21.15.155:0/99048742 >> 172.21.15.155:0/99048742 conn(0x7fed680f1aa0 msgr2=0x7fed680f2460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:09:54.376 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:54.375+0000 7fed6d184700 1 -- 172.21.15.155:0/99048742 shutdown_connections 2024-02-10T09:09:54.377 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:54.375+0000 7fed6d184700 1 -- 172.21.15.155:0/99048742 wait complete. 2024-02-10T09:09:54.377 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:09:55.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:09:54 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:09:53.739855+0000 mgr.smithi077.dxntvw (mgr.14162) 344 : cluster [DBG] pgmap v328: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:09:55.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:09:54 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:09:54.374143+0000 mon.smithi077 (mon.0) 311 : audit [DBG] from='client.? 172.21.15.155:0/99048742' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:09:56.128 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:09:56.128 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:09:57.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:09:56 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:09:55.740148+0000 mgr.smithi077.dxntvw (mgr.14162) 345 : cluster [DBG] pgmap v329: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:09:58.226 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:58.224+0000 7f60609bc700 1 -- 172.21.15.155:0/943627625 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f605c060280 msgr2=0x7f605c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:09:58.227 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:58.224+0000 7f60609bc700 1 --2- 172.21.15.155:0/943627625 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f605c060280 0x7f605c0f90f0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f604c009fb0 tx=0x7f604c009b30).stop 2024-02-10T09:09:58.227 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:58.224+0000 7f60609bc700 1 -- 172.21.15.155:0/943627625 shutdown_connections 2024-02-10T09:09:58.227 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:58.224+0000 7f60609bc700 1 --2- 172.21.15.155:0/943627625 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f605c060280 0x7f605c0f90f0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:09:58.227 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:58.224+0000 7f60609bc700 1 -- 172.21.15.155:0/943627625 >> 172.21.15.155:0/943627625 conn(0x7f605c0f0b70 msgr2=0x7f605c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:09:58.227 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:58.224+0000 7f60609bc700 1 -- 172.21.15.155:0/943627625 shutdown_connections 2024-02-10T09:09:58.227 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:58.225+0000 7f60609bc700 1 -- 172.21.15.155:0/943627625 wait complete. 2024-02-10T09:09:58.227 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:58.225+0000 7f60609bc700 1 Processor -- start 2024-02-10T09:09:58.227 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:58.225+0000 7f60609bc700 1 -- start start 2024-02-10T09:09:58.227 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:58.225+0000 7f60609bc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f605c060280 0x7f605c106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:09:58.227 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:58.225+0000 7f60609bc700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f605c106c70 con 0x7f605c060280 2024-02-10T09:09:58.227 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:58.226+0000 7f605a59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f605c060280 0x7f605c106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:09:58.228 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:58.226+0000 7f605a59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f605c060280 0x7f605c106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:39422/0 (socket says 172.21.15.155:39422) 2024-02-10T09:09:58.228 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:58.226+0000 7f605a59c700 1 -- 172.21.15.155:0/403052041 learned_addr learned my addr 172.21.15.155:0/403052041 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:09:58.229 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:58.226+0000 7f605a59c700 1 -- 172.21.15.155:0/403052041 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f604c005040 con 0x7f605c060280 2024-02-10T09:09:58.229 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:58.226+0000 7f605a59c700 1 --2- 172.21.15.155:0/403052041 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f605c060280 0x7f605c106780 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f604c009830 tx=0x7f604c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:09:58.229 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:58.226+0000 7f604affd700 1 -- 172.21.15.155:0/403052041 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f604c009320 con 0x7f605c060280 2024-02-10T09:09:58.229 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:58.226+0000 7f60609bc700 1 -- 172.21.15.155:0/403052041 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f605c106e70 con 0x7f605c060280 2024-02-10T09:09:58.230 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:58.226+0000 7f604affd700 1 -- 172.21.15.155:0/403052041 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f604c013380 con 0x7f605c060280 2024-02-10T09:09:58.230 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:58.226+0000 7f604affd700 1 -- 172.21.15.155:0/403052041 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f604c014420 con 0x7f605c060280 2024-02-10T09:09:58.230 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:58.227+0000 7f60609bc700 1 -- 172.21.15.155:0/403052041 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f605c107300 con 0x7f605c060280 2024-02-10T09:09:58.230 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:58.227+0000 7f604affd700 1 -- 172.21.15.155:0/403052041 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f604c01e420 con 0x7f605c060280 2024-02-10T09:09:58.230 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:58.227+0000 7f60609bc700 1 -- 172.21.15.155:0/403052041 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f605c045ad0 con 0x7f605c060280 2024-02-10T09:09:58.230 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:58.227+0000 7f604affd700 1 --2- 172.21.15.155:0/403052041 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f604402ff60 0x7f6044032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:09:58.230 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:58.227+0000 7f604affd700 1 -- 172.21.15.155:0/403052041 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f604c040ef0 con 0x7f605c060280 2024-02-10T09:09:58.230 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:58.228+0000 7f6059d9b700 1 --2- 172.21.15.155:0/403052041 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f604402ff60 0x7f6044032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:09:58.230 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:58.228+0000 7f6059d9b700 1 --2- 172.21.15.155:0/403052041 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f604402ff60 0x7f6044032420 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f60500096a0 tx=0x7f6050006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:09:58.233 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:58.232+0000 7f604affd700 1 -- 172.21.15.155:0/403052041 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f604c01cb10 con 0x7f605c060280 2024-02-10T09:09:58.441 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:58.440+0000 7f60609bc700 1 -- 172.21.15.155:0/403052041 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f605c0fc090 con 0x7f605c060280 2024-02-10T09:09:58.442 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:58.441+0000 7f604affd700 1 -- 172.21.15.155:0/403052041 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f604c004490 con 0x7f605c060280 2024-02-10T09:09:58.442 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:09:58.443 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:09:58.444 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:58.443+0000 7f60609bc700 1 -- 172.21.15.155:0/403052041 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f604402ff60 msgr2=0x7f6044032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:09:58.444 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:58.443+0000 7f60609bc700 1 --2- 172.21.15.155:0/403052041 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f604402ff60 0x7f6044032420 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f60500096a0 tx=0x7f6050006b40).stop 2024-02-10T09:09:58.444 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:58.443+0000 7f60609bc700 1 -- 172.21.15.155:0/403052041 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f605c060280 msgr2=0x7f605c106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:09:58.445 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:58.443+0000 7f60609bc700 1 --2- 172.21.15.155:0/403052041 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f605c060280 0x7f605c106780 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f604c009830 tx=0x7f604c00b040).stop 2024-02-10T09:09:58.445 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:58.443+0000 7f60609bc700 1 -- 172.21.15.155:0/403052041 shutdown_connections 2024-02-10T09:09:58.445 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:58.443+0000 7f60609bc700 1 --2- 172.21.15.155:0/403052041 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f604402ff60 0x7f6044032420 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:09:58.445 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:58.443+0000 7f60609bc700 1 --2- 172.21.15.155:0/403052041 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f605c060280 0x7f605c106780 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:09:58.445 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:58.443+0000 7f60609bc700 1 -- 172.21.15.155:0/403052041 >> 172.21.15.155:0/403052041 conn(0x7f605c0f0b70 msgr2=0x7f605c0f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:09:58.445 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:58.443+0000 7f60609bc700 1 -- 172.21.15.155:0/403052041 shutdown_connections 2024-02-10T09:09:58.445 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:09:58.443+0000 7f60609bc700 1 -- 172.21.15.155:0/403052041 wait complete. 2024-02-10T09:09:58.445 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:09:59.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:09:58 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:09:57.740517+0000 mgr.smithi077.dxntvw (mgr.14162) 346 : cluster [DBG] pgmap v330: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:09:59.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:09:58 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:09:58.442524+0000 mon.smithi077 (mon.0) 312 : audit [DBG] from='client.? 172.21.15.155:0/403052041' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:10:00.151 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:10:00.151 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:10:01.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:10:00 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:09:59.740896+0000 mgr.smithi077.dxntvw (mgr.14162) 347 : cluster [DBG] pgmap v331: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:10:02.584 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:02.555+0000 7f1f1ceb5700 1 -- 172.21.15.155:0/4275154671 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f180f62a0 msgr2=0x7f1f180f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:10:02.585 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:02.555+0000 7f1f1ceb5700 1 --2- 172.21.15.155:0/4275154671 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f180f62a0 0x7f1f180f66c0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f1f08009fb0 tx=0x7f1f08009b30).stop 2024-02-10T09:10:02.585 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:02.555+0000 7f1f1ceb5700 1 -- 172.21.15.155:0/4275154671 shutdown_connections 2024-02-10T09:10:02.585 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:02.555+0000 7f1f1ceb5700 1 --2- 172.21.15.155:0/4275154671 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f180f62a0 0x7f1f180f66c0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:10:02.585 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:02.555+0000 7f1f1ceb5700 1 -- 172.21.15.155:0/4275154671 >> 172.21.15.155:0/4275154671 conn(0x7f1f180f1aa0 msgr2=0x7f1f180f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:10:02.585 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:02.555+0000 7f1f1ceb5700 1 -- 172.21.15.155:0/4275154671 shutdown_connections 2024-02-10T09:10:02.585 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:02.555+0000 7f1f1ceb5700 1 -- 172.21.15.155:0/4275154671 wait complete. 2024-02-10T09:10:02.585 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:02.555+0000 7f1f1ceb5700 1 Processor -- start 2024-02-10T09:10:02.586 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:02.555+0000 7f1f1ceb5700 1 -- start start 2024-02-10T09:10:02.586 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:02.556+0000 7f1f1ceb5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f180f62a0 0x7f1f181065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:10:02.586 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:02.556+0000 7f1f1ceb5700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f18106ac0 con 0x7f1f180f62a0 2024-02-10T09:10:02.586 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:02.556+0000 7f1f177fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f180f62a0 0x7f1f181065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:10:02.586 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:02.556+0000 7f1f177fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f180f62a0 0x7f1f181065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:40974/0 (socket says 172.21.15.155:40974) 2024-02-10T09:10:02.586 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:02.556+0000 7f1f177fe700 1 -- 172.21.15.155:0/1127152147 learned_addr learned my addr 172.21.15.155:0/1127152147 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:10:02.586 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:02.556+0000 7f1f177fe700 1 -- 172.21.15.155:0/1127152147 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f08005040 con 0x7f1f180f62a0 2024-02-10T09:10:02.586 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:02.556+0000 7f1f177fe700 1 --2- 172.21.15.155:0/1127152147 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f180f62a0 0x7f1f181065d0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f1f080089f0 tx=0x7f1f0800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:10:02.586 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:02.557+0000 7f1efffff700 1 -- 172.21.15.155:0/1127152147 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f08009320 con 0x7f1f180f62a0 2024-02-10T09:10:02.586 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:02.557+0000 7f1f1ceb5700 1 -- 172.21.15.155:0/1127152147 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f18106cc0 con 0x7f1f180f62a0 2024-02-10T09:10:02.586 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:02.557+0000 7f1efffff700 1 -- 172.21.15.155:0/1127152147 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1f08013380 con 0x7f1f180f62a0 2024-02-10T09:10:02.586 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:02.557+0000 7f1efffff700 1 -- 172.21.15.155:0/1127152147 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f08014e20 con 0x7f1f180f62a0 2024-02-10T09:10:02.586 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:02.557+0000 7f1f1ceb5700 1 -- 172.21.15.155:0/1127152147 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f18107150 con 0x7f1f180f62a0 2024-02-10T09:10:02.587 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:02.557+0000 7f1efffff700 1 -- 172.21.15.155:0/1127152147 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1f080095f0 con 0x7f1f180f62a0 2024-02-10T09:10:02.587 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:02.558+0000 7f1f1ceb5700 1 -- 172.21.15.155:0/1127152147 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f18045ad0 con 0x7f1f180f62a0 2024-02-10T09:10:02.587 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:02.558+0000 7f1efffff700 1 --2- 172.21.15.155:0/1127152147 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f1f0002ff10 0x7f1f000323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:10:02.587 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:02.558+0000 7f1efffff700 1 -- 172.21.15.155:0/1127152147 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1f08040fb0 con 0x7f1f180f62a0 2024-02-10T09:10:02.587 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:02.558+0000 7f1f16ffd700 1 --2- 172.21.15.155:0/1127152147 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f1f0002ff10 0x7f1f000323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:10:02.587 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:02.558+0000 7f1f16ffd700 1 --2- 172.21.15.155:0/1127152147 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f1f0002ff10 0x7f1f000323d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f1f0c0096a0 tx=0x7f1f0c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:10:02.587 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:02.562+0000 7f1efffff700 1 -- 172.21.15.155:0/1127152147 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1f0806d2b0 con 0x7f1f180f62a0 2024-02-10T09:10:02.773 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:02.772+0000 7f1f1ceb5700 1 -- 172.21.15.155:0/1127152147 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1f18108000 con 0x7f1f180f62a0 2024-02-10T09:10:02.773 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:02.772+0000 7f1efffff700 1 -- 172.21.15.155:0/1127152147 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1f0801d980 con 0x7f1f180f62a0 2024-02-10T09:10:02.774 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:10:02.774 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:10:02.775 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:02.774+0000 7f1f1ceb5700 1 -- 172.21.15.155:0/1127152147 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f1f0002ff10 msgr2=0x7f1f000323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:10:02.775 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:02.774+0000 7f1f1ceb5700 1 --2- 172.21.15.155:0/1127152147 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f1f0002ff10 0x7f1f000323d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f1f0c0096a0 tx=0x7f1f0c006b40).stop 2024-02-10T09:10:02.775 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:02.774+0000 7f1f1ceb5700 1 -- 172.21.15.155:0/1127152147 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f180f62a0 msgr2=0x7f1f181065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:10:02.775 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:02.774+0000 7f1f1ceb5700 1 --2- 172.21.15.155:0/1127152147 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f180f62a0 0x7f1f181065d0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f1f080089f0 tx=0x7f1f0800b040).stop 2024-02-10T09:10:02.776 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:02.774+0000 7f1f1ceb5700 1 -- 172.21.15.155:0/1127152147 shutdown_connections 2024-02-10T09:10:02.776 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:02.774+0000 7f1f1ceb5700 1 --2- 172.21.15.155:0/1127152147 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f1f0002ff10 0x7f1f000323d0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:10:02.776 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:02.774+0000 7f1f1ceb5700 1 --2- 172.21.15.155:0/1127152147 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1f180f62a0 0x7f1f181065d0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:10:02.776 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:02.774+0000 7f1f1ceb5700 1 -- 172.21.15.155:0/1127152147 >> 172.21.15.155:0/1127152147 conn(0x7f1f180f1aa0 msgr2=0x7f1f180f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:10:02.776 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:02.774+0000 7f1f1ceb5700 1 -- 172.21.15.155:0/1127152147 shutdown_connections 2024-02-10T09:10:02.776 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:02.774+0000 7f1f1ceb5700 1 -- 172.21.15.155:0/1127152147 wait complete. 2024-02-10T09:10:02.776 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:10:03.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:10:02 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:10:01.741261+0000 mgr.smithi077.dxntvw (mgr.14162) 348 : cluster [DBG] pgmap v332: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:10:03.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:10:02 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:10:02.773878+0000 mon.smithi077 (mon.0) 313 : audit [DBG] from='client.? 172.21.15.155:0/1127152147' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:10:04.571 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:10:04.571 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:10:05.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:10:04 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:10:03.741578+0000 mgr.smithi077.dxntvw (mgr.14162) 349 : cluster [DBG] pgmap v333: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:10:06.738 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:06.734+0000 7f3e4eb8d700 1 -- 172.21.15.155:0/4181897645 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e480f51a0 msgr2=0x7f3e480f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:10:06.738 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:06.734+0000 7f3e4eb8d700 1 --2- 172.21.15.155:0/4181897645 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e480f51a0 0x7f3e480f55c0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7f3e38009fb0 tx=0x7f3e38009b30).stop 2024-02-10T09:10:06.739 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:06.734+0000 7f3e4eb8d700 1 -- 172.21.15.155:0/4181897645 shutdown_connections 2024-02-10T09:10:06.739 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:06.734+0000 7f3e4eb8d700 1 --2- 172.21.15.155:0/4181897645 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e480f51a0 0x7f3e480f55c0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:10:06.739 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:06.734+0000 7f3e4eb8d700 1 -- 172.21.15.155:0/4181897645 >> 172.21.15.155:0/4181897645 conn(0x7f3e480f09a0 msgr2=0x7f3e480f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:10:06.739 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:06.735+0000 7f3e4eb8d700 1 -- 172.21.15.155:0/4181897645 shutdown_connections 2024-02-10T09:10:06.739 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:06.735+0000 7f3e4eb8d700 1 -- 172.21.15.155:0/4181897645 wait complete. 2024-02-10T09:10:06.739 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:06.735+0000 7f3e4eb8d700 1 Processor -- start 2024-02-10T09:10:06.739 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:06.735+0000 7f3e4eb8d700 1 -- start start 2024-02-10T09:10:06.739 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:06.735+0000 7f3e4eb8d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e480f51a0 0x7f3e4810b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:10:06.739 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:06.735+0000 7f3e4eb8d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e481070c0 con 0x7f3e480f51a0 2024-02-10T09:10:06.739 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:06.735+0000 7f3e4db8b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e480f51a0 0x7f3e4810b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:10:06.739 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:06.735+0000 7f3e4db8b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e480f51a0 0x7f3e4810b6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:41004/0 (socket says 172.21.15.155:41004) 2024-02-10T09:10:06.739 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:06.736+0000 7f3e4db8b700 1 -- 172.21.15.155:0/2416385416 learned_addr learned my addr 172.21.15.155:0/2416385416 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:10:06.740 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:06.736+0000 7f3e4db8b700 1 -- 172.21.15.155:0/2416385416 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e38005040 con 0x7f3e480f51a0 2024-02-10T09:10:06.740 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:06.736+0000 7f3e4db8b700 1 --2- 172.21.15.155:0/2416385416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e480f51a0 0x7f3e4810b6d0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f3e380089f0 tx=0x7f3e3800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:10:06.740 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:06.736+0000 7f3e3e7fc700 1 -- 172.21.15.155:0/2416385416 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e38009320 con 0x7f3e480f51a0 2024-02-10T09:10:06.740 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:06.736+0000 7f3e4eb8d700 1 -- 172.21.15.155:0/2416385416 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e48060240 con 0x7f3e480f51a0 2024-02-10T09:10:06.740 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:06.736+0000 7f3e3e7fc700 1 -- 172.21.15.155:0/2416385416 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3e38013380 con 0x7f3e480f51a0 2024-02-10T09:10:06.740 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:06.736+0000 7f3e3e7fc700 1 -- 172.21.15.155:0/2416385416 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e38014e20 con 0x7f3e480f51a0 2024-02-10T09:10:06.740 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:06.736+0000 7f3e4eb8d700 1 -- 172.21.15.155:0/2416385416 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e4810bbc0 con 0x7f3e480f51a0 2024-02-10T09:10:06.740 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:06.737+0000 7f3e3e7fc700 1 -- 172.21.15.155:0/2416385416 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3e38014420 con 0x7f3e480f51a0 2024-02-10T09:10:06.740 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:06.737+0000 7f3e4eb8d700 1 -- 172.21.15.155:0/2416385416 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e48045ad0 con 0x7f3e480f51a0 2024-02-10T09:10:06.740 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:06.737+0000 7f3e3e7fc700 1 --2- 172.21.15.155:0/2416385416 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f3e3402ff10 0x7f3e340323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:10:06.740 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:06.737+0000 7f3e3e7fc700 1 -- 172.21.15.155:0/2416385416 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3e38040f50 con 0x7f3e480f51a0 2024-02-10T09:10:06.741 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:06.738+0000 7f3e4d38a700 1 --2- 172.21.15.155:0/2416385416 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f3e3402ff10 0x7f3e340323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:10:06.741 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:06.738+0000 7f3e4d38a700 1 --2- 172.21.15.155:0/2416385416 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f3e3402ff10 0x7f3e340323d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f3e440096a0 tx=0x7f3e44006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:10:06.743 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:06.742+0000 7f3e3e7fc700 1 -- 172.21.15.155:0/2416385416 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3e3806d2b0 con 0x7f3e480f51a0 2024-02-10T09:10:06.949 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:06.948+0000 7f3e4eb8d700 1 -- 172.21.15.155:0/2416385416 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3e4810c3e0 con 0x7f3e480f51a0 2024-02-10T09:10:06.950 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:06.949+0000 7f3e3e7fc700 1 -- 172.21.15.155:0/2416385416 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3e380095f0 con 0x7f3e480f51a0 2024-02-10T09:10:06.950 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:10:06.950 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:10:06.952 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:06.951+0000 7f3e4eb8d700 1 -- 172.21.15.155:0/2416385416 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f3e3402ff10 msgr2=0x7f3e340323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:10:06.952 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:06.951+0000 7f3e4eb8d700 1 --2- 172.21.15.155:0/2416385416 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f3e3402ff10 0x7f3e340323d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f3e440096a0 tx=0x7f3e44006b40).stop 2024-02-10T09:10:06.952 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:06.951+0000 7f3e4eb8d700 1 -- 172.21.15.155:0/2416385416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e480f51a0 msgr2=0x7f3e4810b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:10:06.952 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:06.951+0000 7f3e4eb8d700 1 --2- 172.21.15.155:0/2416385416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e480f51a0 0x7f3e4810b6d0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f3e380089f0 tx=0x7f3e3800b040).stop 2024-02-10T09:10:06.952 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:06.951+0000 7f3e4eb8d700 1 -- 172.21.15.155:0/2416385416 shutdown_connections 2024-02-10T09:10:06.952 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:06.951+0000 7f3e4eb8d700 1 --2- 172.21.15.155:0/2416385416 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f3e3402ff10 0x7f3e340323d0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:10:06.952 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:06.951+0000 7f3e4eb8d700 1 --2- 172.21.15.155:0/2416385416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e480f51a0 0x7f3e4810b6d0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:10:06.952 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:06.951+0000 7f3e4eb8d700 1 -- 172.21.15.155:0/2416385416 >> 172.21.15.155:0/2416385416 conn(0x7f3e480f09a0 msgr2=0x7f3e480f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:10:06.952 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:06.951+0000 7f3e4eb8d700 1 -- 172.21.15.155:0/2416385416 shutdown_connections 2024-02-10T09:10:06.953 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:06.951+0000 7f3e4eb8d700 1 -- 172.21.15.155:0/2416385416 wait complete. 2024-02-10T09:10:06.953 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:10:07.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:10:06 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:10:05.741897+0000 mgr.smithi077.dxntvw (mgr.14162) 350 : cluster [DBG] pgmap v334: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:10:08.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:10:07 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:10:06.950633+0000 mon.smithi077 (mon.0) 314 : audit [DBG] from='client.? 172.21.15.155:0/2416385416' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:10:08.753 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:10:08.754 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:10:09.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:10:08 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:10:07.742304+0000 mgr.smithi077.dxntvw (mgr.14162) 351 : cluster [DBG] pgmap v335: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:10:11.034 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:11.031+0000 7fe9906da700 1 -- 172.21.15.155:0/2413271470 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9880f6510 msgr2=0x7fe9880f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:10:11.034 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:11.031+0000 7fe9906da700 1 --2- 172.21.15.155:0/2413271470 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9880f6510 0x7fe9880f6930 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7fe97c009fb0 tx=0x7fe97c009b30).stop 2024-02-10T09:10:11.034 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:11.031+0000 7fe9906da700 1 -- 172.21.15.155:0/2413271470 shutdown_connections 2024-02-10T09:10:11.034 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:11.031+0000 7fe9906da700 1 --2- 172.21.15.155:0/2413271470 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9880f6510 0x7fe9880f6930 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:10:11.034 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:11.031+0000 7fe9906da700 1 -- 172.21.15.155:0/2413271470 >> 172.21.15.155:0/2413271470 conn(0x7fe9880f1c70 msgr2=0x7fe9880f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:10:11.034 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:11.031+0000 7fe9906da700 1 -- 172.21.15.155:0/2413271470 shutdown_connections 2024-02-10T09:10:11.035 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:11.031+0000 7fe9906da700 1 -- 172.21.15.155:0/2413271470 wait complete. 2024-02-10T09:10:11.035 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:11.032+0000 7fe9906da700 1 Processor -- start 2024-02-10T09:10:11.035 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:11.032+0000 7fe9906da700 1 -- start start 2024-02-10T09:10:11.035 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:11.032+0000 7fe9906da700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9880f6510 0x7fe9881067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:10:11.035 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:11.032+0000 7fe9906da700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe988106cd0 con 0x7fe9880f6510 2024-02-10T09:10:11.035 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:11.032+0000 7fe98e476700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9880f6510 0x7fe9881067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:10:11.035 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:11.032+0000 7fe98e476700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9880f6510 0x7fe9881067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:41020/0 (socket says 172.21.15.155:41020) 2024-02-10T09:10:11.035 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:11.032+0000 7fe98e476700 1 -- 172.21.15.155:0/2265502806 learned_addr learned my addr 172.21.15.155:0/2265502806 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:10:11.035 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:11.032+0000 7fe98e476700 1 -- 172.21.15.155:0/2265502806 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe97c005040 con 0x7fe9880f6510 2024-02-10T09:10:11.035 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:11.033+0000 7fe98e476700 1 --2- 172.21.15.155:0/2265502806 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9880f6510 0x7fe9881067e0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7fe97c0089f0 tx=0x7fe97c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:10:11.035 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:11.033+0000 7fe97affd700 1 -- 172.21.15.155:0/2265502806 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe97c009320 con 0x7fe9880f6510 2024-02-10T09:10:11.035 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:11.033+0000 7fe97affd700 1 -- 172.21.15.155:0/2265502806 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe97c013380 con 0x7fe9880f6510 2024-02-10T09:10:11.035 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:11.033+0000 7fe9906da700 1 -- 172.21.15.155:0/2265502806 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe988106ed0 con 0x7fe9880f6510 2024-02-10T09:10:11.035 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:11.033+0000 7fe97affd700 1 -- 172.21.15.155:0/2265502806 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe97c01ed80 con 0x7fe9880f6510 2024-02-10T09:10:11.035 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:11.033+0000 7fe9906da700 1 -- 172.21.15.155:0/2265502806 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe988107360 con 0x7fe9880f6510 2024-02-10T09:10:11.036 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:11.034+0000 7fe97affd700 1 -- 172.21.15.155:0/2265502806 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe97c01e420 con 0x7fe9880f6510 2024-02-10T09:10:11.036 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:11.034+0000 7fe97affd700 1 --2- 172.21.15.155:0/2265502806 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fe97402ffb0 0x7fe974032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:10:11.037 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:11.034+0000 7fe97affd700 1 -- 172.21.15.155:0/2265502806 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe97c041180 con 0x7fe9880f6510 2024-02-10T09:10:11.037 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:11.034+0000 7fe9906da700 1 -- 172.21.15.155:0/2265502806 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe988045ad0 con 0x7fe9880f6510 2024-02-10T09:10:11.037 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:11.034+0000 7fe98dc75700 1 --2- 172.21.15.155:0/2265502806 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fe97402ffb0 0x7fe974032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:10:11.037 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:11.035+0000 7fe98dc75700 1 --2- 172.21.15.155:0/2265502806 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fe97402ffb0 0x7fe974032470 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fe9840096a0 tx=0x7fe984006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:10:11.040 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:11.039+0000 7fe97affd700 1 -- 172.21.15.155:0/2265502806 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe97c01cb10 con 0x7fe9880f6510 2024-02-10T09:10:11.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:10:10 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:10:09.742650+0000 mgr.smithi077.dxntvw (mgr.14162) 352 : cluster [DBG] pgmap v336: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:10:11.244 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:11.243+0000 7fe9906da700 1 -- 172.21.15.155:0/2265502806 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe988108080 con 0x7fe9880f6510 2024-02-10T09:10:11.244 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:11.243+0000 7fe97affd700 1 -- 172.21.15.155:0/2265502806 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe97c01cb10 con 0x7fe9880f6510 2024-02-10T09:10:11.245 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:10:11.245 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:10:11.246 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:11.245+0000 7fe9906da700 1 -- 172.21.15.155:0/2265502806 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fe97402ffb0 msgr2=0x7fe974032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:10:11.246 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:11.245+0000 7fe9906da700 1 --2- 172.21.15.155:0/2265502806 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fe97402ffb0 0x7fe974032470 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fe9840096a0 tx=0x7fe984006b40).stop 2024-02-10T09:10:11.246 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:11.245+0000 7fe9906da700 1 -- 172.21.15.155:0/2265502806 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9880f6510 msgr2=0x7fe9881067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:10:11.246 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:11.245+0000 7fe9906da700 1 --2- 172.21.15.155:0/2265502806 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9880f6510 0x7fe9881067e0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7fe97c0089f0 tx=0x7fe97c00b040).stop 2024-02-10T09:10:11.246 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:11.245+0000 7fe9906da700 1 -- 172.21.15.155:0/2265502806 shutdown_connections 2024-02-10T09:10:11.246 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:11.245+0000 7fe9906da700 1 --2- 172.21.15.155:0/2265502806 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fe97402ffb0 0x7fe974032470 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:10:11.246 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:11.245+0000 7fe9906da700 1 --2- 172.21.15.155:0/2265502806 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9880f6510 0x7fe9881067e0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:10:11.247 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:11.245+0000 7fe9906da700 1 -- 172.21.15.155:0/2265502806 >> 172.21.15.155:0/2265502806 conn(0x7fe9880f1c70 msgr2=0x7fe9880f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:10:11.247 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:11.245+0000 7fe9906da700 1 -- 172.21.15.155:0/2265502806 shutdown_connections 2024-02-10T09:10:11.247 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:11.245+0000 7fe9906da700 1 -- 172.21.15.155:0/2265502806 wait complete. 2024-02-10T09:10:11.247 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:10:12.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:10:11 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:10:11.245127+0000 mon.smithi077 (mon.0) 315 : audit [DBG] from='client.? 172.21.15.155:0/2265502806' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:10:12.920 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:10:12.920 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:10:13.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:10:12 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:10:11.743059+0000 mgr.smithi077.dxntvw (mgr.14162) 353 : cluster [DBG] pgmap v337: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:10:15.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:10:14 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:10:13.743452+0000 mgr.smithi077.dxntvw (mgr.14162) 354 : cluster [DBG] pgmap v338: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:10:15.221 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:15.218+0000 7f71c7ca7700 1 -- 172.21.15.155:0/2838874208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71c00f5410 msgr2=0x7f71c00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:10:15.221 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:15.218+0000 7f71c7ca7700 1 --2- 172.21.15.155:0/2838874208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71c00f5410 0x7f71c00f5830 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7f71b0009fb0 tx=0x7f71b0009b30).stop 2024-02-10T09:10:15.221 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:15.218+0000 7f71c7ca7700 1 -- 172.21.15.155:0/2838874208 shutdown_connections 2024-02-10T09:10:15.221 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:15.218+0000 7f71c7ca7700 1 --2- 172.21.15.155:0/2838874208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71c00f5410 0x7f71c00f5830 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:10:15.221 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:15.218+0000 7f71c7ca7700 1 -- 172.21.15.155:0/2838874208 >> 172.21.15.155:0/2838874208 conn(0x7f71c00f0b70 msgr2=0x7f71c00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:10:15.221 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:15.218+0000 7f71c7ca7700 1 -- 172.21.15.155:0/2838874208 shutdown_connections 2024-02-10T09:10:15.221 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:15.218+0000 7f71c7ca7700 1 -- 172.21.15.155:0/2838874208 wait complete. 2024-02-10T09:10:15.222 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:15.218+0000 7f71c7ca7700 1 Processor -- start 2024-02-10T09:10:15.222 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:15.219+0000 7f71c7ca7700 1 -- start start 2024-02-10T09:10:15.222 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:15.219+0000 7f71c7ca7700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71c00f5410 0x7f71c00fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:10:15.222 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:15.219+0000 7f71c7ca7700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71c00fb960 con 0x7f71c00f5410 2024-02-10T09:10:15.222 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:15.219+0000 7f71c5a43700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71c00f5410 0x7f71c00fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:10:15.222 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:15.219+0000 7f71c5a43700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71c00f5410 0x7f71c00fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:43388/0 (socket says 172.21.15.155:43388) 2024-02-10T09:10:15.222 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:15.219+0000 7f71c5a43700 1 -- 172.21.15.155:0/586112597 learned_addr learned my addr 172.21.15.155:0/586112597 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:10:15.222 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:15.219+0000 7f71c5a43700 1 -- 172.21.15.155:0/586112597 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71b0005040 con 0x7f71c00f5410 2024-02-10T09:10:15.222 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:15.219+0000 7f71c5a43700 1 --2- 172.21.15.155:0/586112597 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71c00f5410 0x7f71c00fd4e0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f71b0005010 tx=0x7f71b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:10:15.222 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:15.219+0000 7f71b67fc700 1 -- 172.21.15.155:0/586112597 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f71b0009320 con 0x7f71c00f5410 2024-02-10T09:10:15.222 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:15.220+0000 7f71b67fc700 1 -- 172.21.15.155:0/586112597 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f71b0013380 con 0x7f71c00f5410 2024-02-10T09:10:15.222 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:15.220+0000 7f71c7ca7700 1 -- 172.21.15.155:0/586112597 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71c00fbb00 con 0x7f71c00f5410 2024-02-10T09:10:15.222 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:15.220+0000 7f71b67fc700 1 -- 172.21.15.155:0/586112597 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f71b0014e50 con 0x7f71c00f5410 2024-02-10T09:10:15.222 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:15.220+0000 7f71c7ca7700 1 -- 172.21.15.155:0/586112597 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71c00fbf90 con 0x7f71c00f5410 2024-02-10T09:10:15.223 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:15.220+0000 7f71b67fc700 1 -- 172.21.15.155:0/586112597 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f71b0014420 con 0x7f71c00f5410 2024-02-10T09:10:15.223 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:15.220+0000 7f71b67fc700 1 --2- 172.21.15.155:0/586112597 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f71ac02ff10 0x7f71ac0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:10:15.224 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:15.220+0000 7f71b67fc700 1 -- 172.21.15.155:0/586112597 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f71b0040e60 con 0x7f71c00f5410 2024-02-10T09:10:15.224 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:15.221+0000 7f71c7ca7700 1 -- 172.21.15.155:0/586112597 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71c0059100 con 0x7f71c00f5410 2024-02-10T09:10:15.224 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:15.221+0000 7f71c5242700 1 --2- 172.21.15.155:0/586112597 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f71ac02ff10 0x7f71ac0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:10:15.224 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:15.221+0000 7f71c5242700 1 --2- 172.21.15.155:0/586112597 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f71ac02ff10 0x7f71ac0323d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f71bc0096a0 tx=0x7f71bc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:10:15.226 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:15.225+0000 7f71b67fc700 1 -- 172.21.15.155:0/586112597 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f71b00095f0 con 0x7f71c00f5410 2024-02-10T09:10:15.429 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:15.428+0000 7f71c7ca7700 1 -- 172.21.15.155:0/586112597 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f71c00fc5f0 con 0x7f71c00f5410 2024-02-10T09:10:15.429 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:15.429+0000 7f71b67fc700 1 -- 172.21.15.155:0/586112597 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f71b001db70 con 0x7f71c00f5410 2024-02-10T09:10:15.430 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:10:15.430 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:10:15.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:15.430+0000 7f71c7ca7700 1 -- 172.21.15.155:0/586112597 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f71ac02ff10 msgr2=0x7f71ac0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:10:15.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:15.430+0000 7f71c7ca7700 1 --2- 172.21.15.155:0/586112597 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f71ac02ff10 0x7f71ac0323d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f71bc0096a0 tx=0x7f71bc006b40).stop 2024-02-10T09:10:15.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:15.430+0000 7f71c7ca7700 1 -- 172.21.15.155:0/586112597 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71c00f5410 msgr2=0x7f71c00fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:10:15.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:15.430+0000 7f71c7ca7700 1 --2- 172.21.15.155:0/586112597 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71c00f5410 0x7f71c00fd4e0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f71b0005010 tx=0x7f71b000b040).stop 2024-02-10T09:10:15.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:15.430+0000 7f71c7ca7700 1 -- 172.21.15.155:0/586112597 shutdown_connections 2024-02-10T09:10:15.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:15.430+0000 7f71c7ca7700 1 --2- 172.21.15.155:0/586112597 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f71ac02ff10 0x7f71ac0323d0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:10:15.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:15.430+0000 7f71c7ca7700 1 --2- 172.21.15.155:0/586112597 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f71c00f5410 0x7f71c00fd4e0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:10:15.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:15.430+0000 7f71c7ca7700 1 -- 172.21.15.155:0/586112597 >> 172.21.15.155:0/586112597 conn(0x7f71c00f0b70 msgr2=0x7f71c005fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:10:15.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:15.430+0000 7f71c7ca7700 1 -- 172.21.15.155:0/586112597 shutdown_connections 2024-02-10T09:10:15.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:15.430+0000 7f71c7ca7700 1 -- 172.21.15.155:0/586112597 wait complete. 2024-02-10T09:10:15.432 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:10:16.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:10:15 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:10:15.430127+0000 mon.smithi077 (mon.0) 316 : audit [DBG] from='client.? 172.21.15.155:0/586112597' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:10:17.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:10:16 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:10:15.743842+0000 mgr.smithi077.dxntvw (mgr.14162) 355 : cluster [DBG] pgmap v339: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:10:17.162 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:10:17.163 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:10:19.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:10:18 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:10:17.744248+0000 mgr.smithi077.dxntvw (mgr.14162) 356 : cluster [DBG] pgmap v340: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:10:19.589 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:19.556+0000 7fb688246700 1 -- 172.21.15.155:0/2499217007 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6800f6510 msgr2=0x7fb6800f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:10:19.589 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:19.556+0000 7fb688246700 1 --2- 172.21.15.155:0/2499217007 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6800f6510 0x7fb6800f6930 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7fb670009fb0 tx=0x7fb670009b30).stop 2024-02-10T09:10:19.589 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:19.556+0000 7fb688246700 1 -- 172.21.15.155:0/2499217007 shutdown_connections 2024-02-10T09:10:19.589 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:19.556+0000 7fb688246700 1 --2- 172.21.15.155:0/2499217007 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6800f6510 0x7fb6800f6930 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:10:19.589 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:19.556+0000 7fb688246700 1 -- 172.21.15.155:0/2499217007 >> 172.21.15.155:0/2499217007 conn(0x7fb6800f1c70 msgr2=0x7fb6800f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:10:19.589 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:19.556+0000 7fb688246700 1 -- 172.21.15.155:0/2499217007 shutdown_connections 2024-02-10T09:10:19.589 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:19.556+0000 7fb688246700 1 -- 172.21.15.155:0/2499217007 wait complete. 2024-02-10T09:10:19.589 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:19.557+0000 7fb688246700 1 Processor -- start 2024-02-10T09:10:19.589 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:19.557+0000 7fb688246700 1 -- start start 2024-02-10T09:10:19.590 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:19.557+0000 7fb688246700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6800f6510 0x7fb6801067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:10:19.590 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:19.557+0000 7fb688246700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb680106cd0 con 0x7fb6800f6510 2024-02-10T09:10:19.590 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:19.557+0000 7fb685fe2700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6800f6510 0x7fb6801067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:10:19.590 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:19.557+0000 7fb685fe2700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6800f6510 0x7fb6801067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:43412/0 (socket says 172.21.15.155:43412) 2024-02-10T09:10:19.590 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:19.557+0000 7fb685fe2700 1 -- 172.21.15.155:0/3168530046 learned_addr learned my addr 172.21.15.155:0/3168530046 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:10:19.590 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:19.557+0000 7fb685fe2700 1 -- 172.21.15.155:0/3168530046 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb670005040 con 0x7fb6800f6510 2024-02-10T09:10:19.590 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:19.558+0000 7fb685fe2700 1 --2- 172.21.15.155:0/3168530046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6800f6510 0x7fb6801067e0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7fb670005010 tx=0x7fb67000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:10:19.590 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:19.558+0000 7fb6767fc700 1 -- 172.21.15.155:0/3168530046 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb670009320 con 0x7fb6800f6510 2024-02-10T09:10:19.590 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:19.558+0000 7fb688246700 1 -- 172.21.15.155:0/3168530046 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb680106ed0 con 0x7fb6800f6510 2024-02-10T09:10:19.590 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:19.558+0000 7fb688246700 1 -- 172.21.15.155:0/3168530046 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb680107360 con 0x7fb6800f6510 2024-02-10T09:10:19.590 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:19.558+0000 7fb6767fc700 1 -- 172.21.15.155:0/3168530046 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb670013380 con 0x7fb6800f6510 2024-02-10T09:10:19.590 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:19.558+0000 7fb6767fc700 1 -- 172.21.15.155:0/3168530046 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb670014df0 con 0x7fb6800f6510 2024-02-10T09:10:19.590 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:19.559+0000 7fb688246700 1 -- 172.21.15.155:0/3168530046 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb680045ad0 con 0x7fb6800f6510 2024-02-10T09:10:19.591 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:19.559+0000 7fb6767fc700 1 -- 172.21.15.155:0/3168530046 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb6700095f0 con 0x7fb6800f6510 2024-02-10T09:10:19.591 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:19.559+0000 7fb6767fc700 1 --2- 172.21.15.155:0/3168530046 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb66c02ff10 0x7fb66c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:10:19.591 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:19.559+0000 7fb6767fc700 1 -- 172.21.15.155:0/3168530046 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb670040f50 con 0x7fb6800f6510 2024-02-10T09:10:19.591 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:19.559+0000 7fb6857e1700 1 --2- 172.21.15.155:0/3168530046 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb66c02ff10 0x7fb66c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:10:19.591 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:19.560+0000 7fb6857e1700 1 --2- 172.21.15.155:0/3168530046 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb66c02ff10 0x7fb66c0323d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fb67c0096a0 tx=0x7fb67c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:10:19.591 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:19.564+0000 7fb6767fc700 1 -- 172.21.15.155:0/3168530046 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb670014420 con 0x7fb6800f6510 2024-02-10T09:10:19.769 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:19.768+0000 7fb688246700 1 -- 172.21.15.155:0/3168530046 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb680108210 con 0x7fb6800f6510 2024-02-10T09:10:19.770 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:19.770+0000 7fb6767fc700 1 -- 172.21.15.155:0/3168530046 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb67001d9d0 con 0x7fb6800f6510 2024-02-10T09:10:19.770 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:10:19.770 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:10:19.771 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:19.770+0000 7fb688246700 1 -- 172.21.15.155:0/3168530046 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb66c02ff10 msgr2=0x7fb66c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:10:19.772 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:19.770+0000 7fb688246700 1 --2- 172.21.15.155:0/3168530046 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb66c02ff10 0x7fb66c0323d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fb67c0096a0 tx=0x7fb67c006b40).stop 2024-02-10T09:10:19.772 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:19.770+0000 7fb688246700 1 -- 172.21.15.155:0/3168530046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6800f6510 msgr2=0x7fb6801067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:10:19.772 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:19.770+0000 7fb688246700 1 --2- 172.21.15.155:0/3168530046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6800f6510 0x7fb6801067e0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7fb670005010 tx=0x7fb67000b040).stop 2024-02-10T09:10:19.772 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:19.770+0000 7fb688246700 1 -- 172.21.15.155:0/3168530046 shutdown_connections 2024-02-10T09:10:19.772 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:19.770+0000 7fb688246700 1 --2- 172.21.15.155:0/3168530046 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb66c02ff10 0x7fb66c0323d0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:10:19.772 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:19.770+0000 7fb688246700 1 --2- 172.21.15.155:0/3168530046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6800f6510 0x7fb6801067e0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:10:19.772 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:19.770+0000 7fb688246700 1 -- 172.21.15.155:0/3168530046 >> 172.21.15.155:0/3168530046 conn(0x7fb6800f1c70 msgr2=0x7fb6800f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:10:19.772 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:19.770+0000 7fb688246700 1 -- 172.21.15.155:0/3168530046 shutdown_connections 2024-02-10T09:10:19.772 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:19.770+0000 7fb688246700 1 -- 172.21.15.155:0/3168530046 wait complete. 2024-02-10T09:10:19.773 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:10:20.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:10:19 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:10:19.770318+0000 mon.smithi077 (mon.0) 317 : audit [DBG] from='client.? 172.21.15.155:0/3168530046' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:10:21.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:10:20 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:10:19.744590+0000 mgr.smithi077.dxntvw (mgr.14162) 357 : cluster [DBG] pgmap v341: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:10:21.566 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:10:21.567 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:10:23.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:10:22 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:10:21.744911+0000 mgr.smithi077.dxntvw (mgr.14162) 358 : cluster [DBG] pgmap v342: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:10:23.828 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:23.825+0000 7fc967d67700 1 -- 172.21.15.155:0/4140380363 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9600f6510 msgr2=0x7fc9600f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:10:23.828 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:23.825+0000 7fc967d67700 1 --2- 172.21.15.155:0/4140380363 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9600f6510 0x7fc9600f6930 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7fc954009fb0 tx=0x7fc954009b30).stop 2024-02-10T09:10:23.828 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:23.826+0000 7fc967d67700 1 -- 172.21.15.155:0/4140380363 shutdown_connections 2024-02-10T09:10:23.828 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:23.826+0000 7fc967d67700 1 --2- 172.21.15.155:0/4140380363 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9600f6510 0x7fc9600f6930 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:10:23.828 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:23.826+0000 7fc967d67700 1 -- 172.21.15.155:0/4140380363 >> 172.21.15.155:0/4140380363 conn(0x7fc9600f1c70 msgr2=0x7fc9600f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:10:23.828 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:23.826+0000 7fc967d67700 1 -- 172.21.15.155:0/4140380363 shutdown_connections 2024-02-10T09:10:23.828 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:23.826+0000 7fc967d67700 1 -- 172.21.15.155:0/4140380363 wait complete. 2024-02-10T09:10:23.828 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:23.826+0000 7fc967d67700 1 Processor -- start 2024-02-10T09:10:23.828 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:23.826+0000 7fc967d67700 1 -- start start 2024-02-10T09:10:23.828 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:23.827+0000 7fc967d67700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9600f6510 0x7fc9601067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:10:23.829 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:23.827+0000 7fc967d67700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc960106cd0 con 0x7fc9600f6510 2024-02-10T09:10:23.829 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:23.827+0000 7fc965b03700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9600f6510 0x7fc9601067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:10:23.829 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:23.827+0000 7fc965b03700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9600f6510 0x7fc9601067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:54102/0 (socket says 172.21.15.155:54102) 2024-02-10T09:10:23.829 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:23.827+0000 7fc965b03700 1 -- 172.21.15.155:0/4139892489 learned_addr learned my addr 172.21.15.155:0/4139892489 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:10:23.829 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:23.827+0000 7fc965b03700 1 -- 172.21.15.155:0/4139892489 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc954005040 con 0x7fc9600f6510 2024-02-10T09:10:23.829 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:23.827+0000 7fc965b03700 1 --2- 172.21.15.155:0/4139892489 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9600f6510 0x7fc9601067e0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7fc9540089f0 tx=0x7fc95400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:10:23.829 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:23.828+0000 7fc9527fc700 1 -- 172.21.15.155:0/4139892489 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc954009320 con 0x7fc9600f6510 2024-02-10T09:10:23.829 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:23.828+0000 7fc967d67700 1 -- 172.21.15.155:0/4139892489 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc960106ed0 con 0x7fc9600f6510 2024-02-10T09:10:23.829 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:23.828+0000 7fc9527fc700 1 -- 172.21.15.155:0/4139892489 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc954013380 con 0x7fc9600f6510 2024-02-10T09:10:23.829 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:23.828+0000 7fc9527fc700 1 -- 172.21.15.155:0/4139892489 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc95401ed80 con 0x7fc9600f6510 2024-02-10T09:10:23.829 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:23.828+0000 7fc967d67700 1 -- 172.21.15.155:0/4139892489 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc960107360 con 0x7fc9600f6510 2024-02-10T09:10:23.830 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:23.828+0000 7fc9527fc700 1 -- 172.21.15.155:0/4139892489 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc95401e420 con 0x7fc9600f6510 2024-02-10T09:10:23.830 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:23.828+0000 7fc967d67700 1 -- 172.21.15.155:0/4139892489 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc960045ad0 con 0x7fc9600f6510 2024-02-10T09:10:23.830 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:23.829+0000 7fc9527fc700 1 --2- 172.21.15.155:0/4139892489 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fc94c02ff60 0x7fc94c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:10:23.830 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:23.829+0000 7fc9527fc700 1 -- 172.21.15.155:0/4139892489 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc954041110 con 0x7fc9600f6510 2024-02-10T09:10:23.831 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:23.829+0000 7fc965302700 1 --2- 172.21.15.155:0/4139892489 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fc94c02ff60 0x7fc94c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:10:23.831 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:23.829+0000 7fc965302700 1 --2- 172.21.15.155:0/4139892489 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fc94c02ff60 0x7fc94c032420 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7fc95c0096a0 tx=0x7fc95c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:10:23.833 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:23.833+0000 7fc9527fc700 1 -- 172.21.15.155:0/4139892489 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc95401cb10 con 0x7fc9600f6510 2024-02-10T09:10:24.038 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:24.038+0000 7fc967d67700 1 -- 172.21.15.155:0/4139892489 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc960108210 con 0x7fc9600f6510 2024-02-10T09:10:24.039 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:24.039+0000 7fc9527fc700 1 -- 172.21.15.155:0/4139892489 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc954004490 con 0x7fc9600f6510 2024-02-10T09:10:24.039 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:10:24.039 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:10:24.041 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:24.041+0000 7fc967d67700 1 -- 172.21.15.155:0/4139892489 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fc94c02ff60 msgr2=0x7fc94c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:10:24.041 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:24.041+0000 7fc967d67700 1 --2- 172.21.15.155:0/4139892489 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fc94c02ff60 0x7fc94c032420 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7fc95c0096a0 tx=0x7fc95c006b40).stop 2024-02-10T09:10:24.041 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:24.041+0000 7fc967d67700 1 -- 172.21.15.155:0/4139892489 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9600f6510 msgr2=0x7fc9601067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:10:24.041 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:24.041+0000 7fc967d67700 1 --2- 172.21.15.155:0/4139892489 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9600f6510 0x7fc9601067e0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7fc9540089f0 tx=0x7fc95400b040).stop 2024-02-10T09:10:24.041 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:24.041+0000 7fc967d67700 1 -- 172.21.15.155:0/4139892489 shutdown_connections 2024-02-10T09:10:24.041 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:24.041+0000 7fc967d67700 1 --2- 172.21.15.155:0/4139892489 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fc94c02ff60 0x7fc94c032420 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:10:24.041 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:24.041+0000 7fc967d67700 1 --2- 172.21.15.155:0/4139892489 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc9600f6510 0x7fc9601067e0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:10:24.041 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:24.041+0000 7fc967d67700 1 -- 172.21.15.155:0/4139892489 >> 172.21.15.155:0/4139892489 conn(0x7fc9600f1c70 msgr2=0x7fc9600f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:10:24.042 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:24.041+0000 7fc967d67700 1 -- 172.21.15.155:0/4139892489 shutdown_connections 2024-02-10T09:10:24.042 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:24.041+0000 7fc967d67700 1 -- 172.21.15.155:0/4139892489 wait complete. 2024-02-10T09:10:24.042 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:10:25.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:10:24 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:10:23.745302+0000 mgr.smithi077.dxntvw (mgr.14162) 359 : cluster [DBG] pgmap v343: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:10:25.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:10:24 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:10:24.039526+0000 mon.smithi077 (mon.0) 318 : audit [DBG] from='client.? 172.21.15.155:0/4139892489' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:10:25.898 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:10:25.899 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:10:27.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:10:26 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:10:25.745569+0000 mgr.smithi077.dxntvw (mgr.14162) 360 : cluster [DBG] pgmap v344: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:10:28.154 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:28.097+0000 7fb9f30c5700 1 -- 172.21.15.155:0/615694845 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9ec0f51a0 msgr2=0x7fb9ec0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:10:28.154 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:28.097+0000 7fb9f30c5700 1 --2- 172.21.15.155:0/615694845 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9ec0f51a0 0x7fb9ec0f55c0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7fb9e8009fb0 tx=0x7fb9e8009b30).stop 2024-02-10T09:10:28.154 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:28.097+0000 7fb9f30c5700 1 -- 172.21.15.155:0/615694845 shutdown_connections 2024-02-10T09:10:28.154 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:28.097+0000 7fb9f30c5700 1 --2- 172.21.15.155:0/615694845 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9ec0f51a0 0x7fb9ec0f55c0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:10:28.154 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:28.097+0000 7fb9f30c5700 1 -- 172.21.15.155:0/615694845 >> 172.21.15.155:0/615694845 conn(0x7fb9ec0f09a0 msgr2=0x7fb9ec0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:10:28.155 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:28.097+0000 7fb9f30c5700 1 -- 172.21.15.155:0/615694845 shutdown_connections 2024-02-10T09:10:28.155 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:28.097+0000 7fb9f30c5700 1 -- 172.21.15.155:0/615694845 wait complete. 2024-02-10T09:10:28.155 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:28.098+0000 7fb9f30c5700 1 Processor -- start 2024-02-10T09:10:28.155 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:28.098+0000 7fb9f30c5700 1 -- start start 2024-02-10T09:10:28.155 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:28.098+0000 7fb9f30c5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9ec0f51a0 0x7fb9ec1054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:10:28.155 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:28.098+0000 7fb9f30c5700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9ec1059c0 con 0x7fb9ec0f51a0 2024-02-10T09:10:28.156 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:28.098+0000 7fb9f20c3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9ec0f51a0 0x7fb9ec1054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:10:28.156 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:28.098+0000 7fb9f20c3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9ec0f51a0 0x7fb9ec1054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:54128/0 (socket says 172.21.15.155:54128) 2024-02-10T09:10:28.156 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:28.098+0000 7fb9f20c3700 1 -- 172.21.15.155:0/712731272 learned_addr learned my addr 172.21.15.155:0/712731272 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:10:28.156 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:28.099+0000 7fb9f20c3700 1 -- 172.21.15.155:0/712731272 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9e8005040 con 0x7fb9ec0f51a0 2024-02-10T09:10:28.156 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:28.099+0000 7fb9f20c3700 1 --2- 172.21.15.155:0/712731272 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9ec0f51a0 0x7fb9ec1054d0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7fb9e8009830 tx=0x7fb9e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:10:28.156 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:28.099+0000 7fb9e2ffd700 1 -- 172.21.15.155:0/712731272 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb9e8009320 con 0x7fb9ec0f51a0 2024-02-10T09:10:28.156 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:28.099+0000 7fb9f30c5700 1 -- 172.21.15.155:0/712731272 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9ec105bc0 con 0x7fb9ec0f51a0 2024-02-10T09:10:28.156 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:28.099+0000 7fb9e2ffd700 1 -- 172.21.15.155:0/712731272 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb9e8013380 con 0x7fb9ec0f51a0 2024-02-10T09:10:28.156 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:28.099+0000 7fb9e2ffd700 1 -- 172.21.15.155:0/712731272 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb9e8014420 con 0x7fb9ec0f51a0 2024-02-10T09:10:28.156 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:28.099+0000 7fb9f30c5700 1 -- 172.21.15.155:0/712731272 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9ec106050 con 0x7fb9ec0f51a0 2024-02-10T09:10:28.156 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:28.100+0000 7fb9e2ffd700 1 -- 172.21.15.155:0/712731272 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb9e801e420 con 0x7fb9ec0f51a0 2024-02-10T09:10:28.157 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:28.100+0000 7fb9f30c5700 1 -- 172.21.15.155:0/712731272 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9ec045ad0 con 0x7fb9ec0f51a0 2024-02-10T09:10:28.157 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:28.100+0000 7fb9e2ffd700 1 --2- 172.21.15.155:0/712731272 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb9d802ff60 0x7fb9d8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:10:28.157 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:28.100+0000 7fb9e2ffd700 1 -- 172.21.15.155:0/712731272 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb9e8040e20 con 0x7fb9ec0f51a0 2024-02-10T09:10:28.157 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:28.100+0000 7fb9f18c2700 1 --2- 172.21.15.155:0/712731272 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb9d802ff60 0x7fb9d8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:10:28.157 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:28.101+0000 7fb9f18c2700 1 --2- 172.21.15.155:0/712731272 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb9d802ff60 0x7fb9d8032420 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fb9dc0096a0 tx=0x7fb9dc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:10:28.157 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:28.105+0000 7fb9e2ffd700 1 -- 172.21.15.155:0/712731272 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb9e801c930 con 0x7fb9ec0f51a0 2024-02-10T09:10:28.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:28.311+0000 7fb9f30c5700 1 -- 172.21.15.155:0/712731272 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb9ec0fad30 con 0x7fb9ec0f51a0 2024-02-10T09:10:28.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:28.312+0000 7fb9e2ffd700 1 -- 172.21.15.155:0/712731272 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb9e802c2d0 con 0x7fb9ec0f51a0 2024-02-10T09:10:28.313 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:10:28.313 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:10:28.314 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:28.314+0000 7fb9f30c5700 1 -- 172.21.15.155:0/712731272 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb9d802ff60 msgr2=0x7fb9d8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:10:28.314 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:28.314+0000 7fb9f30c5700 1 --2- 172.21.15.155:0/712731272 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb9d802ff60 0x7fb9d8032420 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fb9dc0096a0 tx=0x7fb9dc006b40).stop 2024-02-10T09:10:28.314 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:28.314+0000 7fb9f30c5700 1 -- 172.21.15.155:0/712731272 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9ec0f51a0 msgr2=0x7fb9ec1054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:10:28.314 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:28.314+0000 7fb9f30c5700 1 --2- 172.21.15.155:0/712731272 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9ec0f51a0 0x7fb9ec1054d0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7fb9e8009830 tx=0x7fb9e800b040).stop 2024-02-10T09:10:28.315 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:28.314+0000 7fb9f30c5700 1 -- 172.21.15.155:0/712731272 shutdown_connections 2024-02-10T09:10:28.315 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:28.314+0000 7fb9f30c5700 1 --2- 172.21.15.155:0/712731272 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb9d802ff60 0x7fb9d8032420 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:10:28.315 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:28.314+0000 7fb9f30c5700 1 --2- 172.21.15.155:0/712731272 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9ec0f51a0 0x7fb9ec1054d0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:10:28.315 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:28.314+0000 7fb9f30c5700 1 -- 172.21.15.155:0/712731272 >> 172.21.15.155:0/712731272 conn(0x7fb9ec0f09a0 msgr2=0x7fb9ec0f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:10:28.315 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:28.314+0000 7fb9f30c5700 1 -- 172.21.15.155:0/712731272 shutdown_connections 2024-02-10T09:10:28.315 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:28.314+0000 7fb9f30c5700 1 -- 172.21.15.155:0/712731272 wait complete. 2024-02-10T09:10:28.316 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:10:29.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:10:28 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:10:27.745941+0000 mgr.smithi077.dxntvw (mgr.14162) 361 : cluster [DBG] pgmap v345: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:10:29.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:10:28 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:10:28.312692+0000 mon.smithi077 (mon.0) 319 : audit [DBG] from='client.? 172.21.15.155:0/712731272' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:10:30.242 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:10:30.242 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:10:31.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:10:30 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:10:29.746334+0000 mgr.smithi077.dxntvw (mgr.14162) 362 : cluster [DBG] pgmap v346: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:10:32.438 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:32.437+0000 7fcafc43a700 1 -- 172.21.15.155:0/3271359523 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcaf40f6510 msgr2=0x7fcaf40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:10:32.438 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:32.437+0000 7fcafc43a700 1 --2- 172.21.15.155:0/3271359523 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcaf40f6510 0x7fcaf40f6930 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7fcae4009fb0 tx=0x7fcae4009b30).stop 2024-02-10T09:10:32.439 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:32.437+0000 7fcafc43a700 1 -- 172.21.15.155:0/3271359523 shutdown_connections 2024-02-10T09:10:32.439 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:32.437+0000 7fcafc43a700 1 --2- 172.21.15.155:0/3271359523 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcaf40f6510 0x7fcaf40f6930 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:10:32.439 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:32.437+0000 7fcafc43a700 1 -- 172.21.15.155:0/3271359523 >> 172.21.15.155:0/3271359523 conn(0x7fcaf40f1c70 msgr2=0x7fcaf40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:10:32.439 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:32.437+0000 7fcafc43a700 1 -- 172.21.15.155:0/3271359523 shutdown_connections 2024-02-10T09:10:32.439 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:32.437+0000 7fcafc43a700 1 -- 172.21.15.155:0/3271359523 wait complete. 2024-02-10T09:10:32.439 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:32.438+0000 7fcafc43a700 1 Processor -- start 2024-02-10T09:10:32.439 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:32.438+0000 7fcafc43a700 1 -- start start 2024-02-10T09:10:32.439 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:32.438+0000 7fcafc43a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcaf40f6510 0x7fcaf406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:10:32.439 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:32.438+0000 7fcafc43a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf4069e20 con 0x7fcaf40f6510 2024-02-10T09:10:32.439 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:32.438+0000 7fcafa1d6700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcaf40f6510 0x7fcaf406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:10:32.439 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:32.438+0000 7fcafa1d6700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcaf40f6510 0x7fcaf406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:40648/0 (socket says 172.21.15.155:40648) 2024-02-10T09:10:32.439 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:32.438+0000 7fcafa1d6700 1 -- 172.21.15.155:0/2368605659 learned_addr learned my addr 172.21.15.155:0/2368605659 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:10:32.441 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:32.439+0000 7fcafa1d6700 1 -- 172.21.15.155:0/2368605659 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcae4005040 con 0x7fcaf40f6510 2024-02-10T09:10:32.441 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:32.439+0000 7fcafa1d6700 1 --2- 172.21.15.155:0/2368605659 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcaf40f6510 0x7fcaf406b9a0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7fcae4005010 tx=0x7fcae400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:10:32.441 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:32.439+0000 7fcaeaffd700 1 -- 172.21.15.155:0/2368605659 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcae4009320 con 0x7fcaf40f6510 2024-02-10T09:10:32.441 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:32.439+0000 7fcafc43a700 1 -- 172.21.15.155:0/2368605659 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcaf4069fc0 con 0x7fcaf40f6510 2024-02-10T09:10:32.441 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:32.439+0000 7fcaeaffd700 1 -- 172.21.15.155:0/2368605659 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcae4013380 con 0x7fcaf40f6510 2024-02-10T09:10:32.441 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:32.439+0000 7fcaeaffd700 1 -- 172.21.15.155:0/2368605659 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcae4014e50 con 0x7fcaf40f6510 2024-02-10T09:10:32.441 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:32.439+0000 7fcafc43a700 1 -- 172.21.15.155:0/2368605659 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaf406a450 con 0x7fcaf40f6510 2024-02-10T09:10:32.441 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:32.440+0000 7fcaeaffd700 1 -- 172.21.15.155:0/2368605659 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcae4014420 con 0x7fcaf40f6510 2024-02-10T09:10:32.441 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:32.440+0000 7fcafc43a700 1 -- 172.21.15.155:0/2368605659 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcaf4059100 con 0x7fcaf40f6510 2024-02-10T09:10:32.441 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:32.440+0000 7fcaeaffd700 1 --2- 172.21.15.155:0/2368605659 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fcae002ff10 0x7fcae00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:10:32.442 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:32.440+0000 7fcaeaffd700 1 -- 172.21.15.155:0/2368605659 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcae4040f40 con 0x7fcaf40f6510 2024-02-10T09:10:32.442 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:32.440+0000 7fcaf99d5700 1 --2- 172.21.15.155:0/2368605659 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fcae002ff10 0x7fcae00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:10:32.442 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:32.441+0000 7fcaf99d5700 1 --2- 172.21.15.155:0/2368605659 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fcae002ff10 0x7fcae00323d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7fcaf00096a0 tx=0x7fcaf0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:10:32.445 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:32.445+0000 7fcaeaffd700 1 -- 172.21.15.155:0/2368605659 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcae40095f0 con 0x7fcaf40f6510 2024-02-10T09:10:32.652 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:32.652+0000 7fcafc43a700 1 -- 172.21.15.155:0/2368605659 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcaf40fc0a0 con 0x7fcaf40f6510 2024-02-10T09:10:32.653 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:32.653+0000 7fcaeaffd700 1 -- 172.21.15.155:0/2368605659 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcae401db70 con 0x7fcaf40f6510 2024-02-10T09:10:32.653 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:10:32.653 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:10:32.654 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:32.654+0000 7fcafc43a700 1 -- 172.21.15.155:0/2368605659 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fcae002ff10 msgr2=0x7fcae00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:10:32.655 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:32.654+0000 7fcafc43a700 1 --2- 172.21.15.155:0/2368605659 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fcae002ff10 0x7fcae00323d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7fcaf00096a0 tx=0x7fcaf0006b40).stop 2024-02-10T09:10:32.655 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:32.654+0000 7fcafc43a700 1 -- 172.21.15.155:0/2368605659 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcaf40f6510 msgr2=0x7fcaf406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:10:32.655 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:32.654+0000 7fcafc43a700 1 --2- 172.21.15.155:0/2368605659 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcaf40f6510 0x7fcaf406b9a0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7fcae4005010 tx=0x7fcae400b040).stop 2024-02-10T09:10:32.655 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:32.654+0000 7fcafc43a700 1 -- 172.21.15.155:0/2368605659 shutdown_connections 2024-02-10T09:10:32.655 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:32.654+0000 7fcafc43a700 1 --2- 172.21.15.155:0/2368605659 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fcae002ff10 0x7fcae00323d0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:10:32.655 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:32.654+0000 7fcafc43a700 1 --2- 172.21.15.155:0/2368605659 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcaf40f6510 0x7fcaf406b9a0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:10:32.655 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:32.654+0000 7fcafc43a700 1 -- 172.21.15.155:0/2368605659 >> 172.21.15.155:0/2368605659 conn(0x7fcaf40f1c70 msgr2=0x7fcaf40f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:10:32.655 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:32.654+0000 7fcafc43a700 1 -- 172.21.15.155:0/2368605659 shutdown_connections 2024-02-10T09:10:32.655 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:32.655+0000 7fcafc43a700 1 -- 172.21.15.155:0/2368605659 wait complete. 2024-02-10T09:10:32.655 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:10:33.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:10:32 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:10:31.746616+0000 mgr.smithi077.dxntvw (mgr.14162) 363 : cluster [DBG] pgmap v347: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:10:33.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:10:32 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:10:32.653474+0000 mon.smithi077 (mon.0) 320 : audit [DBG] from='client.? 172.21.15.155:0/2368605659' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:10:34.491 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:10:34.491 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:10:35.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:10:34 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:10:33.746954+0000 mgr.smithi077.dxntvw (mgr.14162) 364 : cluster [DBG] pgmap v348: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:10:36.790 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:36.788+0000 7f12da107700 1 -- 172.21.15.155:0/3474378274 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12d40f51a0 msgr2=0x7f12d40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:10:36.790 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:36.788+0000 7f12da107700 1 --2- 172.21.15.155:0/3474378274 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12d40f51a0 0x7f12d40f55c0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7f12c4009fb0 tx=0x7f12c4009b30).stop 2024-02-10T09:10:36.790 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:36.788+0000 7f12da107700 1 -- 172.21.15.155:0/3474378274 shutdown_connections 2024-02-10T09:10:36.790 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:36.788+0000 7f12da107700 1 --2- 172.21.15.155:0/3474378274 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12d40f51a0 0x7f12d40f55c0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:10:36.790 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:36.788+0000 7f12da107700 1 -- 172.21.15.155:0/3474378274 >> 172.21.15.155:0/3474378274 conn(0x7f12d40f09a0 msgr2=0x7f12d40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:10:36.790 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:36.788+0000 7f12da107700 1 -- 172.21.15.155:0/3474378274 shutdown_connections 2024-02-10T09:10:36.790 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:36.788+0000 7f12da107700 1 -- 172.21.15.155:0/3474378274 wait complete. 2024-02-10T09:10:36.790 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:36.789+0000 7f12da107700 1 Processor -- start 2024-02-10T09:10:36.791 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:36.789+0000 7f12da107700 1 -- start start 2024-02-10T09:10:36.791 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:36.789+0000 7f12da107700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12d40f51a0 0x7f12d41054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:10:36.791 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:36.789+0000 7f12da107700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12d41059c0 con 0x7f12d40f51a0 2024-02-10T09:10:36.791 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:36.789+0000 7f12d9105700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12d40f51a0 0x7f12d41054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:10:36.791 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:36.789+0000 7f12d9105700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12d40f51a0 0x7f12d41054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:40668/0 (socket says 172.21.15.155:40668) 2024-02-10T09:10:36.791 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:36.789+0000 7f12d9105700 1 -- 172.21.15.155:0/3906516717 learned_addr learned my addr 172.21.15.155:0/3906516717 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:10:36.791 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:36.789+0000 7f12d9105700 1 -- 172.21.15.155:0/3906516717 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12c4005040 con 0x7f12d40f51a0 2024-02-10T09:10:36.791 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:36.790+0000 7f12d9105700 1 --2- 172.21.15.155:0/3906516717 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12d40f51a0 0x7f12d41054d0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f12c40089f0 tx=0x7f12c400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:10:36.791 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:36.790+0000 7f12c9ffb700 1 -- 172.21.15.155:0/3906516717 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12c4009320 con 0x7f12d40f51a0 2024-02-10T09:10:36.791 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:36.790+0000 7f12da107700 1 -- 172.21.15.155:0/3906516717 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12d4105bc0 con 0x7f12d40f51a0 2024-02-10T09:10:36.791 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:36.790+0000 7f12c9ffb700 1 -- 172.21.15.155:0/3906516717 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f12c4013380 con 0x7f12d40f51a0 2024-02-10T09:10:36.791 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:36.790+0000 7f12c9ffb700 1 -- 172.21.15.155:0/3906516717 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12c4014e20 con 0x7f12d40f51a0 2024-02-10T09:10:36.791 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:36.790+0000 7f12da107700 1 -- 172.21.15.155:0/3906516717 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12d4106050 con 0x7f12d40f51a0 2024-02-10T09:10:36.792 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:36.791+0000 7f12c9ffb700 1 -- 172.21.15.155:0/3906516717 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f12c40095f0 con 0x7f12d40f51a0 2024-02-10T09:10:36.792 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:36.791+0000 7f12da107700 1 -- 172.21.15.155:0/3906516717 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12d4045ad0 con 0x7f12d40f51a0 2024-02-10T09:10:36.792 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:36.791+0000 7f12c9ffb700 1 --2- 172.21.15.155:0/3906516717 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f12c002ff10 0x7f12c00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:10:36.792 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:36.791+0000 7f12c9ffb700 1 -- 172.21.15.155:0/3906516717 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f12c4042c70 con 0x7f12d40f51a0 2024-02-10T09:10:36.793 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:36.791+0000 7f12d8904700 1 --2- 172.21.15.155:0/3906516717 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f12c002ff10 0x7f12c00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:10:36.793 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:36.792+0000 7f12d8904700 1 --2- 172.21.15.155:0/3906516717 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f12c002ff10 0x7f12c00323d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f12d00096a0 tx=0x7f12d0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:10:36.796 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:36.796+0000 7f12c9ffb700 1 -- 172.21.15.155:0/3906516717 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f12c4014420 con 0x7f12d40f51a0 2024-02-10T09:10:37.002 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:37.002+0000 7f12da107700 1 -- 172.21.15.155:0/3906516717 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f12d4106f00 con 0x7f12d40f51a0 2024-02-10T09:10:37.003 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:37.003+0000 7f12c9ffb700 1 -- 172.21.15.155:0/3906516717 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f12c4011290 con 0x7f12d40f51a0 2024-02-10T09:10:37.003 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:10:37.003 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:10:37.005 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:37.004+0000 7f12da107700 1 -- 172.21.15.155:0/3906516717 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f12c002ff10 msgr2=0x7f12c00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:10:37.005 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:37.004+0000 7f12da107700 1 --2- 172.21.15.155:0/3906516717 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f12c002ff10 0x7f12c00323d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f12d00096a0 tx=0x7f12d0006b40).stop 2024-02-10T09:10:37.005 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:37.004+0000 7f12da107700 1 -- 172.21.15.155:0/3906516717 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12d40f51a0 msgr2=0x7f12d41054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:10:37.005 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:37.004+0000 7f12da107700 1 --2- 172.21.15.155:0/3906516717 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12d40f51a0 0x7f12d41054d0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f12c40089f0 tx=0x7f12c400b040).stop 2024-02-10T09:10:37.005 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:37.004+0000 7f12da107700 1 -- 172.21.15.155:0/3906516717 shutdown_connections 2024-02-10T09:10:37.005 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:37.004+0000 7f12da107700 1 --2- 172.21.15.155:0/3906516717 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f12c002ff10 0x7f12c00323d0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:10:37.005 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:37.004+0000 7f12da107700 1 --2- 172.21.15.155:0/3906516717 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12d40f51a0 0x7f12d41054d0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:10:37.005 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:37.004+0000 7f12da107700 1 -- 172.21.15.155:0/3906516717 >> 172.21.15.155:0/3906516717 conn(0x7f12d40f09a0 msgr2=0x7f12d40f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:10:37.005 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:37.005+0000 7f12da107700 1 -- 172.21.15.155:0/3906516717 shutdown_connections 2024-02-10T09:10:37.005 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:37.005+0000 7f12da107700 1 -- 172.21.15.155:0/3906516717 wait complete. 2024-02-10T09:10:37.006 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:10:37.148 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:10:36 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:10:35.747373+0000 mgr.smithi077.dxntvw (mgr.14162) 365 : cluster [DBG] pgmap v349: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:10:38.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:10:37 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:10:37.003451+0000 mon.smithi077 (mon.0) 321 : audit [DBG] from='client.? 172.21.15.155:0/3906516717' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:10:38.883 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:10:38.883 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:10:39.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:10:38 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:10:37.747631+0000 mgr.smithi077.dxntvw (mgr.14162) 366 : cluster [DBG] pgmap v350: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:10:41.009 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:41.007+0000 7f9d6537f700 1 -- 172.21.15.155:0/1103087245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d600f51a0 msgr2=0x7f9d600f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:10:41.009 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:41.007+0000 7f9d6537f700 1 --2- 172.21.15.155:0/1103087245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d600f51a0 0x7f9d600f55c0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7f9d50009fb0 tx=0x7f9d50009b30).stop 2024-02-10T09:10:41.009 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:41.008+0000 7f9d6537f700 1 -- 172.21.15.155:0/1103087245 shutdown_connections 2024-02-10T09:10:41.009 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:41.008+0000 7f9d6537f700 1 --2- 172.21.15.155:0/1103087245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d600f51a0 0x7f9d600f55c0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:10:41.010 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:41.008+0000 7f9d6537f700 1 -- 172.21.15.155:0/1103087245 >> 172.21.15.155:0/1103087245 conn(0x7f9d600f09a0 msgr2=0x7f9d600f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:10:41.010 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:41.008+0000 7f9d6537f700 1 -- 172.21.15.155:0/1103087245 shutdown_connections 2024-02-10T09:10:41.010 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:41.008+0000 7f9d6537f700 1 -- 172.21.15.155:0/1103087245 wait complete. 2024-02-10T09:10:41.010 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:41.008+0000 7f9d6537f700 1 Processor -- start 2024-02-10T09:10:41.010 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:41.008+0000 7f9d6537f700 1 -- start start 2024-02-10T09:10:41.012 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:41.008+0000 7f9d6537f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d600f51a0 0x7f9d600fd270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:10:41.012 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:41.008+0000 7f9d6537f700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d600fb6f0 con 0x7f9d600f51a0 2024-02-10T09:10:41.012 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:41.009+0000 7f9d5ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d600f51a0 0x7f9d600fd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:10:41.012 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:41.009+0000 7f9d5ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d600f51a0 0x7f9d600fd270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:40680/0 (socket says 172.21.15.155:40680) 2024-02-10T09:10:41.012 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:41.009+0000 7f9d5ffff700 1 -- 172.21.15.155:0/3460462322 learned_addr learned my addr 172.21.15.155:0/3460462322 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:10:41.012 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:41.009+0000 7f9d5ffff700 1 -- 172.21.15.155:0/3460462322 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d50005040 con 0x7f9d600f51a0 2024-02-10T09:10:41.012 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:41.009+0000 7f9d5ffff700 1 --2- 172.21.15.155:0/3460462322 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d600f51a0 0x7f9d600fd270 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f9d50000c00 tx=0x7f9d5000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:10:41.012 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:41.009+0000 7f9d5cff9700 1 -- 172.21.15.155:0/3460462322 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d500095f0 con 0x7f9d600f51a0 2024-02-10T09:10:41.012 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:41.009+0000 7f9d6537f700 1 -- 172.21.15.155:0/3460462322 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d600fb890 con 0x7f9d600f51a0 2024-02-10T09:10:41.012 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:41.009+0000 7f9d5cff9700 1 -- 172.21.15.155:0/3460462322 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9d50007e90 con 0x7f9d600f51a0 2024-02-10T09:10:41.012 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:41.009+0000 7f9d5cff9700 1 -- 172.21.15.155:0/3460462322 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d50014db0 con 0x7f9d600f51a0 2024-02-10T09:10:41.013 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:41.009+0000 7f9d6537f700 1 -- 172.21.15.155:0/3460462322 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d600fbd20 con 0x7f9d600f51a0 2024-02-10T09:10:41.013 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:41.010+0000 7f9d5cff9700 1 -- 172.21.15.155:0/3460462322 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9d50025070 con 0x7f9d600f51a0 2024-02-10T09:10:41.013 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:41.010+0000 7f9d5cff9700 1 --2- 172.21.15.155:0/3460462322 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f9d4802ff60 0x7f9d48032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:10:41.013 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:41.010+0000 7f9d5cff9700 1 -- 172.21.15.155:0/3460462322 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9d500409e0 con 0x7f9d600f51a0 2024-02-10T09:10:41.013 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:41.010+0000 7f9d6537f700 1 -- 172.21.15.155:0/3460462322 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d60059100 con 0x7f9d600f51a0 2024-02-10T09:10:41.013 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:41.011+0000 7f9d5f7fe700 1 --2- 172.21.15.155:0/3460462322 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f9d4802ff60 0x7f9d48032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:10:41.013 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:41.011+0000 7f9d5f7fe700 1 --2- 172.21.15.155:0/3460462322 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f9d4802ff60 0x7f9d48032420 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f9d540096a0 tx=0x7f9d54006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:10:41.016 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:41.015+0000 7f9d5cff9700 1 -- 172.21.15.155:0/3460462322 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9d50026350 con 0x7f9d600f51a0 2024-02-10T09:10:41.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:10:40 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:10:39.747908+0000 mgr.smithi077.dxntvw (mgr.14162) 367 : cluster [DBG] pgmap v351: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:10:41.222 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:41.221+0000 7f9d6537f700 1 -- 172.21.15.155:0/3460462322 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9d600fad30 con 0x7f9d600f51a0 2024-02-10T09:10:41.223 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:41.221+0000 7f9d5cff9700 1 -- 172.21.15.155:0/3460462322 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9d50025300 con 0x7f9d600f51a0 2024-02-10T09:10:41.223 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:10:41.223 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:10:41.224 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:41.223+0000 7f9d6537f700 1 -- 172.21.15.155:0/3460462322 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f9d4802ff60 msgr2=0x7f9d48032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:10:41.225 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:41.223+0000 7f9d6537f700 1 --2- 172.21.15.155:0/3460462322 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f9d4802ff60 0x7f9d48032420 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f9d540096a0 tx=0x7f9d54006b40).stop 2024-02-10T09:10:41.225 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:41.223+0000 7f9d6537f700 1 -- 172.21.15.155:0/3460462322 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d600f51a0 msgr2=0x7f9d600fd270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:10:41.225 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:41.223+0000 7f9d6537f700 1 --2- 172.21.15.155:0/3460462322 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d600f51a0 0x7f9d600fd270 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f9d50000c00 tx=0x7f9d5000b040).stop 2024-02-10T09:10:41.225 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:41.223+0000 7f9d6537f700 1 -- 172.21.15.155:0/3460462322 shutdown_connections 2024-02-10T09:10:41.225 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:41.223+0000 7f9d6537f700 1 --2- 172.21.15.155:0/3460462322 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f9d4802ff60 0x7f9d48032420 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:10:41.225 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:41.223+0000 7f9d6537f700 1 --2- 172.21.15.155:0/3460462322 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9d600f51a0 0x7f9d600fd270 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:10:41.225 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:41.223+0000 7f9d6537f700 1 -- 172.21.15.155:0/3460462322 >> 172.21.15.155:0/3460462322 conn(0x7f9d600f09a0 msgr2=0x7f9d6005fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:10:41.225 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:41.223+0000 7f9d6537f700 1 -- 172.21.15.155:0/3460462322 shutdown_connections 2024-02-10T09:10:41.225 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:41.223+0000 7f9d6537f700 1 -- 172.21.15.155:0/3460462322 wait complete. 2024-02-10T09:10:41.226 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:10:42.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:10:41 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:10:41.223042+0000 mon.smithi077 (mon.0) 322 : audit [DBG] from='client.? 172.21.15.155:0/3460462322' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:10:42.900 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:10:42.900 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:10:43.148 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:10:42 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:10:41.748294+0000 mgr.smithi077.dxntvw (mgr.14162) 368 : cluster [DBG] pgmap v352: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:10:45.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:10:44 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:10:43.748636+0000 mgr.smithi077.dxntvw (mgr.14162) 369 : cluster [DBG] pgmap v353: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:10:45.151 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:45.148+0000 7f79a0f03700 1 -- 172.21.15.155:0/3283565008 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f799c0f6510 msgr2=0x7f799c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:10:45.151 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:45.148+0000 7f79a0f03700 1 --2- 172.21.15.155:0/3283565008 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f799c0f6510 0x7f799c0f6930 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7f798c009fb0 tx=0x7f798c009b30).stop 2024-02-10T09:10:45.151 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:45.148+0000 7f79a0f03700 1 -- 172.21.15.155:0/3283565008 shutdown_connections 2024-02-10T09:10:45.151 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:45.148+0000 7f79a0f03700 1 --2- 172.21.15.155:0/3283565008 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f799c0f6510 0x7f799c0f6930 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:10:45.151 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:45.148+0000 7f79a0f03700 1 -- 172.21.15.155:0/3283565008 >> 172.21.15.155:0/3283565008 conn(0x7f799c0f1c70 msgr2=0x7f799c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:10:45.151 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:45.148+0000 7f79a0f03700 1 -- 172.21.15.155:0/3283565008 shutdown_connections 2024-02-10T09:10:45.151 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:45.148+0000 7f79a0f03700 1 -- 172.21.15.155:0/3283565008 wait complete. 2024-02-10T09:10:45.151 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:45.148+0000 7f79a0f03700 1 Processor -- start 2024-02-10T09:10:45.151 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:45.148+0000 7f79a0f03700 1 -- start start 2024-02-10T09:10:45.152 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:45.148+0000 7f79a0f03700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f799c0f6510 0x7f799c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:10:45.152 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:45.148+0000 7f79a0f03700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f799c069e20 con 0x7f799c0f6510 2024-02-10T09:10:45.152 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:45.149+0000 7f799a59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f799c0f6510 0x7f799c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:10:45.152 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:45.149+0000 7f799a59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f799c0f6510 0x7f799c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:34254/0 (socket says 172.21.15.155:34254) 2024-02-10T09:10:45.152 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:45.149+0000 7f799a59c700 1 -- 172.21.15.155:0/3856355727 learned_addr learned my addr 172.21.15.155:0/3856355727 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:10:45.152 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:45.149+0000 7f799a59c700 1 -- 172.21.15.155:0/3856355727 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f798c005040 con 0x7f799c0f6510 2024-02-10T09:10:45.152 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:45.149+0000 7f799a59c700 1 --2- 172.21.15.155:0/3856355727 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f799c0f6510 0x7f799c06b9a0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f798c005010 tx=0x7f798c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:10:45.152 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:45.149+0000 7f798affd700 1 -- 172.21.15.155:0/3856355727 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f798c009320 con 0x7f799c0f6510 2024-02-10T09:10:45.152 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:45.149+0000 7f79a0f03700 1 -- 172.21.15.155:0/3856355727 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f799c069fc0 con 0x7f799c0f6510 2024-02-10T09:10:45.152 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:45.149+0000 7f79a0f03700 1 -- 172.21.15.155:0/3856355727 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f799c06a450 con 0x7f799c0f6510 2024-02-10T09:10:45.152 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:45.149+0000 7f798affd700 1 -- 172.21.15.155:0/3856355727 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f798c013380 con 0x7f799c0f6510 2024-02-10T09:10:45.152 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:45.149+0000 7f798affd700 1 -- 172.21.15.155:0/3856355727 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f798c014df0 con 0x7f799c0f6510 2024-02-10T09:10:45.153 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:45.150+0000 7f798affd700 1 -- 172.21.15.155:0/3856355727 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f798c0095f0 con 0x7f799c0f6510 2024-02-10T09:10:45.153 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:45.150+0000 7f79a0f03700 1 -- 172.21.15.155:0/3856355727 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7980004fa0 con 0x7f799c0f6510 2024-02-10T09:10:45.154 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:45.150+0000 7f798affd700 1 --2- 172.21.15.155:0/3856355727 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f798402ff10 0x7f79840323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:10:45.154 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:45.150+0000 7f798affd700 1 -- 172.21.15.155:0/3856355727 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f798c040df0 con 0x7f799c0f6510 2024-02-10T09:10:45.154 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:45.151+0000 7f7999d9b700 1 --2- 172.21.15.155:0/3856355727 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f798402ff10 0x7f79840323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:10:45.154 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:45.151+0000 7f7999d9b700 1 --2- 172.21.15.155:0/3856355727 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f798402ff10 0x7f79840323d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f79900096a0 tx=0x7f7990006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:10:45.156 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:45.155+0000 7f798affd700 1 -- 172.21.15.155:0/3856355727 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f798c014420 con 0x7f799c0f6510 2024-02-10T09:10:45.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:45.358+0000 7f79a0f03700 1 -- 172.21.15.155:0/3856355727 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7980005670 con 0x7f799c0f6510 2024-02-10T09:10:45.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:45.359+0000 7f798affd700 1 -- 172.21.15.155:0/3856355727 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f798c01d9d0 con 0x7f799c0f6510 2024-02-10T09:10:45.360 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:10:45.360 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:10:45.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:45.360+0000 7f79a0f03700 1 -- 172.21.15.155:0/3856355727 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f798402ff10 msgr2=0x7f79840323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:10:45.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:45.360+0000 7f79a0f03700 1 --2- 172.21.15.155:0/3856355727 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f798402ff10 0x7f79840323d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f79900096a0 tx=0x7f7990006b40).stop 2024-02-10T09:10:45.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:45.360+0000 7f79a0f03700 1 -- 172.21.15.155:0/3856355727 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f799c0f6510 msgr2=0x7f799c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:10:45.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:45.360+0000 7f79a0f03700 1 --2- 172.21.15.155:0/3856355727 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f799c0f6510 0x7f799c06b9a0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f798c005010 tx=0x7f798c00b040).stop 2024-02-10T09:10:45.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:45.361+0000 7f79a0f03700 1 -- 172.21.15.155:0/3856355727 shutdown_connections 2024-02-10T09:10:45.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:45.361+0000 7f79a0f03700 1 --2- 172.21.15.155:0/3856355727 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f798402ff10 0x7f79840323d0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:10:45.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:45.361+0000 7f79a0f03700 1 --2- 172.21.15.155:0/3856355727 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f799c0f6510 0x7f799c06b9a0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:10:45.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:45.361+0000 7f79a0f03700 1 -- 172.21.15.155:0/3856355727 >> 172.21.15.155:0/3856355727 conn(0x7f799c0f1c70 msgr2=0x7f799c0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:10:45.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:45.361+0000 7f79a0f03700 1 -- 172.21.15.155:0/3856355727 shutdown_connections 2024-02-10T09:10:45.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:45.361+0000 7f79a0f03700 1 -- 172.21.15.155:0/3856355727 wait complete. 2024-02-10T09:10:45.363 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:10:46.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:10:45 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:10:45.360494+0000 mon.smithi077 (mon.0) 323 : audit [DBG] from='client.? 172.21.15.155:0/3856355727' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:10:47.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:10:46 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:10:45.748911+0000 mgr.smithi077.dxntvw (mgr.14162) 370 : cluster [DBG] pgmap v354: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:10:47.207 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:10:47.208 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:10:48.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:10:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:10:47.100644+0000 mon.smithi077 (mon.0) 324 : audit [INF] from='mgr.14162 172.21.15.77:0/2029106299' entity='mgr.smithi077.dxntvw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi077.dxntvw/mirror_snapshot_schedule"}]: dispatch 2024-02-10T09:10:48.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:10:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:10:47.106103+0000 mon.smithi077 (mon.0) 325 : audit [INF] from='mgr.14162 172.21.15.77:0/2029106299' entity='mgr.smithi077.dxntvw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi077.dxntvw/trash_purge_schedule"}]: dispatch 2024-02-10T09:10:49.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:10:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:10:47.749193+0000 mgr.smithi077.dxntvw (mgr.14162) 371 : cluster [DBG] pgmap v355: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:10:49.377 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:49.374+0000 7f2bdc460700 1 -- 172.21.15.155:0/1925009991 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2bd40f6480 msgr2=0x7f2bd40f68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:10:49.377 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:49.374+0000 7f2bdc460700 1 --2- 172.21.15.155:0/1925009991 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2bd40f6480 0x7f2bd40f68a0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7f2bc4009fb0 tx=0x7f2bc4009b30).stop 2024-02-10T09:10:49.377 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:49.374+0000 7f2bdc460700 1 -- 172.21.15.155:0/1925009991 shutdown_connections 2024-02-10T09:10:49.377 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:49.374+0000 7f2bdc460700 1 --2- 172.21.15.155:0/1925009991 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2bd40f6480 0x7f2bd40f68a0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:10:49.377 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:49.374+0000 7f2bdc460700 1 -- 172.21.15.155:0/1925009991 >> 172.21.15.155:0/1925009991 conn(0x7f2bd40f1c00 msgr2=0x7f2bd40f4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:10:49.377 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:49.374+0000 7f2bdc460700 1 -- 172.21.15.155:0/1925009991 shutdown_connections 2024-02-10T09:10:49.377 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:49.374+0000 7f2bdc460700 1 -- 172.21.15.155:0/1925009991 wait complete. 2024-02-10T09:10:49.377 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:49.375+0000 7f2bdc460700 1 Processor -- start 2024-02-10T09:10:49.378 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:49.375+0000 7f2bdc460700 1 -- start start 2024-02-10T09:10:49.378 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:49.375+0000 7f2bdc460700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2bd40f6480 0x7f2bd406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:10:49.378 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:49.375+0000 7f2bdc460700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd4069e20 con 0x7f2bd40f6480 2024-02-10T09:10:49.378 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:49.375+0000 7f2bda1fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2bd40f6480 0x7f2bd406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:10:49.378 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:49.375+0000 7f2bda1fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2bd40f6480 0x7f2bd406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:34266/0 (socket says 172.21.15.155:34266) 2024-02-10T09:10:49.378 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:49.375+0000 7f2bda1fc700 1 -- 172.21.15.155:0/1932173730 learned_addr learned my addr 172.21.15.155:0/1932173730 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:10:49.378 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:49.375+0000 7f2bda1fc700 1 -- 172.21.15.155:0/1932173730 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bc4005040 con 0x7f2bd40f6480 2024-02-10T09:10:49.378 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:49.375+0000 7f2bda1fc700 1 --2- 172.21.15.155:0/1932173730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2bd40f6480 0x7f2bd406b9a0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f2bc4005010 tx=0x7f2bc400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:10:49.378 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:49.376+0000 7f2bcaffd700 1 -- 172.21.15.155:0/1932173730 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2bc4009320 con 0x7f2bd40f6480 2024-02-10T09:10:49.378 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:49.376+0000 7f2bdc460700 1 -- 172.21.15.155:0/1932173730 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bd4069fc0 con 0x7f2bd40f6480 2024-02-10T09:10:49.378 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:49.376+0000 7f2bcaffd700 1 -- 172.21.15.155:0/1932173730 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2bc4013380 con 0x7f2bd40f6480 2024-02-10T09:10:49.378 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:49.376+0000 7f2bcaffd700 1 -- 172.21.15.155:0/1932173730 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2bc4014e50 con 0x7f2bd40f6480 2024-02-10T09:10:49.378 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:49.376+0000 7f2bdc460700 1 -- 172.21.15.155:0/1932173730 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bd406a450 con 0x7f2bd40f6480 2024-02-10T09:10:49.379 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:49.376+0000 7f2bcaffd700 1 -- 172.21.15.155:0/1932173730 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2bc4014420 con 0x7f2bd40f6480 2024-02-10T09:10:49.379 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:49.377+0000 7f2bcaffd700 1 --2- 172.21.15.155:0/1932173730 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f2bc002ff10 0x7f2bc00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:10:49.379 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:49.377+0000 7f2bdc460700 1 -- 172.21.15.155:0/1932173730 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bd4059100 con 0x7f2bd40f6480 2024-02-10T09:10:49.379 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:49.377+0000 7f2bcaffd700 1 -- 172.21.15.155:0/1932173730 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2bc4040f90 con 0x7f2bd40f6480 2024-02-10T09:10:49.379 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:49.377+0000 7f2bd99fb700 1 --2- 172.21.15.155:0/1932173730 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f2bc002ff10 0x7f2bc00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:10:49.380 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:49.377+0000 7f2bd99fb700 1 --2- 172.21.15.155:0/1932173730 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f2bc002ff10 0x7f2bc00323d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f2bd00096a0 tx=0x7f2bd0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:10:49.382 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:49.381+0000 7f2bcaffd700 1 -- 172.21.15.155:0/1932173730 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2bc4014790 con 0x7f2bd40f6480 2024-02-10T09:10:49.591 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:49.589+0000 7f2bdc460700 1 -- 172.21.15.155:0/1932173730 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2bd40fc090 con 0x7f2bd40f6480 2024-02-10T09:10:49.591 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:49.590+0000 7f2bcaffd700 1 -- 172.21.15.155:0/1932173730 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2bc402c090 con 0x7f2bd40f6480 2024-02-10T09:10:49.592 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:10:49.592 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:10:49.593 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:49.591+0000 7f2bdc460700 1 -- 172.21.15.155:0/1932173730 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f2bc002ff10 msgr2=0x7f2bc00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:10:49.593 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:49.591+0000 7f2bdc460700 1 --2- 172.21.15.155:0/1932173730 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f2bc002ff10 0x7f2bc00323d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f2bd00096a0 tx=0x7f2bd0006b40).stop 2024-02-10T09:10:49.593 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:49.591+0000 7f2bdc460700 1 -- 172.21.15.155:0/1932173730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2bd40f6480 msgr2=0x7f2bd406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:10:49.593 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:49.591+0000 7f2bdc460700 1 --2- 172.21.15.155:0/1932173730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2bd40f6480 0x7f2bd406b9a0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f2bc4005010 tx=0x7f2bc400b040).stop 2024-02-10T09:10:49.593 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:49.592+0000 7f2bdc460700 1 -- 172.21.15.155:0/1932173730 shutdown_connections 2024-02-10T09:10:49.594 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:49.592+0000 7f2bdc460700 1 --2- 172.21.15.155:0/1932173730 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f2bc002ff10 0x7f2bc00323d0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:10:49.594 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:49.592+0000 7f2bdc460700 1 --2- 172.21.15.155:0/1932173730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2bd40f6480 0x7f2bd406b9a0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:10:49.594 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:49.592+0000 7f2bdc460700 1 -- 172.21.15.155:0/1932173730 >> 172.21.15.155:0/1932173730 conn(0x7f2bd40f1c00 msgr2=0x7f2bd40f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:10:49.594 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:49.592+0000 7f2bdc460700 1 -- 172.21.15.155:0/1932173730 shutdown_connections 2024-02-10T09:10:49.594 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:49.592+0000 7f2bdc460700 1 -- 172.21.15.155:0/1932173730 wait complete. 2024-02-10T09:10:49.594 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:10:50.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:10:49 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:10:49.591719+0000 mon.smithi077 (mon.0) 326 : audit [DBG] from='client.? 172.21.15.155:0/1932173730' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:10:51.148 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:10:50 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:10:49.749584+0000 mgr.smithi077.dxntvw (mgr.14162) 372 : cluster [DBG] pgmap v356: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:10:51.360 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:10:51.361 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:10:53.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:10:52 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:10:51.749864+0000 mgr.smithi077.dxntvw (mgr.14162) 373 : cluster [DBG] pgmap v357: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:10:53.561 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:53.560+0000 7fb706357700 1 -- 172.21.15.155:0/4080496839 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7000f5410 msgr2=0x7fb7000f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:10:53.561 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:53.560+0000 7fb706357700 1 --2- 172.21.15.155:0/4080496839 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7000f5410 0x7fb7000f5830 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7fb6f0009fb0 tx=0x7fb6f0009b30).stop 2024-02-10T09:10:53.561 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:53.560+0000 7fb706357700 1 -- 172.21.15.155:0/4080496839 shutdown_connections 2024-02-10T09:10:53.561 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:53.560+0000 7fb706357700 1 --2- 172.21.15.155:0/4080496839 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7000f5410 0x7fb7000f5830 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:10:53.561 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:53.560+0000 7fb706357700 1 -- 172.21.15.155:0/4080496839 >> 172.21.15.155:0/4080496839 conn(0x7fb7000f0b70 msgr2=0x7fb7000f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:10:53.561 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:53.560+0000 7fb706357700 1 -- 172.21.15.155:0/4080496839 shutdown_connections 2024-02-10T09:10:53.562 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:53.560+0000 7fb706357700 1 -- 172.21.15.155:0/4080496839 wait complete. 2024-02-10T09:10:53.562 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:53.560+0000 7fb706357700 1 Processor -- start 2024-02-10T09:10:53.563 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:53.560+0000 7fb706357700 1 -- start start 2024-02-10T09:10:53.563 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:53.561+0000 7fb706357700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7000f5410 0x7fb700109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:10:53.563 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:53.561+0000 7fb706357700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb700060220 con 0x7fb7000f5410 2024-02-10T09:10:53.563 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:53.561+0000 7fb6fffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7000f5410 0x7fb700109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:10:53.563 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:53.561+0000 7fb6fffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7000f5410 0x7fb700109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:35740/0 (socket says 172.21.15.155:35740) 2024-02-10T09:10:53.563 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:53.561+0000 7fb6fffff700 1 -- 172.21.15.155:0/3037250107 learned_addr learned my addr 172.21.15.155:0/3037250107 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:10:53.564 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:53.561+0000 7fb6fffff700 1 -- 172.21.15.155:0/3037250107 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6f0005040 con 0x7fb7000f5410 2024-02-10T09:10:53.564 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:53.561+0000 7fb6fffff700 1 --2- 172.21.15.155:0/3037250107 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7000f5410 0x7fb700109500 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7fb6f0005010 tx=0x7fb6f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:10:53.564 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:53.561+0000 7fb6fcff9700 1 -- 172.21.15.155:0/3037250107 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb6f0009320 con 0x7fb7000f5410 2024-02-10T09:10:53.564 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:53.561+0000 7fb706357700 1 -- 172.21.15.155:0/3037250107 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb700060420 con 0x7fb7000f5410 2024-02-10T09:10:53.564 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:53.561+0000 7fb6fcff9700 1 -- 172.21.15.155:0/3037250107 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb6f0013380 con 0x7fb7000f5410 2024-02-10T09:10:53.564 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:53.561+0000 7fb6fcff9700 1 -- 172.21.15.155:0/3037250107 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb6f0014e50 con 0x7fb7000f5410 2024-02-10T09:10:53.564 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:53.561+0000 7fb706357700 1 -- 172.21.15.155:0/3037250107 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb700109bb0 con 0x7fb7000f5410 2024-02-10T09:10:53.565 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:53.562+0000 7fb6fcff9700 1 -- 172.21.15.155:0/3037250107 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb6f00263a0 con 0x7fb7000f5410 2024-02-10T09:10:53.565 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:53.563+0000 7fb706357700 1 -- 172.21.15.155:0/3037250107 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb700059100 con 0x7fb7000f5410 2024-02-10T09:10:53.565 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:53.563+0000 7fb6fcff9700 1 --2- 172.21.15.155:0/3037250107 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb6e802ff10 0x7fb6e80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:10:53.565 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:53.563+0000 7fb6fcff9700 1 -- 172.21.15.155:0/3037250107 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb6f0040e60 con 0x7fb7000f5410 2024-02-10T09:10:53.565 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:53.563+0000 7fb6ff7fe700 1 --2- 172.21.15.155:0/3037250107 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb6e802ff10 0x7fb6e80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:10:53.565 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:53.563+0000 7fb6ff7fe700 1 --2- 172.21.15.155:0/3037250107 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb6e802ff10 0x7fb6e80323d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fb6f40096a0 tx=0x7fb6f4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:10:53.568 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:53.567+0000 7fb6fcff9700 1 -- 172.21.15.155:0/3037250107 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb6f00126f0 con 0x7fb7000f5410 2024-02-10T09:10:53.775 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:53.774+0000 7fb706357700 1 -- 172.21.15.155:0/3037250107 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb7000fafa0 con 0x7fb7000f5410 2024-02-10T09:10:53.776 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:53.775+0000 7fb6fcff9700 1 -- 172.21.15.155:0/3037250107 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb6f000f340 con 0x7fb7000f5410 2024-02-10T09:10:53.776 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:10:53.776 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:10:53.778 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:53.776+0000 7fb706357700 1 -- 172.21.15.155:0/3037250107 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb6e802ff10 msgr2=0x7fb6e80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:10:53.778 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:53.776+0000 7fb706357700 1 --2- 172.21.15.155:0/3037250107 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb6e802ff10 0x7fb6e80323d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fb6f40096a0 tx=0x7fb6f4006b40).stop 2024-02-10T09:10:53.778 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:53.776+0000 7fb706357700 1 -- 172.21.15.155:0/3037250107 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7000f5410 msgr2=0x7fb700109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:10:53.778 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:53.776+0000 7fb706357700 1 --2- 172.21.15.155:0/3037250107 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7000f5410 0x7fb700109500 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7fb6f0005010 tx=0x7fb6f000b040).stop 2024-02-10T09:10:53.778 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:53.777+0000 7fb706357700 1 -- 172.21.15.155:0/3037250107 shutdown_connections 2024-02-10T09:10:53.778 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:53.777+0000 7fb706357700 1 --2- 172.21.15.155:0/3037250107 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fb6e802ff10 0x7fb6e80323d0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:10:53.779 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:53.777+0000 7fb706357700 1 --2- 172.21.15.155:0/3037250107 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb7000f5410 0x7fb700109500 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:10:53.779 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:53.777+0000 7fb706357700 1 -- 172.21.15.155:0/3037250107 >> 172.21.15.155:0/3037250107 conn(0x7fb7000f0b70 msgr2=0x7fb7000f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:10:53.779 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:53.777+0000 7fb706357700 1 -- 172.21.15.155:0/3037250107 shutdown_connections 2024-02-10T09:10:53.779 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:53.777+0000 7fb706357700 1 -- 172.21.15.155:0/3037250107 wait complete. 2024-02-10T09:10:53.779 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:10:54.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:10:53 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:10:53.776461+0000 mon.smithi077 (mon.0) 327 : audit [DBG] from='client.? 172.21.15.155:0/3037250107' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:10:55.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:10:54 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:10:53.750153+0000 mgr.smithi077.dxntvw (mgr.14162) 374 : cluster [DBG] pgmap v358: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:10:55.477 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:10:55.478 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:10:57.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:10:56 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:10:55.750439+0000 mgr.smithi077.dxntvw (mgr.14162) 375 : cluster [DBG] pgmap v359: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:10:57.613 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:57.611+0000 7f093556d700 1 -- 172.21.15.155:0/135764944 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09300f62a0 msgr2=0x7f09300f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:10:57.614 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:57.611+0000 7f093556d700 1 --2- 172.21.15.155:0/135764944 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09300f62a0 0x7f09300f66c0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7f0920009fb0 tx=0x7f0920009b30).stop 2024-02-10T09:10:57.614 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:57.611+0000 7f093556d700 1 -- 172.21.15.155:0/135764944 shutdown_connections 2024-02-10T09:10:57.614 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:57.611+0000 7f093556d700 1 --2- 172.21.15.155:0/135764944 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09300f62a0 0x7f09300f66c0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:10:57.614 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:57.611+0000 7f093556d700 1 -- 172.21.15.155:0/135764944 >> 172.21.15.155:0/135764944 conn(0x7f09300f1aa0 msgr2=0x7f09300f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:10:57.614 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:57.611+0000 7f093556d700 1 -- 172.21.15.155:0/135764944 shutdown_connections 2024-02-10T09:10:57.614 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:57.611+0000 7f093556d700 1 -- 172.21.15.155:0/135764944 wait complete. 2024-02-10T09:10:57.614 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:57.612+0000 7f093556d700 1 Processor -- start 2024-02-10T09:10:57.614 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:57.612+0000 7f093556d700 1 -- start start 2024-02-10T09:10:57.614 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:57.612+0000 7f093556d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09300f62a0 0x7f09301065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:10:57.614 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:57.612+0000 7f093556d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0930106ac0 con 0x7f09300f62a0 2024-02-10T09:10:57.614 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:57.612+0000 7f092ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09300f62a0 0x7f09301065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:10:57.614 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:57.612+0000 7f092ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09300f62a0 0x7f09301065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:35764/0 (socket says 172.21.15.155:35764) 2024-02-10T09:10:57.614 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:57.612+0000 7f092ffff700 1 -- 172.21.15.155:0/1371848179 learned_addr learned my addr 172.21.15.155:0/1371848179 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:10:57.615 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:57.613+0000 7f092ffff700 1 -- 172.21.15.155:0/1371848179 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0920005040 con 0x7f09300f62a0 2024-02-10T09:10:57.616 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:57.613+0000 7f092ffff700 1 --2- 172.21.15.155:0/1371848179 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09300f62a0 0x7f09301065d0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f09200089f0 tx=0x7f092000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:10:57.616 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:57.613+0000 7f092cff9700 1 -- 172.21.15.155:0/1371848179 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0920009320 con 0x7f09300f62a0 2024-02-10T09:10:57.616 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:57.613+0000 7f093556d700 1 -- 172.21.15.155:0/1371848179 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0930106cc0 con 0x7f09300f62a0 2024-02-10T09:10:57.616 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:57.613+0000 7f092cff9700 1 -- 172.21.15.155:0/1371848179 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0920013380 con 0x7f09300f62a0 2024-02-10T09:10:57.616 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:57.613+0000 7f092cff9700 1 -- 172.21.15.155:0/1371848179 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0920014e20 con 0x7f09300f62a0 2024-02-10T09:10:57.616 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:57.613+0000 7f093556d700 1 -- 172.21.15.155:0/1371848179 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0930107150 con 0x7f09300f62a0 2024-02-10T09:10:57.616 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:57.614+0000 7f092cff9700 1 -- 172.21.15.155:0/1371848179 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f09200095f0 con 0x7f09300f62a0 2024-02-10T09:10:57.616 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:57.614+0000 7f093556d700 1 -- 172.21.15.155:0/1371848179 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0930045ad0 con 0x7f09300f62a0 2024-02-10T09:10:57.616 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:57.614+0000 7f092cff9700 1 --2- 172.21.15.155:0/1371848179 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f091802ff10 0x7f09180323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:10:57.616 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:57.614+0000 7f092cff9700 1 -- 172.21.15.155:0/1371848179 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0920040f50 con 0x7f09300f62a0 2024-02-10T09:10:57.617 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:57.614+0000 7f092f7fe700 1 --2- 172.21.15.155:0/1371848179 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f091802ff10 0x7f09180323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:10:57.617 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:57.615+0000 7f092f7fe700 1 --2- 172.21.15.155:0/1371848179 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f091802ff10 0x7f09180323d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f09240096a0 tx=0x7f0924006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:10:57.620 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:57.619+0000 7f092cff9700 1 -- 172.21.15.155:0/1371848179 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0920014420 con 0x7f09300f62a0 2024-02-10T09:10:57.830 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:57.828+0000 7f093556d700 1 -- 172.21.15.155:0/1371848179 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0930108000 con 0x7f09300f62a0 2024-02-10T09:10:57.830 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:57.829+0000 7f092cff9700 1 -- 172.21.15.155:0/1371848179 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f092001d9d0 con 0x7f09300f62a0 2024-02-10T09:10:57.831 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:10:57.831 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:10:57.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:57.831+0000 7f093556d700 1 -- 172.21.15.155:0/1371848179 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f091802ff10 msgr2=0x7f09180323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:10:57.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:57.831+0000 7f093556d700 1 --2- 172.21.15.155:0/1371848179 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f091802ff10 0x7f09180323d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f09240096a0 tx=0x7f0924006b40).stop 2024-02-10T09:10:57.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:57.831+0000 7f093556d700 1 -- 172.21.15.155:0/1371848179 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09300f62a0 msgr2=0x7f09301065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:10:57.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:57.831+0000 7f093556d700 1 --2- 172.21.15.155:0/1371848179 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09300f62a0 0x7f09301065d0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f09200089f0 tx=0x7f092000b040).stop 2024-02-10T09:10:57.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:57.831+0000 7f093556d700 1 -- 172.21.15.155:0/1371848179 shutdown_connections 2024-02-10T09:10:57.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:57.831+0000 7f093556d700 1 --2- 172.21.15.155:0/1371848179 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f091802ff10 0x7f09180323d0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:10:57.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:57.831+0000 7f093556d700 1 --2- 172.21.15.155:0/1371848179 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f09300f62a0 0x7f09301065d0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:10:57.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:57.831+0000 7f093556d700 1 -- 172.21.15.155:0/1371848179 >> 172.21.15.155:0/1371848179 conn(0x7f09300f1aa0 msgr2=0x7f09300f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:10:57.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:57.831+0000 7f093556d700 1 -- 172.21.15.155:0/1371848179 shutdown_connections 2024-02-10T09:10:57.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:10:57.831+0000 7f093556d700 1 -- 172.21.15.155:0/1371848179 wait complete. 2024-02-10T09:10:57.833 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:10:59.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:10:58 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:10:57.750795+0000 mgr.smithi077.dxntvw (mgr.14162) 376 : cluster [DBG] pgmap v360: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:10:59.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:10:58 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:10:57.830724+0000 mon.smithi077 (mon.0) 328 : audit [DBG] from='client.? 172.21.15.155:0/1371848179' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:10:59.539 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:10:59.539 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:11:01.151 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:11:00 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:10:59.751172+0000 mgr.smithi077.dxntvw (mgr.14162) 377 : cluster [DBG] pgmap v361: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:11:01.427 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:01.423+0000 7fdd5a38b700 1 -- 172.21.15.155:0/4275753208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd540601e0 msgr2=0x7fdd540f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:11:01.427 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:01.423+0000 7fdd5a38b700 1 --2- 172.21.15.155:0/4275753208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd540601e0 0x7fdd540f9190 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7fdd44009fb0 tx=0x7fdd44009b30).stop 2024-02-10T09:11:01.427 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:01.423+0000 7fdd5a38b700 1 -- 172.21.15.155:0/4275753208 shutdown_connections 2024-02-10T09:11:01.427 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:01.423+0000 7fdd5a38b700 1 --2- 172.21.15.155:0/4275753208 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd540601e0 0x7fdd540f9190 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:11:01.428 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:01.423+0000 7fdd5a38b700 1 -- 172.21.15.155:0/4275753208 >> 172.21.15.155:0/4275753208 conn(0x7fdd540f0b70 msgr2=0x7fdd540f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:11:01.428 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:01.423+0000 7fdd5a38b700 1 -- 172.21.15.155:0/4275753208 shutdown_connections 2024-02-10T09:11:01.428 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:01.424+0000 7fdd5a38b700 1 -- 172.21.15.155:0/4275753208 wait complete. 2024-02-10T09:11:01.428 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:01.424+0000 7fdd5a38b700 1 Processor -- start 2024-02-10T09:11:01.428 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:01.424+0000 7fdd5a38b700 1 -- start start 2024-02-10T09:11:01.428 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:01.424+0000 7fdd5a38b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd540601e0 0x7fdd540f6c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:11:01.428 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:01.424+0000 7fdd5a38b700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd540f5090 con 0x7fdd540601e0 2024-02-10T09:11:01.428 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:01.424+0000 7fdd53fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd540601e0 0x7fdd540f6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:11:01.428 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:01.425+0000 7fdd53fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd540601e0 0x7fdd540f6c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:35790/0 (socket says 172.21.15.155:35790) 2024-02-10T09:11:01.428 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:01.425+0000 7fdd53fff700 1 -- 172.21.15.155:0/2131481945 learned_addr learned my addr 172.21.15.155:0/2131481945 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:11:01.428 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:01.425+0000 7fdd53fff700 1 -- 172.21.15.155:0/2131481945 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd44005040 con 0x7fdd540601e0 2024-02-10T09:11:01.428 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:01.425+0000 7fdd53fff700 1 --2- 172.21.15.155:0/2131481945 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd540601e0 0x7fdd540f6c10 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7fdd44007a10 tx=0x7fdd44009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:11:01.429 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:01.425+0000 7fdd50ff9700 1 -- 172.21.15.155:0/2131481945 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd44018070 con 0x7fdd540601e0 2024-02-10T09:11:01.429 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:01.425+0000 7fdd5a38b700 1 -- 172.21.15.155:0/2131481945 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd540f5230 con 0x7fdd540601e0 2024-02-10T09:11:01.429 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:01.425+0000 7fdd50ff9700 1 -- 172.21.15.155:0/2131481945 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdd44013410 con 0x7fdd540601e0 2024-02-10T09:11:01.429 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:01.425+0000 7fdd50ff9700 1 -- 172.21.15.155:0/2131481945 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd4401c3c0 con 0x7fdd540601e0 2024-02-10T09:11:01.429 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:01.425+0000 7fdd5a38b700 1 -- 172.21.15.155:0/2131481945 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd540f56c0 con 0x7fdd540601e0 2024-02-10T09:11:01.430 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:01.426+0000 7fdd50ff9700 1 -- 172.21.15.155:0/2131481945 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdd44013ca0 con 0x7fdd540601e0 2024-02-10T09:11:01.430 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:01.426+0000 7fdd5a38b700 1 -- 172.21.15.155:0/2131481945 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd54059100 con 0x7fdd540601e0 2024-02-10T09:11:01.430 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:01.426+0000 7fdd50ff9700 1 --2- 172.21.15.155:0/2131481945 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fdd3c02ff60 0x7fdd3c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:11:01.430 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:01.426+0000 7fdd50ff9700 1 -- 172.21.15.155:0/2131481945 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdd440414b0 con 0x7fdd540601e0 2024-02-10T09:11:01.430 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:01.426+0000 7fdd537fe700 1 --2- 172.21.15.155:0/2131481945 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fdd3c02ff60 0x7fdd3c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:11:01.430 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:01.427+0000 7fdd537fe700 1 --2- 172.21.15.155:0/2131481945 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fdd3c02ff60 0x7fdd3c032420 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fdd480096a0 tx=0x7fdd48006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:11:01.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:01.431+0000 7fdd50ff9700 1 -- 172.21.15.155:0/2131481945 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdd4401d420 con 0x7fdd540601e0 2024-02-10T09:11:01.637 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:01.635+0000 7fdd5a38b700 1 -- 172.21.15.155:0/2131481945 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdd540fc0a0 con 0x7fdd540601e0 2024-02-10T09:11:01.637 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:01.636+0000 7fdd50ff9700 1 -- 172.21.15.155:0/2131481945 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdd44025ce0 con 0x7fdd540601e0 2024-02-10T09:11:01.637 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:11:01.638 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:11:01.639 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:01.638+0000 7fdd5a38b700 1 -- 172.21.15.155:0/2131481945 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fdd3c02ff60 msgr2=0x7fdd3c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:11:01.639 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:01.638+0000 7fdd5a38b700 1 --2- 172.21.15.155:0/2131481945 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fdd3c02ff60 0x7fdd3c032420 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fdd480096a0 tx=0x7fdd48006b40).stop 2024-02-10T09:11:01.639 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:01.638+0000 7fdd5a38b700 1 -- 172.21.15.155:0/2131481945 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd540601e0 msgr2=0x7fdd540f6c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:11:01.639 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:01.638+0000 7fdd5a38b700 1 --2- 172.21.15.155:0/2131481945 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd540601e0 0x7fdd540f6c10 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7fdd44007a10 tx=0x7fdd44009b30).stop 2024-02-10T09:11:01.639 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:01.638+0000 7fdd5a38b700 1 -- 172.21.15.155:0/2131481945 shutdown_connections 2024-02-10T09:11:01.639 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:01.638+0000 7fdd5a38b700 1 --2- 172.21.15.155:0/2131481945 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fdd3c02ff60 0x7fdd3c032420 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:11:01.639 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:01.638+0000 7fdd5a38b700 1 --2- 172.21.15.155:0/2131481945 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdd540601e0 0x7fdd540f6c10 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:11:01.640 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:01.638+0000 7fdd5a38b700 1 -- 172.21.15.155:0/2131481945 >> 172.21.15.155:0/2131481945 conn(0x7fdd540f0b70 msgr2=0x7fdd540f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:11:01.640 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:01.638+0000 7fdd5a38b700 1 -- 172.21.15.155:0/2131481945 shutdown_connections 2024-02-10T09:11:01.640 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:01.638+0000 7fdd5a38b700 1 -- 172.21.15.155:0/2131481945 wait complete. 2024-02-10T09:11:01.640 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:11:02.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:11:01 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:11:01.637686+0000 mon.smithi077 (mon.0) 329 : audit [DBG] from='client.? 172.21.15.155:0/2131481945' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:11:03.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:11:02 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:11:01.751540+0000 mgr.smithi077.dxntvw (mgr.14162) 378 : cluster [DBG] pgmap v362: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:11:03.277 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:11:03.277 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:11:05.105 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:05.099+0000 7f3e0d800700 1 -- 172.21.15.155:0/1399123922 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e080f62c0 msgr2=0x7f3e080f66e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:11:05.106 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:05.099+0000 7f3e0d800700 1 --2- 172.21.15.155:0/1399123922 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e080f62c0 0x7f3e080f66e0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7f3dfc009fb0 tx=0x7f3dfc009b30).stop 2024-02-10T09:11:05.106 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:05.099+0000 7f3e0d800700 1 -- 172.21.15.155:0/1399123922 shutdown_connections 2024-02-10T09:11:05.107 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:05.099+0000 7f3e0d800700 1 --2- 172.21.15.155:0/1399123922 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e080f62c0 0x7f3e080f66e0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:11:05.107 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:05.099+0000 7f3e0d800700 1 -- 172.21.15.155:0/1399123922 >> 172.21.15.155:0/1399123922 conn(0x7f3e080f1a20 msgr2=0x7f3e080f3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:11:05.107 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:05.099+0000 7f3e0d800700 1 -- 172.21.15.155:0/1399123922 shutdown_connections 2024-02-10T09:11:05.107 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:05.099+0000 7f3e0d800700 1 -- 172.21.15.155:0/1399123922 wait complete. 2024-02-10T09:11:05.107 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:05.100+0000 7f3e0d800700 1 Processor -- start 2024-02-10T09:11:05.107 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:05.100+0000 7f3e0d800700 1 -- start start 2024-02-10T09:11:05.107 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:05.100+0000 7f3e0d800700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e080f62c0 0x7f3e08106620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:11:05.107 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:05.100+0000 7f3e0d800700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e08106b10 con 0x7f3e080f62c0 2024-02-10T09:11:05.107 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:05.100+0000 7f3e06ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e080f62c0 0x7f3e08106620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:11:05.107 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:05.100+0000 7f3e06ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e080f62c0 0x7f3e08106620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:59408/0 (socket says 172.21.15.155:59408) 2024-02-10T09:11:05.107 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:05.100+0000 7f3e06ffd700 1 -- 172.21.15.155:0/2829385552 learned_addr learned my addr 172.21.15.155:0/2829385552 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:11:05.107 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:05.100+0000 7f3e06ffd700 1 -- 172.21.15.155:0/2829385552 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dfc005040 con 0x7f3e080f62c0 2024-02-10T09:11:05.107 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:05.101+0000 7f3e06ffd700 1 --2- 172.21.15.155:0/2829385552 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e080f62c0 0x7f3e08106620 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f3dfc005010 tx=0x7f3dfc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:11:05.107 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:05.101+0000 7f3def7fe700 1 -- 172.21.15.155:0/2829385552 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3dfc009320 con 0x7f3e080f62c0 2024-02-10T09:11:05.108 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:05.101+0000 7f3e0d800700 1 -- 172.21.15.155:0/2829385552 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e08106d10 con 0x7f3e080f62c0 2024-02-10T09:11:05.108 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:05.101+0000 7f3e0d800700 1 -- 172.21.15.155:0/2829385552 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e081071a0 con 0x7f3e080f62c0 2024-02-10T09:11:05.108 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:05.101+0000 7f3def7fe700 1 -- 172.21.15.155:0/2829385552 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3dfc013380 con 0x7f3e080f62c0 2024-02-10T09:11:05.108 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:05.101+0000 7f3def7fe700 1 -- 172.21.15.155:0/2829385552 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3dfc014df0 con 0x7f3e080f62c0 2024-02-10T09:11:05.108 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:05.102+0000 7f3e0d800700 1 -- 172.21.15.155:0/2829385552 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3de8004fa0 con 0x7f3e080f62c0 2024-02-10T09:11:05.108 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:05.102+0000 7f3def7fe700 1 -- 172.21.15.155:0/2829385552 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3dfc0095f0 con 0x7f3e080f62c0 2024-02-10T09:11:05.108 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:05.102+0000 7f3def7fe700 1 --2- 172.21.15.155:0/2829385552 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f3df002ff10 0x7f3df00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:11:05.108 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:05.102+0000 7f3def7fe700 1 -- 172.21.15.155:0/2829385552 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3dfc040db0 con 0x7f3e080f62c0 2024-02-10T09:11:05.108 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:05.102+0000 7f3e067fc700 1 --2- 172.21.15.155:0/2829385552 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f3df002ff10 0x7f3df00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:11:05.108 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:05.102+0000 7f3e067fc700 1 --2- 172.21.15.155:0/2829385552 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f3df002ff10 0x7f3df00323d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f3df80096a0 tx=0x7f3df8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:11:05.109 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:05.106+0000 7f3def7fe700 1 -- 172.21.15.155:0/2829385552 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3dfc014420 con 0x7f3e080f62c0 2024-02-10T09:11:05.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:11:04 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:11:03.751930+0000 mgr.smithi077.dxntvw (mgr.14162) 379 : cluster [DBG] pgmap v363: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:11:05.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:05.311+0000 7f3e0d800700 1 -- 172.21.15.155:0/2829385552 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3de8005670 con 0x7f3e080f62c0 2024-02-10T09:11:05.313 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:05.312+0000 7f3def7fe700 1 -- 172.21.15.155:0/2829385552 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3dfc01d9d0 con 0x7f3e080f62c0 2024-02-10T09:11:05.313 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:11:05.314 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:11:05.315 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:05.313+0000 7f3e0d800700 1 -- 172.21.15.155:0/2829385552 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f3df002ff10 msgr2=0x7f3df00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:11:05.315 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:05.313+0000 7f3e0d800700 1 --2- 172.21.15.155:0/2829385552 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f3df002ff10 0x7f3df00323d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f3df80096a0 tx=0x7f3df8006b40).stop 2024-02-10T09:11:05.315 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:05.313+0000 7f3e0d800700 1 -- 172.21.15.155:0/2829385552 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e080f62c0 msgr2=0x7f3e08106620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:11:05.315 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:05.313+0000 7f3e0d800700 1 --2- 172.21.15.155:0/2829385552 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e080f62c0 0x7f3e08106620 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f3dfc005010 tx=0x7f3dfc00b040).stop 2024-02-10T09:11:05.315 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:05.313+0000 7f3e0d800700 1 -- 172.21.15.155:0/2829385552 shutdown_connections 2024-02-10T09:11:05.315 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:05.313+0000 7f3e0d800700 1 --2- 172.21.15.155:0/2829385552 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f3df002ff10 0x7f3df00323d0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:11:05.315 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:05.313+0000 7f3e0d800700 1 --2- 172.21.15.155:0/2829385552 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3e080f62c0 0x7f3e08106620 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:11:05.315 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:05.313+0000 7f3e0d800700 1 -- 172.21.15.155:0/2829385552 >> 172.21.15.155:0/2829385552 conn(0x7f3e080f1a20 msgr2=0x7f3e080f2550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:11:05.315 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:05.314+0000 7f3e0d800700 1 -- 172.21.15.155:0/2829385552 shutdown_connections 2024-02-10T09:11:05.315 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:05.314+0000 7f3e0d800700 1 -- 172.21.15.155:0/2829385552 wait complete. 2024-02-10T09:11:05.315 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:11:06.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:11:05 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:11:05.313575+0000 mon.smithi077 (mon.0) 330 : audit [DBG] from='client.? 172.21.15.155:0/2829385552' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:11:07.064 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:11:07.064 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:11:07.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:11:06 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:11:05.752351+0000 mgr.smithi077.dxntvw (mgr.14162) 380 : cluster [DBG] pgmap v364: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:11:08.913 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:08.911+0000 7f27fd088700 1 -- 172.21.15.155:0/402245072 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27f80f5410 msgr2=0x7f27f80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:11:08.913 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:08.911+0000 7f27fd088700 1 --2- 172.21.15.155:0/402245072 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27f80f5410 0x7f27f80f5830 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7f27e8009fb0 tx=0x7f27e8009b30).stop 2024-02-10T09:11:08.914 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:08.911+0000 7f27fd088700 1 -- 172.21.15.155:0/402245072 shutdown_connections 2024-02-10T09:11:08.914 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:08.911+0000 7f27fd088700 1 --2- 172.21.15.155:0/402245072 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27f80f5410 0x7f27f80f5830 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:11:08.914 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:08.911+0000 7f27fd088700 1 -- 172.21.15.155:0/402245072 >> 172.21.15.155:0/402245072 conn(0x7f27f80f0b70 msgr2=0x7f27f80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:11:08.914 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:08.911+0000 7f27fd088700 1 -- 172.21.15.155:0/402245072 shutdown_connections 2024-02-10T09:11:08.914 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:08.911+0000 7f27fd088700 1 -- 172.21.15.155:0/402245072 wait complete. 2024-02-10T09:11:08.914 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:08.912+0000 7f27fd088700 1 Processor -- start 2024-02-10T09:11:08.914 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:08.912+0000 7f27fd088700 1 -- start start 2024-02-10T09:11:08.914 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:08.912+0000 7f27fd088700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27f80f5410 0x7f27f80fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:11:08.914 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:08.912+0000 7f27fd088700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f80fb960 con 0x7f27f80f5410 2024-02-10T09:11:08.914 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:08.912+0000 7f27f6d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27f80f5410 0x7f27f80fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:11:08.914 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:08.912+0000 7f27f6d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27f80f5410 0x7f27f80fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:59436/0 (socket says 172.21.15.155:59436) 2024-02-10T09:11:08.914 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:08.912+0000 7f27f6d9d700 1 -- 172.21.15.155:0/970442994 learned_addr learned my addr 172.21.15.155:0/970442994 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:11:08.914 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:08.912+0000 7f27f6d9d700 1 -- 172.21.15.155:0/970442994 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27e8005040 con 0x7f27f80f5410 2024-02-10T09:11:08.915 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:08.912+0000 7f27f6d9d700 1 --2- 172.21.15.155:0/970442994 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27f80f5410 0x7f27f80fd4e0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f27e8005010 tx=0x7f27e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:11:08.916 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:08.913+0000 7f27ef7fe700 1 -- 172.21.15.155:0/970442994 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f27e8009320 con 0x7f27f80f5410 2024-02-10T09:11:08.916 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:08.913+0000 7f27fd088700 1 -- 172.21.15.155:0/970442994 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27f80fbb00 con 0x7f27f80f5410 2024-02-10T09:11:08.916 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:08.913+0000 7f27ef7fe700 1 -- 172.21.15.155:0/970442994 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f27e8013380 con 0x7f27f80f5410 2024-02-10T09:11:08.916 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:08.913+0000 7f27ef7fe700 1 -- 172.21.15.155:0/970442994 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f27e8014e50 con 0x7f27f80f5410 2024-02-10T09:11:08.916 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:08.913+0000 7f27fd088700 1 -- 172.21.15.155:0/970442994 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27f80fbf90 con 0x7f27f80f5410 2024-02-10T09:11:08.916 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:08.913+0000 7f27ef7fe700 1 -- 172.21.15.155:0/970442994 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f27e80263a0 con 0x7f27f80f5410 2024-02-10T09:11:08.916 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:08.914+0000 7f27ef7fe700 1 --2- 172.21.15.155:0/970442994 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f27e402ff10 0x7f27e40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:11:08.917 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:08.914+0000 7f27ef7fe700 1 -- 172.21.15.155:0/970442994 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f27e8040f30 con 0x7f27f80f5410 2024-02-10T09:11:08.917 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:08.914+0000 7f27fd088700 1 -- 172.21.15.155:0/970442994 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27dc004fa0 con 0x7f27f80f5410 2024-02-10T09:11:08.917 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:08.914+0000 7f27f659c700 1 --2- 172.21.15.155:0/970442994 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f27e402ff10 0x7f27e40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:11:08.917 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:08.914+0000 7f27f659c700 1 --2- 172.21.15.155:0/970442994 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f27e402ff10 0x7f27e40323d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f27e00096a0 tx=0x7f27e0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:11:08.919 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:08.919+0000 7f27ef7fe700 1 -- 172.21.15.155:0/970442994 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f27e8012920 con 0x7f27f80f5410 2024-02-10T09:11:09.123 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:09.122+0000 7f27fd088700 1 -- 172.21.15.155:0/970442994 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f27dc005670 con 0x7f27f80f5410 2024-02-10T09:11:09.124 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:09.123+0000 7f27ef7fe700 1 -- 172.21.15.155:0/970442994 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f27e802b020 con 0x7f27f80f5410 2024-02-10T09:11:09.124 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:11:09.124 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:11:09.126 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:09.125+0000 7f27fd088700 1 -- 172.21.15.155:0/970442994 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f27e402ff10 msgr2=0x7f27e40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:11:09.126 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:09.125+0000 7f27fd088700 1 --2- 172.21.15.155:0/970442994 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f27e402ff10 0x7f27e40323d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f27e00096a0 tx=0x7f27e0006b40).stop 2024-02-10T09:11:09.126 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:09.125+0000 7f27fd088700 1 -- 172.21.15.155:0/970442994 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27f80f5410 msgr2=0x7f27f80fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:11:09.126 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:09.125+0000 7f27fd088700 1 --2- 172.21.15.155:0/970442994 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27f80f5410 0x7f27f80fd4e0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f27e8005010 tx=0x7f27e800b040).stop 2024-02-10T09:11:09.126 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:09.125+0000 7f27fd088700 1 -- 172.21.15.155:0/970442994 shutdown_connections 2024-02-10T09:11:09.126 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:09.125+0000 7f27fd088700 1 --2- 172.21.15.155:0/970442994 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f27e402ff10 0x7f27e40323d0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:11:09.126 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:09.125+0000 7f27fd088700 1 --2- 172.21.15.155:0/970442994 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f27f80f5410 0x7f27f80fd4e0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:11:09.126 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:09.125+0000 7f27fd088700 1 -- 172.21.15.155:0/970442994 >> 172.21.15.155:0/970442994 conn(0x7f27f80f0b70 msgr2=0x7f27f805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:11:09.126 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:09.125+0000 7f27fd088700 1 -- 172.21.15.155:0/970442994 shutdown_connections 2024-02-10T09:11:09.126 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:09.125+0000 7f27fd088700 1 -- 172.21.15.155:0/970442994 wait complete. 2024-02-10T09:11:09.127 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:11:09.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:11:08 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:11:07.752699+0000 mgr.smithi077.dxntvw (mgr.14162) 381 : cluster [DBG] pgmap v365: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:11:10.121 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:11:09 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:11:09.124489+0000 mon.smithi077 (mon.0) 331 : audit [DBG] from='client.? 172.21.15.155:0/970442994' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:11:10.866 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:11:10.866 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:11:11.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:11:10 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:11:09.753002+0000 mgr.smithi077.dxntvw (mgr.14162) 382 : cluster [DBG] pgmap v366: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:11:12.795 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:12.793+0000 7fdbd481f700 1 -- 172.21.15.155:0/3390213190 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbd00f5410 msgr2=0x7fdbd00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:11:12.795 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:12.793+0000 7fdbd481f700 1 --2- 172.21.15.155:0/3390213190 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbd00f5410 0x7fdbd00f5830 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7fdbc4009fb0 tx=0x7fdbc4009b30).stop 2024-02-10T09:11:12.795 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:12.793+0000 7fdbd481f700 1 -- 172.21.15.155:0/3390213190 shutdown_connections 2024-02-10T09:11:12.795 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:12.793+0000 7fdbd481f700 1 --2- 172.21.15.155:0/3390213190 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbd00f5410 0x7fdbd00f5830 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:11:12.795 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:12.793+0000 7fdbd481f700 1 -- 172.21.15.155:0/3390213190 >> 172.21.15.155:0/3390213190 conn(0x7fdbd00f0b70 msgr2=0x7fdbd00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:11:12.796 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:12.793+0000 7fdbd481f700 1 -- 172.21.15.155:0/3390213190 shutdown_connections 2024-02-10T09:11:12.796 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:12.793+0000 7fdbd481f700 1 -- 172.21.15.155:0/3390213190 wait complete. 2024-02-10T09:11:12.796 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:12.794+0000 7fdbd481f700 1 Processor -- start 2024-02-10T09:11:12.796 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:12.794+0000 7fdbd481f700 1 -- start start 2024-02-10T09:11:12.796 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:12.794+0000 7fdbd481f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbd00f5410 0x7fdbd01067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:11:12.796 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:12.794+0000 7fdbd481f700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbd0106ca0 con 0x7fdbd00f5410 2024-02-10T09:11:12.796 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:12.794+0000 7fdbce59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbd00f5410 0x7fdbd01067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:11:12.796 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:12.794+0000 7fdbce59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbd00f5410 0x7fdbd01067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:56944/0 (socket says 172.21.15.155:56944) 2024-02-10T09:11:12.796 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:12.794+0000 7fdbce59c700 1 -- 172.21.15.155:0/3066913429 learned_addr learned my addr 172.21.15.155:0/3066913429 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:11:12.796 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:12.794+0000 7fdbce59c700 1 -- 172.21.15.155:0/3066913429 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbc4005040 con 0x7fdbd00f5410 2024-02-10T09:11:12.796 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:12.795+0000 7fdbce59c700 1 --2- 172.21.15.155:0/3066913429 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbd00f5410 0x7fdbd01067b0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7fdbc4005010 tx=0x7fdbc400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:11:12.796 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:12.795+0000 7fdbbeffd700 1 -- 172.21.15.155:0/3066913429 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdbc4009320 con 0x7fdbd00f5410 2024-02-10T09:11:12.796 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:12.795+0000 7fdbd481f700 1 -- 172.21.15.155:0/3066913429 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbd0106ea0 con 0x7fdbd00f5410 2024-02-10T09:11:12.797 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:12.795+0000 7fdbbeffd700 1 -- 172.21.15.155:0/3066913429 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdbc4013380 con 0x7fdbd00f5410 2024-02-10T09:11:12.797 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:12.795+0000 7fdbbeffd700 1 -- 172.21.15.155:0/3066913429 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdbc4014e50 con 0x7fdbd00f5410 2024-02-10T09:11:12.797 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:12.795+0000 7fdbd481f700 1 -- 172.21.15.155:0/3066913429 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbd0107330 con 0x7fdbd00f5410 2024-02-10T09:11:12.798 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:12.796+0000 7fdbd481f700 1 -- 172.21.15.155:0/3066913429 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbd0045ad0 con 0x7fdbd00f5410 2024-02-10T09:11:12.798 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:12.796+0000 7fdbbeffd700 1 -- 172.21.15.155:0/3066913429 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdbc4014420 con 0x7fdbd00f5410 2024-02-10T09:11:12.798 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:12.796+0000 7fdbbeffd700 1 --2- 172.21.15.155:0/3066913429 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fdbb802ff10 0x7fdbb80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:11:12.798 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:12.796+0000 7fdbbeffd700 1 -- 172.21.15.155:0/3066913429 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdbc4040f70 con 0x7fdbd00f5410 2024-02-10T09:11:12.798 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:12.796+0000 7fdbcdd9b700 1 --2- 172.21.15.155:0/3066913429 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fdbb802ff10 0x7fdbb80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:11:12.798 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:12.796+0000 7fdbcdd9b700 1 --2- 172.21.15.155:0/3066913429 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fdbb802ff10 0x7fdbb80323d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fdbc00096a0 tx=0x7fdbc0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:11:12.800 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:12.800+0000 7fdbbeffd700 1 -- 172.21.15.155:0/3066913429 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdbc4014740 con 0x7fdbd00f5410 2024-02-10T09:11:13.004 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:13.004+0000 7fdbd481f700 1 -- 172.21.15.155:0/3066913429 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdbd0108180 con 0x7fdbd00f5410 2024-02-10T09:11:13.005 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:13.004+0000 7fdbbeffd700 1 -- 172.21.15.155:0/3066913429 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdbc4010230 con 0x7fdbd00f5410 2024-02-10T09:11:13.005 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:11:13.005 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:11:13.006 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:13.006+0000 7fdbd481f700 1 -- 172.21.15.155:0/3066913429 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fdbb802ff10 msgr2=0x7fdbb80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:11:13.006 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:13.006+0000 7fdbd481f700 1 --2- 172.21.15.155:0/3066913429 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fdbb802ff10 0x7fdbb80323d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fdbc00096a0 tx=0x7fdbc0006b40).stop 2024-02-10T09:11:13.006 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:13.006+0000 7fdbd481f700 1 -- 172.21.15.155:0/3066913429 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbd00f5410 msgr2=0x7fdbd01067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:11:13.006 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:13.006+0000 7fdbd481f700 1 --2- 172.21.15.155:0/3066913429 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbd00f5410 0x7fdbd01067b0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7fdbc4005010 tx=0x7fdbc400b040).stop 2024-02-10T09:11:13.006 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:13.006+0000 7fdbd481f700 1 -- 172.21.15.155:0/3066913429 shutdown_connections 2024-02-10T09:11:13.006 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:13.006+0000 7fdbd481f700 1 --2- 172.21.15.155:0/3066913429 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fdbb802ff10 0x7fdbb80323d0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:11:13.006 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:13.006+0000 7fdbd481f700 1 --2- 172.21.15.155:0/3066913429 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdbd00f5410 0x7fdbd01067b0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:11:13.007 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:13.006+0000 7fdbd481f700 1 -- 172.21.15.155:0/3066913429 >> 172.21.15.155:0/3066913429 conn(0x7fdbd00f0b70 msgr2=0x7fdbd005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:11:13.007 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:13.006+0000 7fdbd481f700 1 -- 172.21.15.155:0/3066913429 shutdown_connections 2024-02-10T09:11:13.007 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:13.006+0000 7fdbd481f700 1 -- 172.21.15.155:0/3066913429 wait complete. 2024-02-10T09:11:13.007 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:11:13.148 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:11:12 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:11:11.753257+0000 mgr.smithi077.dxntvw (mgr.14162) 383 : cluster [DBG] pgmap v367: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:11:14.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:11:13 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:11:13.005057+0000 mon.smithi077 (mon.0) 332 : audit [DBG] from='client.? 172.21.15.155:0/3066913429' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:11:14.723 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:11:14.724 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:11:15.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:11:14 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:11:13.753531+0000 mgr.smithi077.dxntvw (mgr.14162) 384 : cluster [DBG] pgmap v368: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:11:16.339 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:16.337+0000 7f551d3d7700 1 -- 172.21.15.155:0/1545196526 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55180f5410 msgr2=0x7f55180f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:11:16.339 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:16.337+0000 7f551d3d7700 1 --2- 172.21.15.155:0/1545196526 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55180f5410 0x7f55180f5830 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7f550c009fb0 tx=0x7f550c009b30).stop 2024-02-10T09:11:16.339 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:16.337+0000 7f551d3d7700 1 -- 172.21.15.155:0/1545196526 shutdown_connections 2024-02-10T09:11:16.339 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:16.337+0000 7f551d3d7700 1 --2- 172.21.15.155:0/1545196526 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55180f5410 0x7f55180f5830 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:11:16.339 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:16.337+0000 7f551d3d7700 1 -- 172.21.15.155:0/1545196526 >> 172.21.15.155:0/1545196526 conn(0x7f55180f0b70 msgr2=0x7f55180f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:11:16.339 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:16.337+0000 7f551d3d7700 1 -- 172.21.15.155:0/1545196526 shutdown_connections 2024-02-10T09:11:16.339 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:16.337+0000 7f551d3d7700 1 -- 172.21.15.155:0/1545196526 wait complete. 2024-02-10T09:11:16.339 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:16.338+0000 7f551d3d7700 1 Processor -- start 2024-02-10T09:11:16.339 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:16.338+0000 7f551d3d7700 1 -- start start 2024-02-10T09:11:16.339 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:16.338+0000 7f551d3d7700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55180f5410 0x7f55181067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:11:16.340 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:16.338+0000 7f551d3d7700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5518106cc0 con 0x7f55180f5410 2024-02-10T09:11:16.340 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:16.338+0000 7f5516ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55180f5410 0x7f55181067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:11:16.340 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:16.338+0000 7f5516ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55180f5410 0x7f55181067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:56958/0 (socket says 172.21.15.155:56958) 2024-02-10T09:11:16.340 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:16.338+0000 7f5516ffd700 1 -- 172.21.15.155:0/2336953416 learned_addr learned my addr 172.21.15.155:0/2336953416 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:11:16.340 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:16.338+0000 7f5516ffd700 1 -- 172.21.15.155:0/2336953416 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f550c005040 con 0x7f55180f5410 2024-02-10T09:11:16.340 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:16.338+0000 7f5516ffd700 1 --2- 172.21.15.155:0/2336953416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55180f5410 0x7f55181067d0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f550c005010 tx=0x7f550c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:11:16.340 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:16.339+0000 7f54ff7fe700 1 -- 172.21.15.155:0/2336953416 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f550c009320 con 0x7f55180f5410 2024-02-10T09:11:16.340 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:16.339+0000 7f551d3d7700 1 -- 172.21.15.155:0/2336953416 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5518106ec0 con 0x7f55180f5410 2024-02-10T09:11:16.340 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:16.339+0000 7f54ff7fe700 1 -- 172.21.15.155:0/2336953416 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f550c013380 con 0x7f55180f5410 2024-02-10T09:11:16.340 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:16.339+0000 7f54ff7fe700 1 -- 172.21.15.155:0/2336953416 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f550c014e50 con 0x7f55180f5410 2024-02-10T09:11:16.340 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:16.339+0000 7f551d3d7700 1 -- 172.21.15.155:0/2336953416 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5518107350 con 0x7f55180f5410 2024-02-10T09:11:16.341 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:16.339+0000 7f54ff7fe700 1 -- 172.21.15.155:0/2336953416 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f550c014420 con 0x7f55180f5410 2024-02-10T09:11:16.341 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:16.340+0000 7f551d3d7700 1 -- 172.21.15.155:0/2336953416 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5518045ad0 con 0x7f55180f5410 2024-02-10T09:11:16.341 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:16.340+0000 7f54ff7fe700 1 --2- 172.21.15.155:0/2336953416 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f550002ff10 0x7f55000323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:11:16.341 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:16.340+0000 7f54ff7fe700 1 -- 172.21.15.155:0/2336953416 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f550c040f70 con 0x7f55180f5410 2024-02-10T09:11:16.342 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:16.340+0000 7f55167fc700 1 --2- 172.21.15.155:0/2336953416 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f550002ff10 0x7f55000323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:11:16.342 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:16.340+0000 7f55167fc700 1 --2- 172.21.15.155:0/2336953416 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f550002ff10 0x7f55000323d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f55080096a0 tx=0x7f5508006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:11:16.344 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:16.344+0000 7f54ff7fe700 1 -- 172.21.15.155:0/2336953416 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f550c0095f0 con 0x7f55180f5410 2024-02-10T09:11:16.547 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:16.547+0000 7f551d3d7700 1 -- 172.21.15.155:0/2336953416 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f55181081a0 con 0x7f55180f5410 2024-02-10T09:11:16.547 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:16.547+0000 7f54ff7fe700 1 -- 172.21.15.155:0/2336953416 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f550c01db70 con 0x7f55180f5410 2024-02-10T09:11:16.548 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:11:16.548 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:11:16.549 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:16.549+0000 7f551d3d7700 1 -- 172.21.15.155:0/2336953416 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f550002ff10 msgr2=0x7f55000323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:11:16.549 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:16.549+0000 7f551d3d7700 1 --2- 172.21.15.155:0/2336953416 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f550002ff10 0x7f55000323d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f55080096a0 tx=0x7f5508006b40).stop 2024-02-10T09:11:16.550 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:16.549+0000 7f551d3d7700 1 -- 172.21.15.155:0/2336953416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55180f5410 msgr2=0x7f55181067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:11:16.550 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:16.549+0000 7f551d3d7700 1 --2- 172.21.15.155:0/2336953416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55180f5410 0x7f55181067d0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f550c005010 tx=0x7f550c00b040).stop 2024-02-10T09:11:16.550 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:16.549+0000 7f551d3d7700 1 -- 172.21.15.155:0/2336953416 shutdown_connections 2024-02-10T09:11:16.550 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:16.549+0000 7f551d3d7700 1 --2- 172.21.15.155:0/2336953416 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f550002ff10 0x7f55000323d0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:11:16.550 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:16.549+0000 7f551d3d7700 1 --2- 172.21.15.155:0/2336953416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f55180f5410 0x7f55181067d0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:11:16.550 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:16.549+0000 7f551d3d7700 1 -- 172.21.15.155:0/2336953416 >> 172.21.15.155:0/2336953416 conn(0x7f55180f0b70 msgr2=0x7f551805fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:11:16.550 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:16.549+0000 7f551d3d7700 1 -- 172.21.15.155:0/2336953416 shutdown_connections 2024-02-10T09:11:16.550 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:16.549+0000 7f551d3d7700 1 -- 172.21.15.155:0/2336953416 wait complete. 2024-02-10T09:11:16.550 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:11:17.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:11:16 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:11:15.753786+0000 mgr.smithi077.dxntvw (mgr.14162) 385 : cluster [DBG] pgmap v369: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:11:17.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:11:16 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:11:16.548081+0000 mon.smithi077 (mon.0) 333 : audit [DBG] from='client.? 172.21.15.155:0/2336953416' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:11:18.257 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:11:18.258 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:11:19.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:11:18 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:11:17.754149+0000 mgr.smithi077.dxntvw (mgr.14162) 386 : cluster [DBG] pgmap v370: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:11:20.164 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:20.164+0000 7f7fbd4a8700 1 -- 172.21.15.155:0/3086713531 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fb80f47c0 msgr2=0x7f7fb80f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:11:20.164 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:20.164+0000 7f7fbd4a8700 1 --2- 172.21.15.155:0/3086713531 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fb80f47c0 0x7f7fb80f4be0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7f7fac009fb0 tx=0x7f7fac009b30).stop 2024-02-10T09:11:20.164 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:20.164+0000 7f7fbd4a8700 1 -- 172.21.15.155:0/3086713531 shutdown_connections 2024-02-10T09:11:20.165 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:20.164+0000 7f7fbd4a8700 1 --2- 172.21.15.155:0/3086713531 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fb80f47c0 0x7f7fb80f4be0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:11:20.165 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:20.164+0000 7f7fbd4a8700 1 -- 172.21.15.155:0/3086713531 >> 172.21.15.155:0/3086713531 conn(0x7f7fb80f02e0 msgr2=0x7f7fb80f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:11:20.165 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:20.164+0000 7f7fbd4a8700 1 -- 172.21.15.155:0/3086713531 shutdown_connections 2024-02-10T09:11:20.165 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:20.164+0000 7f7fbd4a8700 1 -- 172.21.15.155:0/3086713531 wait complete. 2024-02-10T09:11:20.167 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:20.165+0000 7f7fbd4a8700 1 Processor -- start 2024-02-10T09:11:20.167 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:20.165+0000 7f7fbd4a8700 1 -- start start 2024-02-10T09:11:20.167 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:20.165+0000 7f7fbd4a8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fb80f47c0 0x7f7fb81067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:11:20.167 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:20.165+0000 7f7fbd4a8700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb8106cc0 con 0x7f7fb80f47c0 2024-02-10T09:11:20.167 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:20.165+0000 7f7fb6ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fb80f47c0 0x7f7fb81067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:11:20.167 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:20.165+0000 7f7fb6ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fb80f47c0 0x7f7fb81067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:56976/0 (socket says 172.21.15.155:56976) 2024-02-10T09:11:20.167 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:20.165+0000 7f7fb6ffd700 1 -- 172.21.15.155:0/1773362853 learned_addr learned my addr 172.21.15.155:0/1773362853 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:11:20.167 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:20.165+0000 7f7fb6ffd700 1 -- 172.21.15.155:0/1773362853 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fac005040 con 0x7f7fb80f47c0 2024-02-10T09:11:20.167 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:20.166+0000 7f7fb6ffd700 1 --2- 172.21.15.155:0/1773362853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fb80f47c0 0x7f7fb81067d0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f7fac0089f0 tx=0x7f7fac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:11:20.168 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:20.166+0000 7f7f9f7fe700 1 -- 172.21.15.155:0/1773362853 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7fac009320 con 0x7f7fb80f47c0 2024-02-10T09:11:20.168 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:20.166+0000 7f7f9f7fe700 1 -- 172.21.15.155:0/1773362853 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7fac013380 con 0x7f7fb80f47c0 2024-02-10T09:11:20.168 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:20.166+0000 7f7fbd4a8700 1 -- 172.21.15.155:0/1773362853 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fb8106ec0 con 0x7f7fb80f47c0 2024-02-10T09:11:20.168 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:20.166+0000 7f7f9f7fe700 1 -- 172.21.15.155:0/1773362853 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7fac01ed80 con 0x7f7fb80f47c0 2024-02-10T09:11:20.168 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:20.166+0000 7f7fbd4a8700 1 -- 172.21.15.155:0/1773362853 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fb8107350 con 0x7f7fb80f47c0 2024-02-10T09:11:20.168 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:20.166+0000 7f7f9f7fe700 1 -- 172.21.15.155:0/1773362853 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7fac01e420 con 0x7f7fb80f47c0 2024-02-10T09:11:20.168 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:20.167+0000 7f7fbd4a8700 1 -- 172.21.15.155:0/1773362853 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fb8045ad0 con 0x7f7fb80f47c0 2024-02-10T09:11:20.168 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:20.167+0000 7f7f9f7fe700 1 --2- 172.21.15.155:0/1773362853 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f7fa002ff60 0x7f7fa0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:11:20.168 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:20.167+0000 7f7f9f7fe700 1 -- 172.21.15.155:0/1773362853 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7fac040fc0 con 0x7f7fb80f47c0 2024-02-10T09:11:20.168 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:20.167+0000 7f7fb67fc700 1 --2- 172.21.15.155:0/1773362853 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f7fa002ff60 0x7f7fa0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:11:20.169 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:20.167+0000 7f7fb67fc700 1 --2- 172.21.15.155:0/1773362853 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f7fa002ff60 0x7f7fa0032420 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f7fa80096a0 tx=0x7f7fa8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:11:20.172 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:20.172+0000 7f7f9f7fe700 1 -- 172.21.15.155:0/1773362853 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7fac06d2b0 con 0x7f7fb80f47c0 2024-02-10T09:11:20.375 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:20.375+0000 7f7fbd4a8700 1 -- 172.21.15.155:0/1773362853 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7fb81081a0 con 0x7f7fb80f47c0 2024-02-10T09:11:20.376 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:20.376+0000 7f7f9f7fe700 1 -- 172.21.15.155:0/1773362853 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7fac02c2d0 con 0x7f7fb80f47c0 2024-02-10T09:11:20.376 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:11:20.376 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:11:20.377 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:20.377+0000 7f7fbd4a8700 1 -- 172.21.15.155:0/1773362853 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f7fa002ff60 msgr2=0x7f7fa0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:11:20.377 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:20.377+0000 7f7fbd4a8700 1 --2- 172.21.15.155:0/1773362853 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f7fa002ff60 0x7f7fa0032420 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f7fa80096a0 tx=0x7f7fa8006b40).stop 2024-02-10T09:11:20.378 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:20.377+0000 7f7fbd4a8700 1 -- 172.21.15.155:0/1773362853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fb80f47c0 msgr2=0x7f7fb81067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:11:20.378 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:20.377+0000 7f7fbd4a8700 1 --2- 172.21.15.155:0/1773362853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fb80f47c0 0x7f7fb81067d0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f7fac0089f0 tx=0x7f7fac00b040).stop 2024-02-10T09:11:20.378 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:20.377+0000 7f7fbd4a8700 1 -- 172.21.15.155:0/1773362853 shutdown_connections 2024-02-10T09:11:20.378 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:20.377+0000 7f7fbd4a8700 1 --2- 172.21.15.155:0/1773362853 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f7fa002ff60 0x7f7fa0032420 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:11:20.378 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:20.377+0000 7f7fbd4a8700 1 --2- 172.21.15.155:0/1773362853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7fb80f47c0 0x7f7fb81067d0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:11:20.378 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:20.377+0000 7f7fbd4a8700 1 -- 172.21.15.155:0/1773362853 >> 172.21.15.155:0/1773362853 conn(0x7f7fb80f02e0 msgr2=0x7f7fb80f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:11:20.378 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:20.377+0000 7f7fbd4a8700 1 -- 172.21.15.155:0/1773362853 shutdown_connections 2024-02-10T09:11:20.378 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:20.377+0000 7f7fbd4a8700 1 -- 172.21.15.155:0/1773362853 wait complete. 2024-02-10T09:11:20.378 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:11:21.148 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:11:20 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:11:19.754531+0000 mgr.smithi077.dxntvw (mgr.14162) 387 : cluster [DBG] pgmap v371: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:11:21.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:11:20 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:11:20.376197+0000 mon.smithi077 (mon.0) 334 : audit [DBG] from='client.? 172.21.15.155:0/1773362853' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:11:22.139 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:11:22.139 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:11:23.148 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:11:22 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:11:21.754838+0000 mgr.smithi077.dxntvw (mgr.14162) 388 : cluster [DBG] pgmap v372: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:11:23.964 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:23.962+0000 7f9c3bfe2700 1 -- 172.21.15.155:0/1160290245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c340f5410 msgr2=0x7f9c340f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:11:23.964 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:23.962+0000 7f9c3bfe2700 1 --2- 172.21.15.155:0/1160290245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c340f5410 0x7f9c340f5830 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7f9c24009fb0 tx=0x7f9c24009b30).stop 2024-02-10T09:11:23.964 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:23.962+0000 7f9c3bfe2700 1 -- 172.21.15.155:0/1160290245 shutdown_connections 2024-02-10T09:11:23.964 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:23.962+0000 7f9c3bfe2700 1 --2- 172.21.15.155:0/1160290245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c340f5410 0x7f9c340f5830 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:11:23.965 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:23.962+0000 7f9c3bfe2700 1 -- 172.21.15.155:0/1160290245 >> 172.21.15.155:0/1160290245 conn(0x7f9c340f0b70 msgr2=0x7f9c340f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:11:23.965 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:23.962+0000 7f9c3bfe2700 1 -- 172.21.15.155:0/1160290245 shutdown_connections 2024-02-10T09:11:23.965 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:23.962+0000 7f9c3bfe2700 1 -- 172.21.15.155:0/1160290245 wait complete. 2024-02-10T09:11:23.965 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:23.963+0000 7f9c3bfe2700 1 Processor -- start 2024-02-10T09:11:23.965 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:23.963+0000 7f9c3bfe2700 1 -- start start 2024-02-10T09:11:23.965 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:23.963+0000 7f9c3bfe2700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c340f5410 0x7f9c34109530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:11:23.965 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:23.963+0000 7f9c3bfe2700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c34060220 con 0x7f9c340f5410 2024-02-10T09:11:23.965 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:23.963+0000 7f9c39d7e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c340f5410 0x7f9c34109530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:11:23.965 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:23.963+0000 7f9c39d7e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c340f5410 0x7f9c34109530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:45320/0 (socket says 172.21.15.155:45320) 2024-02-10T09:11:23.965 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:23.963+0000 7f9c39d7e700 1 -- 172.21.15.155:0/1411699313 learned_addr learned my addr 172.21.15.155:0/1411699313 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:11:23.965 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:23.963+0000 7f9c39d7e700 1 -- 172.21.15.155:0/1411699313 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c24005040 con 0x7f9c340f5410 2024-02-10T09:11:23.966 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:23.964+0000 7f9c39d7e700 1 --2- 172.21.15.155:0/1411699313 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c340f5410 0x7f9c34109530 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f9c24005010 tx=0x7f9c2400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:11:23.966 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:23.964+0000 7f9c2a7fc700 1 -- 172.21.15.155:0/1411699313 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c24009320 con 0x7f9c340f5410 2024-02-10T09:11:23.966 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:23.964+0000 7f9c3bfe2700 1 -- 172.21.15.155:0/1411699313 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c34060420 con 0x7f9c340f5410 2024-02-10T09:11:23.966 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:23.964+0000 7f9c2a7fc700 1 -- 172.21.15.155:0/1411699313 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9c24013380 con 0x7f9c340f5410 2024-02-10T09:11:23.966 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:23.964+0000 7f9c2a7fc700 1 -- 172.21.15.155:0/1411699313 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c24014e50 con 0x7f9c340f5410 2024-02-10T09:11:23.966 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:23.964+0000 7f9c3bfe2700 1 -- 172.21.15.155:0/1411699313 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c34109be0 con 0x7f9c340f5410 2024-02-10T09:11:23.967 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:23.965+0000 7f9c2a7fc700 1 -- 172.21.15.155:0/1411699313 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9c240095f0 con 0x7f9c340f5410 2024-02-10T09:11:23.967 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:23.965+0000 7f9c3bfe2700 1 -- 172.21.15.155:0/1411699313 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c34059100 con 0x7f9c340f5410 2024-02-10T09:11:23.967 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:23.965+0000 7f9c2a7fc700 1 --2- 172.21.15.155:0/1411699313 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f9c2002ff10 0x7f9c200323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:11:23.967 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:23.965+0000 7f9c2a7fc700 1 -- 172.21.15.155:0/1411699313 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9c24040e50 con 0x7f9c340f5410 2024-02-10T09:11:23.967 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:23.965+0000 7f9c3957d700 1 --2- 172.21.15.155:0/1411699313 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f9c2002ff10 0x7f9c200323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:11:23.967 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:23.966+0000 7f9c3957d700 1 --2- 172.21.15.155:0/1411699313 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f9c2002ff10 0x7f9c200323d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f9c300096a0 tx=0x7f9c30006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:11:23.970 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:23.970+0000 7f9c2a7fc700 1 -- 172.21.15.155:0/1411699313 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9c24014420 con 0x7f9c340f5410 2024-02-10T09:11:24.174 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:24.173+0000 7f9c3bfe2700 1 -- 172.21.15.155:0/1411699313 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9c34045ad0 con 0x7f9c340f5410 2024-02-10T09:11:24.174 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:24.174+0000 7f9c2a7fc700 1 -- 172.21.15.155:0/1411699313 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9c24014420 con 0x7f9c340f5410 2024-02-10T09:11:24.174 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:11:24.174 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:11:24.176 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:24.175+0000 7f9c3bfe2700 1 -- 172.21.15.155:0/1411699313 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f9c2002ff10 msgr2=0x7f9c200323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:11:24.176 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:24.175+0000 7f9c3bfe2700 1 --2- 172.21.15.155:0/1411699313 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f9c2002ff10 0x7f9c200323d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f9c300096a0 tx=0x7f9c30006b40).stop 2024-02-10T09:11:24.176 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:24.175+0000 7f9c3bfe2700 1 -- 172.21.15.155:0/1411699313 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c340f5410 msgr2=0x7f9c34109530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:11:24.176 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:24.175+0000 7f9c3bfe2700 1 --2- 172.21.15.155:0/1411699313 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c340f5410 0x7f9c34109530 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f9c24005010 tx=0x7f9c2400b040).stop 2024-02-10T09:11:24.176 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:24.176+0000 7f9c3bfe2700 1 -- 172.21.15.155:0/1411699313 shutdown_connections 2024-02-10T09:11:24.176 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:24.176+0000 7f9c3bfe2700 1 --2- 172.21.15.155:0/1411699313 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f9c2002ff10 0x7f9c200323d0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:11:24.176 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:24.176+0000 7f9c3bfe2700 1 --2- 172.21.15.155:0/1411699313 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9c340f5410 0x7f9c34109530 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:11:24.176 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:24.176+0000 7f9c3bfe2700 1 -- 172.21.15.155:0/1411699313 >> 172.21.15.155:0/1411699313 conn(0x7f9c340f0b70 msgr2=0x7f9c340f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:11:24.176 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:24.176+0000 7f9c3bfe2700 1 -- 172.21.15.155:0/1411699313 shutdown_connections 2024-02-10T09:11:24.176 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:24.176+0000 7f9c3bfe2700 1 -- 172.21.15.155:0/1411699313 wait complete. 2024-02-10T09:11:24.177 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:11:25.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:11:24 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:11:23.755210+0000 mgr.smithi077.dxntvw (mgr.14162) 389 : cluster [DBG] pgmap v373: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:11:25.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:11:24 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:11:24.174415+0000 mon.smithi077 (mon.0) 335 : audit [DBG] from='client.? 172.21.15.155:0/1411699313' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:11:25.872 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:11:25.872 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:11:27.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:11:26 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:11:25.755541+0000 mgr.smithi077.dxntvw (mgr.14162) 390 : cluster [DBG] pgmap v374: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:11:27.827 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:27.825+0000 7fd9c097e700 1 -- 172.21.15.155:0/4256675281 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9bc0f5410 msgr2=0x7fd9bc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:11:27.827 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:27.825+0000 7fd9c097e700 1 --2- 172.21.15.155:0/4256675281 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9bc0f5410 0x7fd9bc0f5830 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7fd9a4009fb0 tx=0x7fd9a4009b30).stop 2024-02-10T09:11:27.827 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:27.825+0000 7fd9c097e700 1 -- 172.21.15.155:0/4256675281 shutdown_connections 2024-02-10T09:11:27.827 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:27.825+0000 7fd9c097e700 1 --2- 172.21.15.155:0/4256675281 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9bc0f5410 0x7fd9bc0f5830 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:11:27.827 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:27.825+0000 7fd9c097e700 1 -- 172.21.15.155:0/4256675281 >> 172.21.15.155:0/4256675281 conn(0x7fd9bc0f0b70 msgr2=0x7fd9bc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:11:27.828 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:27.825+0000 7fd9c097e700 1 -- 172.21.15.155:0/4256675281 shutdown_connections 2024-02-10T09:11:27.828 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:27.825+0000 7fd9c097e700 1 -- 172.21.15.155:0/4256675281 wait complete. 2024-02-10T09:11:27.828 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:27.826+0000 7fd9c097e700 1 Processor -- start 2024-02-10T09:11:27.828 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:27.826+0000 7fd9c097e700 1 -- start start 2024-02-10T09:11:27.828 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:27.826+0000 7fd9c097e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9bc0f5410 0x7fd9bc0fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:11:27.828 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:27.826+0000 7fd9c097e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9bc0fb960 con 0x7fd9bc0f5410 2024-02-10T09:11:27.828 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:27.826+0000 7fd9ba59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9bc0f5410 0x7fd9bc0fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:11:27.828 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:27.826+0000 7fd9ba59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9bc0f5410 0x7fd9bc0fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:45324/0 (socket says 172.21.15.155:45324) 2024-02-10T09:11:27.828 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:27.826+0000 7fd9ba59c700 1 -- 172.21.15.155:0/4011456636 learned_addr learned my addr 172.21.15.155:0/4011456636 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:11:27.828 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:27.826+0000 7fd9ba59c700 1 -- 172.21.15.155:0/4011456636 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9a4005040 con 0x7fd9bc0f5410 2024-02-10T09:11:27.828 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:27.826+0000 7fd9ba59c700 1 --2- 172.21.15.155:0/4011456636 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9bc0f5410 0x7fd9bc0fd4e0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7fd9a4005010 tx=0x7fd9a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:11:27.828 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:27.827+0000 7fd9b2ffd700 1 -- 172.21.15.155:0/4011456636 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd9a4009320 con 0x7fd9bc0f5410 2024-02-10T09:11:27.828 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:27.827+0000 7fd9c097e700 1 -- 172.21.15.155:0/4011456636 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9bc0fbb00 con 0x7fd9bc0f5410 2024-02-10T09:11:27.828 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:27.827+0000 7fd9b2ffd700 1 -- 172.21.15.155:0/4011456636 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd9a4013380 con 0x7fd9bc0f5410 2024-02-10T09:11:27.829 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:27.827+0000 7fd9b2ffd700 1 -- 172.21.15.155:0/4011456636 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd9a4014e50 con 0x7fd9bc0f5410 2024-02-10T09:11:27.829 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:27.827+0000 7fd9c097e700 1 -- 172.21.15.155:0/4011456636 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9bc0fbf90 con 0x7fd9bc0f5410 2024-02-10T09:11:27.829 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:27.828+0000 7fd9b2ffd700 1 -- 172.21.15.155:0/4011456636 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd9a40095f0 con 0x7fd9bc0f5410 2024-02-10T09:11:27.830 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:27.828+0000 7fd9c097e700 1 -- 172.21.15.155:0/4011456636 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9bc059100 con 0x7fd9bc0f5410 2024-02-10T09:11:27.830 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:27.828+0000 7fd9b2ffd700 1 --2- 172.21.15.155:0/4011456636 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fd9a802ff10 0x7fd9a80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:11:27.830 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:27.828+0000 7fd9b2ffd700 1 -- 172.21.15.155:0/4011456636 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd9a4040e60 con 0x7fd9bc0f5410 2024-02-10T09:11:27.830 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:27.828+0000 7fd9b9d9b700 1 --2- 172.21.15.155:0/4011456636 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fd9a802ff10 0x7fd9a80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:11:27.830 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:27.828+0000 7fd9b9d9b700 1 --2- 172.21.15.155:0/4011456636 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fd9a802ff10 0x7fd9a80323d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7fd9ac0096a0 tx=0x7fd9ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:11:27.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:27.832+0000 7fd9b2ffd700 1 -- 172.21.15.155:0/4011456636 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd9a4014420 con 0x7fd9bc0f5410 2024-02-10T09:11:28.038 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:28.038+0000 7fd9c097e700 1 -- 172.21.15.155:0/4011456636 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd9bc0fafa0 con 0x7fd9bc0f5410 2024-02-10T09:11:28.039 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:28.039+0000 7fd9b2ffd700 1 -- 172.21.15.155:0/4011456636 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd9a401d980 con 0x7fd9bc0f5410 2024-02-10T09:11:28.039 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:11:28.039 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:11:28.040 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:28.040+0000 7fd9c097e700 1 -- 172.21.15.155:0/4011456636 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fd9a802ff10 msgr2=0x7fd9a80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:11:28.040 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:28.040+0000 7fd9c097e700 1 --2- 172.21.15.155:0/4011456636 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fd9a802ff10 0x7fd9a80323d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7fd9ac0096a0 tx=0x7fd9ac006b40).stop 2024-02-10T09:11:28.041 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:28.040+0000 7fd9c097e700 1 -- 172.21.15.155:0/4011456636 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9bc0f5410 msgr2=0x7fd9bc0fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:11:28.041 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:28.040+0000 7fd9c097e700 1 --2- 172.21.15.155:0/4011456636 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9bc0f5410 0x7fd9bc0fd4e0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7fd9a4005010 tx=0x7fd9a400b040).stop 2024-02-10T09:11:28.041 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:28.040+0000 7fd9c097e700 1 -- 172.21.15.155:0/4011456636 shutdown_connections 2024-02-10T09:11:28.041 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:28.040+0000 7fd9c097e700 1 --2- 172.21.15.155:0/4011456636 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7fd9a802ff10 0x7fd9a80323d0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:11:28.041 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:28.040+0000 7fd9c097e700 1 --2- 172.21.15.155:0/4011456636 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9bc0f5410 0x7fd9bc0fd4e0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:11:28.041 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:28.040+0000 7fd9c097e700 1 -- 172.21.15.155:0/4011456636 >> 172.21.15.155:0/4011456636 conn(0x7fd9bc0f0b70 msgr2=0x7fd9bc05fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:11:28.041 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:28.040+0000 7fd9c097e700 1 -- 172.21.15.155:0/4011456636 shutdown_connections 2024-02-10T09:11:28.041 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:28.040+0000 7fd9c097e700 1 -- 172.21.15.155:0/4011456636 wait complete. 2024-02-10T09:11:28.041 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:11:29.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:11:28 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:11:27.755855+0000 mgr.smithi077.dxntvw (mgr.14162) 391 : cluster [DBG] pgmap v375: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:11:29.160 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:11:28 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:11:28.039391+0000 mon.smithi077 (mon.0) 336 : audit [DBG] from='client.? 172.21.15.155:0/4011456636' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:11:29.693 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:11:29.693 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:11:31.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:11:30 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:11:29.756147+0000 mgr.smithi077.dxntvw (mgr.14162) 392 : cluster [DBG] pgmap v376: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:11:31.505 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:31.503+0000 7efe777f6700 1 -- 172.21.15.155:0/947249055 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe700f47c0 msgr2=0x7efe700f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:11:31.505 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:31.503+0000 7efe777f6700 1 --2- 172.21.15.155:0/947249055 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe700f47c0 0x7efe700f4be0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7efe6c009fb0 tx=0x7efe6c009b30).stop 2024-02-10T09:11:31.505 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:31.503+0000 7efe777f6700 1 -- 172.21.15.155:0/947249055 shutdown_connections 2024-02-10T09:11:31.505 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:31.503+0000 7efe777f6700 1 --2- 172.21.15.155:0/947249055 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe700f47c0 0x7efe700f4be0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:11:31.505 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:31.503+0000 7efe777f6700 1 -- 172.21.15.155:0/947249055 >> 172.21.15.155:0/947249055 conn(0x7efe700f02e0 msgr2=0x7efe700f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:11:31.505 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:31.503+0000 7efe777f6700 1 -- 172.21.15.155:0/947249055 shutdown_connections 2024-02-10T09:11:31.505 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:31.503+0000 7efe777f6700 1 -- 172.21.15.155:0/947249055 wait complete. 2024-02-10T09:11:31.505 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:31.503+0000 7efe777f6700 1 Processor -- start 2024-02-10T09:11:31.505 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:31.503+0000 7efe777f6700 1 -- start start 2024-02-10T09:11:31.506 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:31.504+0000 7efe777f6700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe700f47c0 0x7efe701067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:11:31.506 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:31.504+0000 7efe777f6700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe70106cc0 con 0x7efe700f47c0 2024-02-10T09:11:31.508 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:31.504+0000 7efe75592700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe700f47c0 0x7efe701067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:11:31.508 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:31.504+0000 7efe75592700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe700f47c0 0x7efe701067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:45344/0 (socket says 172.21.15.155:45344) 2024-02-10T09:11:31.508 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:31.504+0000 7efe75592700 1 -- 172.21.15.155:0/2390103131 learned_addr learned my addr 172.21.15.155:0/2390103131 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:11:31.508 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:31.504+0000 7efe75592700 1 -- 172.21.15.155:0/2390103131 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe6c005040 con 0x7efe700f47c0 2024-02-10T09:11:31.508 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:31.504+0000 7efe75592700 1 --2- 172.21.15.155:0/2390103131 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe700f47c0 0x7efe701067d0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7efe6c005010 tx=0x7efe6c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:11:31.508 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:31.505+0000 7efe65ffb700 1 -- 172.21.15.155:0/2390103131 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe6c009320 con 0x7efe700f47c0 2024-02-10T09:11:31.508 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:31.505+0000 7efe777f6700 1 -- 172.21.15.155:0/2390103131 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe70106ec0 con 0x7efe700f47c0 2024-02-10T09:11:31.508 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:31.505+0000 7efe65ffb700 1 -- 172.21.15.155:0/2390103131 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efe6c013380 con 0x7efe700f47c0 2024-02-10T09:11:31.508 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:31.505+0000 7efe65ffb700 1 -- 172.21.15.155:0/2390103131 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe6c014e50 con 0x7efe700f47c0 2024-02-10T09:11:31.509 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:31.505+0000 7efe777f6700 1 -- 172.21.15.155:0/2390103131 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe70107350 con 0x7efe700f47c0 2024-02-10T09:11:31.509 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:31.506+0000 7efe65ffb700 1 -- 172.21.15.155:0/2390103131 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efe6c0095f0 con 0x7efe700f47c0 2024-02-10T09:11:31.509 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:31.506+0000 7efe777f6700 1 -- 172.21.15.155:0/2390103131 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe70045ad0 con 0x7efe700f47c0 2024-02-10T09:11:31.509 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:31.506+0000 7efe65ffb700 1 --2- 172.21.15.155:0/2390103131 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7efe5c02ff10 0x7efe5c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:11:31.509 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:31.506+0000 7efe65ffb700 1 -- 172.21.15.155:0/2390103131 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efe6c040e90 con 0x7efe700f47c0 2024-02-10T09:11:31.509 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:31.506+0000 7efe74d91700 1 --2- 172.21.15.155:0/2390103131 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7efe5c02ff10 0x7efe5c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:11:31.509 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:31.507+0000 7efe74d91700 1 --2- 172.21.15.155:0/2390103131 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7efe5c02ff10 0x7efe5c0323d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7efe600096a0 tx=0x7efe60006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:11:31.511 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:31.511+0000 7efe65ffb700 1 -- 172.21.15.155:0/2390103131 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efe6c014420 con 0x7efe700f47c0 2024-02-10T09:11:31.715 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:31.714+0000 7efe777f6700 1 -- 172.21.15.155:0/2390103131 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efe70108200 con 0x7efe700f47c0 2024-02-10T09:11:31.716 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:31.714+0000 7efe65ffb700 1 -- 172.21.15.155:0/2390103131 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7efe6c010230 con 0x7efe700f47c0 2024-02-10T09:11:31.716 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:11:31.716 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:11:31.717 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:31.716+0000 7efe777f6700 1 -- 172.21.15.155:0/2390103131 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7efe5c02ff10 msgr2=0x7efe5c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:11:31.717 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:31.716+0000 7efe777f6700 1 --2- 172.21.15.155:0/2390103131 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7efe5c02ff10 0x7efe5c0323d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7efe600096a0 tx=0x7efe60006b40).stop 2024-02-10T09:11:31.717 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:31.716+0000 7efe777f6700 1 -- 172.21.15.155:0/2390103131 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe700f47c0 msgr2=0x7efe701067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:11:31.718 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:31.716+0000 7efe777f6700 1 --2- 172.21.15.155:0/2390103131 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe700f47c0 0x7efe701067d0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7efe6c005010 tx=0x7efe6c00b040).stop 2024-02-10T09:11:31.718 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:31.716+0000 7efe777f6700 1 -- 172.21.15.155:0/2390103131 shutdown_connections 2024-02-10T09:11:31.718 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:31.716+0000 7efe777f6700 1 --2- 172.21.15.155:0/2390103131 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7efe5c02ff10 0x7efe5c0323d0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:11:31.718 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:31.716+0000 7efe777f6700 1 --2- 172.21.15.155:0/2390103131 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efe700f47c0 0x7efe701067d0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:11:31.718 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:31.716+0000 7efe777f6700 1 -- 172.21.15.155:0/2390103131 >> 172.21.15.155:0/2390103131 conn(0x7efe700f02e0 msgr2=0x7efe700f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:11:31.718 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:31.716+0000 7efe777f6700 1 -- 172.21.15.155:0/2390103131 shutdown_connections 2024-02-10T09:11:31.718 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:31.716+0000 7efe777f6700 1 -- 172.21.15.155:0/2390103131 wait complete. 2024-02-10T09:11:31.718 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:11:32.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:11:31 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:11:31.716048+0000 mon.smithi077 (mon.0) 337 : audit [DBG] from='client.? 172.21.15.155:0/2390103131' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:11:33.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:11:32 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:11:31.756474+0000 mgr.smithi077.dxntvw (mgr.14162) 393 : cluster [DBG] pgmap v377: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:11:33.338 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:11:33.338 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:11:35.117 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:35.114+0000 7f3c2d5b3700 1 -- 172.21.15.155:0/690549467 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c280f6510 msgr2=0x7f3c280f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:11:35.117 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:35.114+0000 7f3c2d5b3700 1 --2- 172.21.15.155:0/690549467 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c280f6510 0x7f3c280f6930 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7f3c18009fb0 tx=0x7f3c18009b30).stop 2024-02-10T09:11:35.117 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:35.114+0000 7f3c2d5b3700 1 -- 172.21.15.155:0/690549467 shutdown_connections 2024-02-10T09:11:35.117 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:35.114+0000 7f3c2d5b3700 1 --2- 172.21.15.155:0/690549467 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c280f6510 0x7f3c280f6930 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:11:35.117 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:35.114+0000 7f3c2d5b3700 1 -- 172.21.15.155:0/690549467 >> 172.21.15.155:0/690549467 conn(0x7f3c280f1c70 msgr2=0x7f3c280f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:11:35.117 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:35.114+0000 7f3c2d5b3700 1 -- 172.21.15.155:0/690549467 shutdown_connections 2024-02-10T09:11:35.117 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:35.114+0000 7f3c2d5b3700 1 -- 172.21.15.155:0/690549467 wait complete. 2024-02-10T09:11:35.117 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:35.114+0000 7f3c2d5b3700 1 Processor -- start 2024-02-10T09:11:35.117 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:35.114+0000 7f3c2d5b3700 1 -- start start 2024-02-10T09:11:35.118 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:35.115+0000 7f3c2d5b3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c280f6510 0x7f3c281067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:11:35.118 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:35.115+0000 7f3c2d5b3700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c28106cd0 con 0x7f3c280f6510 2024-02-10T09:11:35.118 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:35.115+0000 7f3c26ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c280f6510 0x7f3c281067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:11:35.118 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:35.115+0000 7f3c26ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c280f6510 0x7f3c281067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:60282/0 (socket says 172.21.15.155:60282) 2024-02-10T09:11:35.118 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:35.115+0000 7f3c26ffd700 1 -- 172.21.15.155:0/4135342596 learned_addr learned my addr 172.21.15.155:0/4135342596 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:11:35.118 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:35.115+0000 7f3c26ffd700 1 -- 172.21.15.155:0/4135342596 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c18005040 con 0x7f3c280f6510 2024-02-10T09:11:35.118 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:35.115+0000 7f3c26ffd700 1 --2- 172.21.15.155:0/4135342596 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c280f6510 0x7f3c281067e0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f3c18005010 tx=0x7f3c1800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:11:35.118 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:35.115+0000 7f3c0f7fe700 1 -- 172.21.15.155:0/4135342596 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c18009320 con 0x7f3c280f6510 2024-02-10T09:11:35.118 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:35.116+0000 7f3c2d5b3700 1 -- 172.21.15.155:0/4135342596 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c28106ed0 con 0x7f3c280f6510 2024-02-10T09:11:35.118 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:35.116+0000 7f3c0f7fe700 1 -- 172.21.15.155:0/4135342596 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3c18013380 con 0x7f3c280f6510 2024-02-10T09:11:35.118 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:35.116+0000 7f3c0f7fe700 1 -- 172.21.15.155:0/4135342596 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c18014e50 con 0x7f3c280f6510 2024-02-10T09:11:35.118 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:35.116+0000 7f3c2d5b3700 1 -- 172.21.15.155:0/4135342596 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c28107360 con 0x7f3c280f6510 2024-02-10T09:11:35.119 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:35.116+0000 7f3c0f7fe700 1 -- 172.21.15.155:0/4135342596 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3c180263a0 con 0x7f3c280f6510 2024-02-10T09:11:35.119 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:35.116+0000 7f3c0f7fe700 1 --2- 172.21.15.155:0/4135342596 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f3c1002ff10 0x7f3c100323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:11:35.119 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:35.117+0000 7f3c0f7fe700 1 -- 172.21.15.155:0/4135342596 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3c18040f80 con 0x7f3c280f6510 2024-02-10T09:11:35.120 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:35.117+0000 7f3c2d5b3700 1 -- 172.21.15.155:0/4135342596 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c28045ad0 con 0x7f3c280f6510 2024-02-10T09:11:35.120 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:35.117+0000 7f3c267fc700 1 --2- 172.21.15.155:0/4135342596 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f3c1002ff10 0x7f3c100323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:11:35.120 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:35.117+0000 7f3c267fc700 1 --2- 172.21.15.155:0/4135342596 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f3c1002ff10 0x7f3c100323d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f3c1c0096a0 tx=0x7f3c1c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:11:35.123 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:35.122+0000 7f3c0f7fe700 1 -- 172.21.15.155:0/4135342596 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3c18012920 con 0x7f3c280f6510 2024-02-10T09:11:35.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:11:34 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:11:33.756826+0000 mgr.smithi077.dxntvw (mgr.14162) 394 : cluster [DBG] pgmap v378: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:11:35.330 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:35.329+0000 7f3c2d5b3700 1 -- 172.21.15.155:0/4135342596 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3c28108080 con 0x7f3c280f6510 2024-02-10T09:11:35.330 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:35.329+0000 7f3c0f7fe700 1 -- 172.21.15.155:0/4135342596 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3c18012330 con 0x7f3c280f6510 2024-02-10T09:11:35.330 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:11:35.331 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:11:35.332 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:35.331+0000 7f3c2d5b3700 1 -- 172.21.15.155:0/4135342596 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f3c1002ff10 msgr2=0x7f3c100323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:11:35.332 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:35.331+0000 7f3c2d5b3700 1 --2- 172.21.15.155:0/4135342596 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f3c1002ff10 0x7f3c100323d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f3c1c0096a0 tx=0x7f3c1c006b40).stop 2024-02-10T09:11:35.332 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:35.331+0000 7f3c2d5b3700 1 -- 172.21.15.155:0/4135342596 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c280f6510 msgr2=0x7f3c281067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:11:35.332 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:35.331+0000 7f3c2d5b3700 1 --2- 172.21.15.155:0/4135342596 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c280f6510 0x7f3c281067e0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f3c18005010 tx=0x7f3c1800b040).stop 2024-02-10T09:11:35.332 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:35.331+0000 7f3c2d5b3700 1 -- 172.21.15.155:0/4135342596 shutdown_connections 2024-02-10T09:11:35.332 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:35.331+0000 7f3c2d5b3700 1 --2- 172.21.15.155:0/4135342596 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f3c1002ff10 0x7f3c100323d0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:11:35.332 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:35.331+0000 7f3c2d5b3700 1 --2- 172.21.15.155:0/4135342596 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3c280f6510 0x7f3c281067e0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:11:35.333 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:35.331+0000 7f3c2d5b3700 1 -- 172.21.15.155:0/4135342596 >> 172.21.15.155:0/4135342596 conn(0x7f3c280f1c70 msgr2=0x7f3c280f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:11:35.333 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:35.331+0000 7f3c2d5b3700 1 -- 172.21.15.155:0/4135342596 shutdown_connections 2024-02-10T09:11:35.333 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:35.331+0000 7f3c2d5b3700 1 -- 172.21.15.155:0/4135342596 wait complete. 2024-02-10T09:11:35.333 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:11:36.148 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:11:35 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:11:35.330937+0000 mon.smithi077 (mon.0) 338 : audit [DBG] from='client.? 172.21.15.155:0/4135342596' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:11:37.024 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:11:37.024 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:11:37.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:11:36 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:11:35.757220+0000 mgr.smithi077.dxntvw (mgr.14162) 395 : cluster [DBG] pgmap v379: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:11:38.859 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:38.858+0000 7f012d9ac700 1 -- 172.21.15.155:0/2273715764 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01280f6a80 msgr2=0x7f01280f6ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:11:38.859 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:38.858+0000 7f012d9ac700 1 --2- 172.21.15.155:0/2273715764 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01280f6a80 0x7f01280f6ea0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f011c009fb0 tx=0x7f011c009b30).stop 2024-02-10T09:11:38.860 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:38.858+0000 7f012d9ac700 1 -- 172.21.15.155:0/2273715764 shutdown_connections 2024-02-10T09:11:38.860 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:38.858+0000 7f012d9ac700 1 --2- 172.21.15.155:0/2273715764 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01280f6a80 0x7f01280f6ea0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:11:38.860 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:38.858+0000 7f012d9ac700 1 -- 172.21.15.155:0/2273715764 >> 172.21.15.155:0/2273715764 conn(0x7f01280f02e0 msgr2=0x7f01280f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:11:38.860 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:38.858+0000 7f012d9ac700 1 -- 172.21.15.155:0/2273715764 shutdown_connections 2024-02-10T09:11:38.860 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:38.858+0000 7f012d9ac700 1 -- 172.21.15.155:0/2273715764 wait complete. 2024-02-10T09:11:38.861 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:38.859+0000 7f012d9ac700 1 Processor -- start 2024-02-10T09:11:38.861 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:38.859+0000 7f012d9ac700 1 -- start start 2024-02-10T09:11:38.862 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:38.859+0000 7f012d9ac700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01280f6a80 0x7f01281056a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:11:38.862 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:38.859+0000 7f012d9ac700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0128105b90 con 0x7f01280f6a80 2024-02-10T09:11:38.862 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:38.859+0000 7f0126ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01280f6a80 0x7f01281056a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:11:38.862 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:38.860+0000 7f0126ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01280f6a80 0x7f01281056a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:60294/0 (socket says 172.21.15.155:60294) 2024-02-10T09:11:38.862 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:38.860+0000 7f0126ffd700 1 -- 172.21.15.155:0/1988431318 learned_addr learned my addr 172.21.15.155:0/1988431318 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:11:38.862 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:38.860+0000 7f0126ffd700 1 -- 172.21.15.155:0/1988431318 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f011c005040 con 0x7f01280f6a80 2024-02-10T09:11:38.862 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:38.860+0000 7f0126ffd700 1 --2- 172.21.15.155:0/1988431318 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01280f6a80 0x7f01281056a0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f011c0089f0 tx=0x7f011c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:11:38.862 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:38.860+0000 7f010f7fe700 1 -- 172.21.15.155:0/1988431318 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f011c009320 con 0x7f01280f6a80 2024-02-10T09:11:38.862 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:38.860+0000 7f012d9ac700 1 -- 172.21.15.155:0/1988431318 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0128105d90 con 0x7f01280f6a80 2024-02-10T09:11:38.862 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:38.860+0000 7f010f7fe700 1 -- 172.21.15.155:0/1988431318 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f011c013380 con 0x7f01280f6a80 2024-02-10T09:11:38.862 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:38.860+0000 7f010f7fe700 1 -- 172.21.15.155:0/1988431318 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f011c014e20 con 0x7f01280f6a80 2024-02-10T09:11:38.862 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:38.860+0000 7f012d9ac700 1 -- 172.21.15.155:0/1988431318 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0128106220 con 0x7f01280f6a80 2024-02-10T09:11:38.863 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:38.861+0000 7f012d9ac700 1 -- 172.21.15.155:0/1988431318 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0128045ad0 con 0x7f01280f6a80 2024-02-10T09:11:38.863 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:38.861+0000 7f010f7fe700 1 -- 172.21.15.155:0/1988431318 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f011c0095f0 con 0x7f01280f6a80 2024-02-10T09:11:38.863 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:38.861+0000 7f010f7fe700 1 --2- 172.21.15.155:0/1988431318 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f011002ff10 0x7f01100323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:11:38.863 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:38.861+0000 7f010f7fe700 1 -- 172.21.15.155:0/1988431318 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f011c040e90 con 0x7f01280f6a80 2024-02-10T09:11:38.864 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:38.862+0000 7f01267fc700 1 --2- 172.21.15.155:0/1988431318 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f011002ff10 0x7f01100323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:11:38.864 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:38.862+0000 7f01267fc700 1 --2- 172.21.15.155:0/1988431318 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f011002ff10 0x7f01100323d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f01180096a0 tx=0x7f0118006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:11:38.867 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:38.866+0000 7f010f7fe700 1 -- 172.21.15.155:0/1988431318 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f011c014420 con 0x7f01280f6a80 2024-02-10T09:11:39.074 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:39.073+0000 7f012d9ac700 1 -- 172.21.15.155:0/1988431318 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f01281070d0 con 0x7f01280f6a80 2024-02-10T09:11:39.074 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:39.073+0000 7f010f7fe700 1 -- 172.21.15.155:0/1988431318 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f011c01d980 con 0x7f01280f6a80 2024-02-10T09:11:39.074 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:11:39.075 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:11:39.076 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:39.075+0000 7f012d9ac700 1 -- 172.21.15.155:0/1988431318 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f011002ff10 msgr2=0x7f01100323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:11:39.076 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:39.075+0000 7f012d9ac700 1 --2- 172.21.15.155:0/1988431318 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f011002ff10 0x7f01100323d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f01180096a0 tx=0x7f0118006b40).stop 2024-02-10T09:11:39.076 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:39.075+0000 7f012d9ac700 1 -- 172.21.15.155:0/1988431318 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01280f6a80 msgr2=0x7f01281056a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:11:39.077 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:39.075+0000 7f012d9ac700 1 --2- 172.21.15.155:0/1988431318 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01280f6a80 0x7f01281056a0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f011c0089f0 tx=0x7f011c00b040).stop 2024-02-10T09:11:39.077 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:39.075+0000 7f012d9ac700 1 -- 172.21.15.155:0/1988431318 shutdown_connections 2024-02-10T09:11:39.077 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:39.075+0000 7f012d9ac700 1 --2- 172.21.15.155:0/1988431318 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f011002ff10 0x7f01100323d0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:11:39.077 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:39.075+0000 7f012d9ac700 1 --2- 172.21.15.155:0/1988431318 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f01280f6a80 0x7f01281056a0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:11:39.077 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:39.075+0000 7f012d9ac700 1 -- 172.21.15.155:0/1988431318 >> 172.21.15.155:0/1988431318 conn(0x7f01280f02e0 msgr2=0x7f01280f0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:11:39.077 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:39.075+0000 7f012d9ac700 1 -- 172.21.15.155:0/1988431318 shutdown_connections 2024-02-10T09:11:39.077 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:39.075+0000 7f012d9ac700 1 -- 172.21.15.155:0/1988431318 wait complete. 2024-02-10T09:11:39.077 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:11:39.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:11:38 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:11:37.757597+0000 mgr.smithi077.dxntvw (mgr.14162) 396 : cluster [DBG] pgmap v380: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:11:40.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:11:39 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:11:39.074895+0000 mon.smithi077 (mon.0) 339 : audit [DBG] from='client.? 172.21.15.155:0/1988431318' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:11:40.716 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:11:40.717 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:11:41.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:11:40 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:11:39.757934+0000 mgr.smithi077.dxntvw (mgr.14162) 397 : cluster [DBG] pgmap v381: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:11:42.530 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:42.527+0000 7f6b131bf700 1 -- 172.21.15.155:0/2362995751 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b0c0f5410 msgr2=0x7f6b0c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:11:42.530 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:42.527+0000 7f6b131bf700 1 --2- 172.21.15.155:0/2362995751 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b0c0f5410 0x7f6b0c0f5830 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7f6b00009fb0 tx=0x7f6b00009b30).stop 2024-02-10T09:11:42.530 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:42.527+0000 7f6b131bf700 1 -- 172.21.15.155:0/2362995751 shutdown_connections 2024-02-10T09:11:42.530 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:42.527+0000 7f6b131bf700 1 --2- 172.21.15.155:0/2362995751 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b0c0f5410 0x7f6b0c0f5830 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:11:42.531 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:42.527+0000 7f6b131bf700 1 -- 172.21.15.155:0/2362995751 >> 172.21.15.155:0/2362995751 conn(0x7f6b0c0f0b70 msgr2=0x7f6b0c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:11:42.531 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:42.527+0000 7f6b131bf700 1 -- 172.21.15.155:0/2362995751 shutdown_connections 2024-02-10T09:11:42.531 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:42.527+0000 7f6b131bf700 1 -- 172.21.15.155:0/2362995751 wait complete. 2024-02-10T09:11:42.531 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:42.528+0000 7f6b131bf700 1 Processor -- start 2024-02-10T09:11:42.531 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:42.528+0000 7f6b131bf700 1 -- start start 2024-02-10T09:11:42.531 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:42.528+0000 7f6b131bf700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b0c0f5410 0x7f6b0c109520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:11:42.531 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:42.528+0000 7f6b131bf700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b0c060240 con 0x7f6b0c0f5410 2024-02-10T09:11:42.531 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:42.528+0000 7f6b10f5b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b0c0f5410 0x7f6b0c109520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:11:42.531 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:42.528+0000 7f6b10f5b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b0c0f5410 0x7f6b0c109520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:42874/0 (socket says 172.21.15.155:42874) 2024-02-10T09:11:42.531 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:42.528+0000 7f6b10f5b700 1 -- 172.21.15.155:0/528748220 learned_addr learned my addr 172.21.15.155:0/528748220 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:11:42.531 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:42.529+0000 7f6b10f5b700 1 -- 172.21.15.155:0/528748220 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b00005040 con 0x7f6b0c0f5410 2024-02-10T09:11:42.531 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:42.529+0000 7f6b10f5b700 1 --2- 172.21.15.155:0/528748220 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b0c0f5410 0x7f6b0c109520 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f6b000089f0 tx=0x7f6b0000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:11:42.532 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:42.529+0000 7f6b097fa700 1 -- 172.21.15.155:0/528748220 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b00009320 con 0x7f6b0c0f5410 2024-02-10T09:11:42.532 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:42.529+0000 7f6b131bf700 1 -- 172.21.15.155:0/528748220 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b0c060440 con 0x7f6b0c0f5410 2024-02-10T09:11:42.532 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:42.529+0000 7f6b097fa700 1 -- 172.21.15.155:0/528748220 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6b00013380 con 0x7f6b0c0f5410 2024-02-10T09:11:42.532 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:42.529+0000 7f6b097fa700 1 -- 172.21.15.155:0/528748220 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b0001ed80 con 0x7f6b0c0f5410 2024-02-10T09:11:42.532 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:42.529+0000 7f6b131bf700 1 -- 172.21.15.155:0/528748220 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b0c109bd0 con 0x7f6b0c0f5410 2024-02-10T09:11:42.533 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:42.530+0000 7f6b097fa700 1 -- 172.21.15.155:0/528748220 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6b0001e420 con 0x7f6b0c0f5410 2024-02-10T09:11:42.533 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:42.530+0000 7f6b131bf700 1 -- 172.21.15.155:0/528748220 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b0c045ad0 con 0x7f6b0c0f5410 2024-02-10T09:11:42.533 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:42.530+0000 7f6b097fa700 1 --2- 172.21.15.155:0/528748220 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f6afc02ff60 0x7f6afc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:11:42.533 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:42.530+0000 7f6b097fa700 1 -- 172.21.15.155:0/528748220 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6b00041130 con 0x7f6b0c0f5410 2024-02-10T09:11:42.533 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:42.530+0000 7f6b0bfff700 1 --2- 172.21.15.155:0/528748220 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f6afc02ff60 0x7f6afc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:11:42.533 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:42.531+0000 7f6b0bfff700 1 --2- 172.21.15.155:0/528748220 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f6afc02ff60 0x7f6afc032420 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f6af80096a0 tx=0x7f6af8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:11:42.536 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:42.535+0000 7f6b097fa700 1 -- 172.21.15.155:0/528748220 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6b000095f0 con 0x7f6b0c0f5410 2024-02-10T09:11:42.738 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:42.737+0000 7f6b131bf700 1 -- 172.21.15.155:0/528748220 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6b0c10a3f0 con 0x7f6b0c0f5410 2024-02-10T09:11:42.739 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:42.738+0000 7f6b097fa700 1 -- 172.21.15.155:0/528748220 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6b0002c2d0 con 0x7f6b0c0f5410 2024-02-10T09:11:42.739 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:11:42.739 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:11:42.741 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:42.740+0000 7f6b131bf700 1 -- 172.21.15.155:0/528748220 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f6afc02ff60 msgr2=0x7f6afc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:11:42.741 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:42.740+0000 7f6b131bf700 1 --2- 172.21.15.155:0/528748220 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f6afc02ff60 0x7f6afc032420 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f6af80096a0 tx=0x7f6af8006b40).stop 2024-02-10T09:11:42.741 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:42.740+0000 7f6b131bf700 1 -- 172.21.15.155:0/528748220 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b0c0f5410 msgr2=0x7f6b0c109520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:11:42.741 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:42.740+0000 7f6b131bf700 1 --2- 172.21.15.155:0/528748220 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b0c0f5410 0x7f6b0c109520 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f6b000089f0 tx=0x7f6b0000b040).stop 2024-02-10T09:11:42.741 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:42.740+0000 7f6b131bf700 1 -- 172.21.15.155:0/528748220 shutdown_connections 2024-02-10T09:11:42.741 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:42.740+0000 7f6b131bf700 1 --2- 172.21.15.155:0/528748220 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f6afc02ff60 0x7f6afc032420 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:11:42.741 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:42.740+0000 7f6b131bf700 1 --2- 172.21.15.155:0/528748220 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6b0c0f5410 0x7f6b0c109520 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:11:42.741 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:42.740+0000 7f6b131bf700 1 -- 172.21.15.155:0/528748220 >> 172.21.15.155:0/528748220 conn(0x7f6b0c0f0b70 msgr2=0x7f6b0c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:11:42.741 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:42.740+0000 7f6b131bf700 1 -- 172.21.15.155:0/528748220 shutdown_connections 2024-02-10T09:11:42.741 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:42.740+0000 7f6b131bf700 1 -- 172.21.15.155:0/528748220 wait complete. 2024-02-10T09:11:42.742 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:11:43.148 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:11:42 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:11:41.758203+0000 mgr.smithi077.dxntvw (mgr.14162) 398 : cluster [DBG] pgmap v382: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:11:43.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:11:42 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:11:42.739668+0000 mon.smithi077 (mon.0) 340 : audit [DBG] from='client.? 172.21.15.155:0/528748220' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:11:44.425 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:11:44.425 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:11:45.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:11:44 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:11:43.758627+0000 mgr.smithi077.dxntvw (mgr.14162) 399 : cluster [DBG] pgmap v383: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:11:46.160 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:46.157+0000 7f0337289700 1 -- 172.21.15.155:0/167876901 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03300f5410 msgr2=0x7f03300f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:11:46.160 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:46.157+0000 7f0337289700 1 --2- 172.21.15.155:0/167876901 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03300f5410 0x7f03300f5830 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7f0328009960 tx=0x7f0328004d10).stop 2024-02-10T09:11:46.160 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:46.157+0000 7f0337289700 1 -- 172.21.15.155:0/167876901 shutdown_connections 2024-02-10T09:11:46.160 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:46.157+0000 7f0337289700 1 --2- 172.21.15.155:0/167876901 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03300f5410 0x7f03300f5830 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:11:46.161 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:46.157+0000 7f0337289700 1 -- 172.21.15.155:0/167876901 >> 172.21.15.155:0/167876901 conn(0x7f03300f0b70 msgr2=0x7f03300f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:11:46.161 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:46.157+0000 7f0337289700 1 -- 172.21.15.155:0/167876901 shutdown_connections 2024-02-10T09:11:46.161 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:46.157+0000 7f0337289700 1 -- 172.21.15.155:0/167876901 wait complete. 2024-02-10T09:11:46.161 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:46.158+0000 7f0337289700 1 Processor -- start 2024-02-10T09:11:46.161 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:46.158+0000 7f0337289700 1 -- start start 2024-02-10T09:11:46.161 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:46.158+0000 7f0337289700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03300f5410 0x7f03300fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:11:46.161 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:46.158+0000 7f0337289700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03300fb960 con 0x7f03300f5410 2024-02-10T09:11:46.161 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:46.158+0000 7f0335025700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03300f5410 0x7f03300fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:11:46.161 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:46.158+0000 7f0335025700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03300f5410 0x7f03300fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:42888/0 (socket says 172.21.15.155:42888) 2024-02-10T09:11:46.161 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:46.158+0000 7f0335025700 1 -- 172.21.15.155:0/243961396 learned_addr learned my addr 172.21.15.155:0/243961396 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:11:46.161 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:46.158+0000 7f0335025700 1 -- 172.21.15.155:0/243961396 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0328009640 con 0x7f03300f5410 2024-02-10T09:11:46.161 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:46.158+0000 7f0335025700 1 --2- 172.21.15.155:0/243961396 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03300f5410 0x7f03300fd4e0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f0328004b70 tx=0x7f0328004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:11:46.161 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:46.159+0000 7f0325ffb700 1 -- 172.21.15.155:0/243961396 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0328018070 con 0x7f03300f5410 2024-02-10T09:11:46.161 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:46.159+0000 7f0337289700 1 -- 172.21.15.155:0/243961396 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03300fbb00 con 0x7f03300f5410 2024-02-10T09:11:46.161 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:46.159+0000 7f0325ffb700 1 -- 172.21.15.155:0/243961396 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f032800b970 con 0x7f03300f5410 2024-02-10T09:11:46.162 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:46.159+0000 7f0325ffb700 1 -- 172.21.15.155:0/243961396 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f032801c560 con 0x7f03300f5410 2024-02-10T09:11:46.162 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:46.159+0000 7f0337289700 1 -- 172.21.15.155:0/243961396 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03300fbf90 con 0x7f03300f5410 2024-02-10T09:11:46.162 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:46.160+0000 7f0325ffb700 1 -- 172.21.15.155:0/243961396 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0328013630 con 0x7f03300f5410 2024-02-10T09:11:46.163 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:46.160+0000 7f0337289700 1 -- 172.21.15.155:0/243961396 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0330059100 con 0x7f03300f5410 2024-02-10T09:11:46.163 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:46.160+0000 7f0325ffb700 1 --2- 172.21.15.155:0/243961396 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f031c02ff60 0x7f031c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:11:46.163 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:46.160+0000 7f0325ffb700 1 -- 172.21.15.155:0/243961396 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0328041240 con 0x7f03300f5410 2024-02-10T09:11:46.163 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:46.160+0000 7f0334824700 1 --2- 172.21.15.155:0/243961396 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f031c02ff60 0x7f031c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:11:46.163 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:46.160+0000 7f0334824700 1 --2- 172.21.15.155:0/243961396 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f031c02ff60 0x7f031c032420 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f03200096a0 tx=0x7f0320006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:11:46.166 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:46.165+0000 7f0325ffb700 1 -- 172.21.15.155:0/243961396 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f03280139f0 con 0x7f03300f5410 2024-02-10T09:11:46.374 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:46.373+0000 7f0337289700 1 -- 172.21.15.155:0/243961396 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0330045ad0 con 0x7f03300f5410 2024-02-10T09:11:46.374 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:46.373+0000 7f0325ffb700 1 -- 172.21.15.155:0/243961396 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0328012380 con 0x7f03300f5410 2024-02-10T09:11:46.374 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:11:46.375 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:11:46.376 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:46.375+0000 7f0337289700 1 -- 172.21.15.155:0/243961396 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f031c02ff60 msgr2=0x7f031c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:11:46.376 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:46.375+0000 7f0337289700 1 --2- 172.21.15.155:0/243961396 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f031c02ff60 0x7f031c032420 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f03200096a0 tx=0x7f0320006b40).stop 2024-02-10T09:11:46.376 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:46.375+0000 7f0337289700 1 -- 172.21.15.155:0/243961396 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03300f5410 msgr2=0x7f03300fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:11:46.376 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:46.375+0000 7f0337289700 1 --2- 172.21.15.155:0/243961396 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03300f5410 0x7f03300fd4e0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f0328004b70 tx=0x7f0328004d10).stop 2024-02-10T09:11:46.376 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:46.375+0000 7f0337289700 1 -- 172.21.15.155:0/243961396 shutdown_connections 2024-02-10T09:11:46.377 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:46.375+0000 7f0337289700 1 --2- 172.21.15.155:0/243961396 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f031c02ff60 0x7f031c032420 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:11:46.377 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:46.375+0000 7f0337289700 1 --2- 172.21.15.155:0/243961396 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f03300f5410 0x7f03300fd4e0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:11:46.377 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:46.375+0000 7f0337289700 1 -- 172.21.15.155:0/243961396 >> 172.21.15.155:0/243961396 conn(0x7f03300f0b70 msgr2=0x7f033005fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:11:46.377 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:46.375+0000 7f0337289700 1 -- 172.21.15.155:0/243961396 shutdown_connections 2024-02-10T09:11:46.377 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:46.375+0000 7f0337289700 1 -- 172.21.15.155:0/243961396 wait complete. 2024-02-10T09:11:46.377 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:11:47.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:11:46 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:11:45.759068+0000 mgr.smithi077.dxntvw (mgr.14162) 400 : cluster [DBG] pgmap v384: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:11:47.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:11:46 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:11:46.374825+0000 mon.smithi077 (mon.0) 341 : audit [DBG] from='client.? 172.21.15.155:0/243961396' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:11:48.117 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-10T09:11:48.118 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T09:11:48.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:11:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:11:47.101890+0000 mon.smithi077 (mon.0) 342 : audit [INF] from='mgr.14162 172.21.15.77:0/2029106299' entity='mgr.smithi077.dxntvw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi077.dxntvw/mirror_snapshot_schedule"}]: dispatch 2024-02-10T09:11:48.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:11:47 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:11:47.107206+0000 mon.smithi077 (mon.0) 343 : audit [INF] from='mgr.14162 172.21.15.77:0/2029106299' entity='mgr.smithi077.dxntvw' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi077.dxntvw/trash_purge_schedule"}]: dispatch 2024-02-10T09:11:49.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:11:48 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:11:47.759517+0000 mgr.smithi077.dxntvw (mgr.14162) 401 : cluster [DBG] pgmap v385: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:11:49.966 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:49.963+0000 7f5b60df6700 1 -- 172.21.15.155:0/970685629 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b5c0f5410 msgr2=0x7f5b5c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:11:49.966 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:49.963+0000 7f5b60df6700 1 --2- 172.21.15.155:0/970685629 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b5c0f5410 0x7f5b5c0f5830 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7f5b44009fb0 tx=0x7f5b44009b30).stop 2024-02-10T09:11:49.966 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:49.963+0000 7f5b60df6700 1 -- 172.21.15.155:0/970685629 shutdown_connections 2024-02-10T09:11:49.966 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:49.963+0000 7f5b60df6700 1 --2- 172.21.15.155:0/970685629 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b5c0f5410 0x7f5b5c0f5830 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:11:49.967 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:49.963+0000 7f5b60df6700 1 -- 172.21.15.155:0/970685629 >> 172.21.15.155:0/970685629 conn(0x7f5b5c0f0b70 msgr2=0x7f5b5c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:11:49.967 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:49.964+0000 7f5b60df6700 1 -- 172.21.15.155:0/970685629 shutdown_connections 2024-02-10T09:11:49.967 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:49.964+0000 7f5b60df6700 1 -- 172.21.15.155:0/970685629 wait complete. 2024-02-10T09:11:49.967 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:49.964+0000 7f5b60df6700 1 Processor -- start 2024-02-10T09:11:49.967 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:49.964+0000 7f5b60df6700 1 -- start start 2024-02-10T09:11:49.967 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:49.964+0000 7f5b60df6700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b5c0f5410 0x7f5b5c1067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:11:49.967 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:49.964+0000 7f5b60df6700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b5c106cc0 con 0x7f5b5c0f5410 2024-02-10T09:11:49.967 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:49.964+0000 7f5b5a59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b5c0f5410 0x7f5b5c1067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:11:49.967 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:49.964+0000 7f5b5a59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b5c0f5410 0x7f5b5c1067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.155:42916/0 (socket says 172.21.15.155:42916) 2024-02-10T09:11:49.967 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:49.964+0000 7f5b5a59c700 1 -- 172.21.15.155:0/3591895933 learned_addr learned my addr 172.21.15.155:0/3591895933 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-02-10T09:11:49.967 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:49.965+0000 7f5b5a59c700 1 -- 172.21.15.155:0/3591895933 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b44005040 con 0x7f5b5c0f5410 2024-02-10T09:11:49.967 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:49.965+0000 7f5b5a59c700 1 --2- 172.21.15.155:0/3591895933 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b5c0f5410 0x7f5b5c1067d0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f5b44005010 tx=0x7f5b4400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:11:49.967 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:49.965+0000 7f5b52ffd700 1 -- 172.21.15.155:0/3591895933 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b44009320 con 0x7f5b5c0f5410 2024-02-10T09:11:49.968 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:49.965+0000 7f5b60df6700 1 -- 172.21.15.155:0/3591895933 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b5c106ec0 con 0x7f5b5c0f5410 2024-02-10T09:11:49.968 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:49.965+0000 7f5b52ffd700 1 -- 172.21.15.155:0/3591895933 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5b44013380 con 0x7f5b5c0f5410 2024-02-10T09:11:49.968 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:49.965+0000 7f5b52ffd700 1 -- 172.21.15.155:0/3591895933 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b44014e50 con 0x7f5b5c0f5410 2024-02-10T09:11:49.968 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:49.965+0000 7f5b60df6700 1 -- 172.21.15.155:0/3591895933 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b5c107350 con 0x7f5b5c0f5410 2024-02-10T09:11:49.969 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:49.966+0000 7f5b52ffd700 1 -- 172.21.15.155:0/3591895933 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5b440095f0 con 0x7f5b5c0f5410 2024-02-10T09:11:49.969 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:49.966+0000 7f5b60df6700 1 -- 172.21.15.155:0/3591895933 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b5c045ad0 con 0x7f5b5c0f5410 2024-02-10T09:11:49.969 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:49.966+0000 7f5b52ffd700 1 --2- 172.21.15.155:0/3591895933 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f5b4802ff10 0x7f5b480323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-10T09:11:49.969 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:49.966+0000 7f5b52ffd700 1 -- 172.21.15.155:0/3591895933 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5b44040fa0 con 0x7f5b5c0f5410 2024-02-10T09:11:49.969 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:49.966+0000 7f5b59d9b700 1 --2- 172.21.15.155:0/3591895933 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f5b4802ff10 0x7f5b480323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-10T09:11:49.969 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:49.967+0000 7f5b59d9b700 1 --2- 172.21.15.155:0/3591895933 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f5b4802ff10 0x7f5b480323d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f5b4c0096a0 tx=0x7f5b4c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-10T09:11:49.972 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:49.971+0000 7f5b52ffd700 1 -- 172.21.15.155:0/3591895933 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5b44014420 con 0x7f5b5c0f5410 2024-02-10T09:11:50.176 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:50.175+0000 7f5b60df6700 1 -- 172.21.15.155:0/3591895933 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5b5c108200 con 0x7f5b5c0f5410 2024-02-10T09:11:50.177 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:50.176+0000 7f5b52ffd700 1 -- 172.21.15.155:0/3591895933 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5b44010230 con 0x7f5b5c0f5410 2024-02-10T09:11:50.177 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-02-10T09:11:50.177 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":1,"fsid":"6081b54a-c7f2-11ee-95b8-87774f69a715","modified":"2024-02-10T08:57:39.474770Z","created":"2024-02-10T08:57:39.474770Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T09:11:50.179 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:50.178+0000 7f5b60df6700 1 -- 172.21.15.155:0/3591895933 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f5b4802ff10 msgr2=0x7f5b480323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:11:50.179 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:50.178+0000 7f5b60df6700 1 --2- 172.21.15.155:0/3591895933 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f5b4802ff10 0x7f5b480323d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f5b4c0096a0 tx=0x7f5b4c006b40).stop 2024-02-10T09:11:50.179 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:50.178+0000 7f5b60df6700 1 -- 172.21.15.155:0/3591895933 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b5c0f5410 msgr2=0x7f5b5c1067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-10T09:11:50.180 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:50.178+0000 7f5b60df6700 1 --2- 172.21.15.155:0/3591895933 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b5c0f5410 0x7f5b5c1067d0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f5b44005010 tx=0x7f5b4400b040).stop 2024-02-10T09:11:50.180 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:50.178+0000 7f5b60df6700 1 -- 172.21.15.155:0/3591895933 shutdown_connections 2024-02-10T09:11:50.180 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:50.178+0000 7f5b60df6700 1 --2- 172.21.15.155:0/3591895933 >> [v2:172.21.15.77:6800/81674875,v1:172.21.15.77:6801/81674875] conn(0x7f5b4802ff10 0x7f5b480323d0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:11:50.180 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:50.178+0000 7f5b60df6700 1 --2- 172.21.15.155:0/3591895933 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b5c0f5410 0x7f5b5c1067d0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-10T09:11:50.180 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:50.178+0000 7f5b60df6700 1 -- 172.21.15.155:0/3591895933 >> 172.21.15.155:0/3591895933 conn(0x7f5b5c0f0b70 msgr2=0x7f5b5c05fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-10T09:11:50.180 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:50.178+0000 7f5b60df6700 1 -- 172.21.15.155:0/3591895933 shutdown_connections 2024-02-10T09:11:50.180 INFO:teuthology.orchestra.run.smithi155.stderr:2024-02-10T09:11:50.178+0000 7f5b60df6700 1 -- 172.21.15.155:0/3591895933 wait complete. 2024-02-10T09:11:50.180 INFO:teuthology.orchestra.run.smithi155.stderr:dumped monmap epoch 1 2024-02-10T09:11:50.870 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ljflores_ceph_57bd6abdec7bb457ae7999d9c96682e9ac678e27/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds 2024-02-10T09:11:50.872 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-02-10T09:11:50.872 DEBUG:teuthology.orchestra.run.smithi077:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-10T09:11:50.897 DEBUG:teuthology.orchestra.run.smithi155:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-10T09:11:50.915 INFO:tasks.cephadm:Stopping all daemons... 2024-02-10T09:11:50.915 INFO:tasks.cephadm.mon.smithi077:Stopping mon.smithi077... 2024-02-10T09:11:50.915 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ceph-6081b54a-c7f2-11ee-95b8-87774f69a715@mon.smithi077 2024-02-10T09:11:51.118 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:11:50 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: cluster 2024-02-10T09:11:49.759869+0000 mgr.smithi077.dxntvw (mgr.14162) 402 : cluster [DBG] pgmap v386: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T09:11:51.118 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:11:50 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: audit 2024-02-10T09:11:50.177501+0000 mon.smithi077 (mon.0) 344 : audit [DBG] from='client.? 172.21.15.155:0/3591895933' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T09:11:51.118 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:11:50 smithi077 systemd[1]: Stopping Ceph mon.smithi077 for 6081b54a-c7f2-11ee-95b8-87774f69a715... 2024-02-10T09:11:51.118 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:11:51 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T09:11:51.117+0000 7f3f67188700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi077 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-02-10T09:11:51.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:11:51 smithi077 ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077[39443]: debug 2024-02-10T09:11:51.117+0000 7f3f67188700 -1 mon.smithi077@0(leader) e1 *** Got Signal Terminated *** 2024-02-10T09:11:51.899 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 10 09:11:51 smithi077 podman[45602]: ceph-6081b54a-c7f2-11ee-95b8-87774f69a715-mon.smithi077 2024-02-10T09:11:51.936 DEBUG:teuthology.orchestra.run.smithi077:> sudo pkill -f 'journalctl -f -n 0 -u ceph-6081b54a-c7f2-11ee-95b8-87774f69a715@mon.smithi077.service' 2024-02-10T09:11:51.987 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T09:11:51.987 INFO:tasks.cephadm.mon.smithi077:Stopped mon.smithi077 2024-02-10T09:11:51.987 INFO:tasks.cephadm.mon.smithi155:Stopping mon.smithi155... 2024-02-10T09:11:51.987 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ceph-6081b54a-c7f2-11ee-95b8-87774f69a715@mon.smithi155 2024-02-10T09:11:52.025 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to stop ceph-6081b54a-c7f2-11ee-95b8-87774f69a715@mon.smithi155.service: Unit ceph-6081b54a-c7f2-11ee-95b8-87774f69a715@mon.smithi155.service not loaded. 2024-02-10T09:11:52.026 DEBUG:teuthology.orchestra.run:got remote process result: 5 2024-02-10T09:11:52.027 ERROR:tasks.cephadm:Failed to stop "ceph.mon.smithi155" Traceback (most recent call last): File "/home/teuthworker/src/github.com_ljflores_ceph_57bd6abdec7bb457ae7999d9c96682e9ac678e27/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ljflores_ceph_57bd6abdec7bb457ae7999d9c96682e9ac678e27/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ljflores_ceph_57bd6abdec7bb457ae7999d9c96682e9ac678e27/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi155 with status 5: 'sudo systemctl stop ceph-6081b54a-c7f2-11ee-95b8-87774f69a715@mon.smithi155' 2024-02-10T09:11:52.028 INFO:tasks.cephadm:Archiving crash dumps... 2024-02-10T09:11:52.029 DEBUG:teuthology.misc:Transferring archived files from smithi077:/var/lib/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-09_21:34:44-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7555330/remote/ubuntu@smithi077.front.sepia.ceph.com/crash 2024-02-10T09:11:52.030 DEBUG:teuthology.orchestra.run.smithi077:> sudo tar c -f - -C /var/lib/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/crash -- . 2024-02-10T09:11:52.073 DEBUG:teuthology.misc:Transferring archived files from smithi155:/var/lib/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-09_21:34:44-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7555330/remote/ubuntu@smithi155.front.sepia.ceph.com/crash 2024-02-10T09:11:52.074 DEBUG:teuthology.orchestra.run.smithi155:> sudo tar c -f - -C /var/lib/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/crash -- . 2024-02-10T09:11:52.112 INFO:teuthology.orchestra.run.smithi155.stderr:tar: /var/lib/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/crash: Cannot open: No such file or directory 2024-02-10T09:11:52.112 INFO:teuthology.orchestra.run.smithi155.stderr:tar: Error is not recoverable: exiting now 2024-02-10T09:11:52.113 INFO:tasks.cephadm:Checking cluster log for badness... 2024-02-10T09:11:52.113 DEBUG:teuthology.orchestra.run.smithi077:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(PG_DEGRADED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-02-10T09:11:52.148 INFO:tasks.cephadm:Compressing logs... 2024-02-10T09:11:52.149 DEBUG:teuthology.orchestra.run.smithi077:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-10T09:11:52.191 DEBUG:teuthology.orchestra.run.smithi155:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-10T09:11:52.217 INFO:teuthology.orchestra.run.smithi077.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-02-10T09:11:52.217 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-10T09:11:52.217 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/ceph-volume.log 2024-02-10T09:11:52.219 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/ceph.audit.log 2024-02-10T09:11:52.219 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/ceph.cephadm.log 2024-02-10T09:11:52.219 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/ceph-volume.log: /var/log/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/ceph-mgr.smithi077.dxntvw.log 2024-02-10T09:11:52.219 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/ceph-mon.smithi077.log 2024-02-10T09:11:52.219 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/ceph-mgr.smithi077.dxntvw.log: 88.9% -- replaced with /var/log/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/ceph.audit.log.gz 2024-02-10T09:11:52.219 INFO:teuthology.orchestra.run.smithi077.stderr: 89.5% -- replaced with /var/log/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/ceph.cephadm.log.gz 2024-02-10T09:11:52.219 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/ceph.log 2024-02-10T09:11:52.219 INFO:teuthology.orchestra.run.smithi077.stderr: 90.2% -- replaced with /var/log/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/ceph-volume.log.gz 2024-02-10T09:11:52.220 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/ceph-mon.smithi077.log: /var/log/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/ceph.log: 88.7% -- replaced with /var/log/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/ceph.log.gz 2024-02-10T09:11:52.223 INFO:teuthology.orchestra.run.smithi155.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- : No such file or directory/var/log/ceph/cephadm.log 2024-02-10T09:11:52.223 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-02-10T09:11:52.223 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/ceph-volume.log 2024-02-10T09:11:52.223 INFO:teuthology.orchestra.run.smithi077.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-10T09:11:52.224 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/ceph-volume.log: 88.4% -- replaced with /var/log/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/ceph-volume.log.gz 2024-02-10T09:11:52.225 INFO:teuthology.orchestra.run.smithi155.stderr: 95.6% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-10T09:11:52.226 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-02-10T09:11:52.227 INFO:teuthology.orchestra.run.smithi155.stderr:real 0m0.015s 2024-02-10T09:11:52.227 INFO:teuthology.orchestra.run.smithi155.stderr:user 0m0.010s 2024-02-10T09:11:52.227 INFO:teuthology.orchestra.run.smithi155.stderr:sys 0m0.016s 2024-02-10T09:11:52.322 INFO:teuthology.orchestra.run.smithi077.stderr:gzip: /var/log/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/ceph-mgr.smithi077.dxntvw.log: file size changed while zipping 2024-02-10T09:11:52.322 INFO:teuthology.orchestra.run.smithi077.stderr: 92.8% -- replaced with /var/log/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/ceph-mgr.smithi077.dxntvw.log.gz 2024-02-10T09:11:52.397 INFO:teuthology.orchestra.run.smithi077.stderr: 90.8% -- replaced with /var/log/ceph/6081b54a-c7f2-11ee-95b8-87774f69a715/ceph-mon.smithi077.log.gz 2024-02-10T09:11:52.399 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-02-10T09:11:52.399 INFO:teuthology.orchestra.run.smithi077.stderr:real 0m0.193s 2024-02-10T09:11:52.399 INFO:teuthology.orchestra.run.smithi077.stderr:user 0m0.292s 2024-02-10T09:11:52.399 INFO:teuthology.orchestra.run.smithi077.stderr:sys 0m0.023s 2024-02-10T09:11:52.399 INFO:tasks.cephadm:Archiving logs... 2024-02-10T09:11:52.400 DEBUG:teuthology.misc:Transferring archived files from smithi077:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-02-09_21:34:44-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7555330/remote/ubuntu@smithi077.front.sepia.ceph.com/log 2024-02-10T09:11:52.400 DEBUG:teuthology.orchestra.run.smithi077:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-10T09:11:52.522 DEBUG:teuthology.misc:Transferring archived files from smithi155:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-02-09_21:34:44-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7555330/remote/ubuntu@smithi155.front.sepia.ceph.com/log 2024-02-10T09:11:52.523 DEBUG:teuthology.orchestra.run.smithi155:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-10T09:11:52.558 INFO:tasks.cephadm:Removing cluster... 2024-02-10T09:11:52.558 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 --force 2024-02-10T09:13:57.582 DEBUG:teuthology.orchestra.run.smithi155:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 6081b54a-c7f2-11ee-95b8-87774f69a715 --force 2024-02-10T09:13:57.888 INFO:tasks.cephadm:Removing cephadm ... 2024-02-10T09:13:57.888 DEBUG:teuthology.orchestra.run.smithi077:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-10T09:13:57.908 DEBUG:teuthology.orchestra.run.smithi155:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-10T09:13:57.925 INFO:tasks.cephadm:Teardown complete 2024-02-10T09:13:57.926 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/github.com_ljflores_ceph_57bd6abdec7bb457ae7999d9c96682e9ac678e27/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ljflores_ceph_57bd6abdec7bb457ae7999d9c96682e9ac678e27/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ljflores_ceph_57bd6abdec7bb457ae7999d9c96682e9ac678e27/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ljflores_ceph_57bd6abdec7bb457ae7999d9c96682e9ac678e27/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ljflores_ceph_57bd6abdec7bb457ae7999d9c96682e9ac678e27/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi155 with status 5: 'sudo systemctl stop ceph-6081b54a-c7f2-11ee-95b8-87774f69a715@mon.smithi155' 2024-02-10T09:13:58.145 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=fde892d9921c45989b0a03a4d75dd586 Traceback (most recent call last): File "/home/teuthworker/src/github.com_ljflores_ceph_57bd6abdec7bb457ae7999d9c96682e9ac678e27/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ljflores_ceph_57bd6abdec7bb457ae7999d9c96682e9ac678e27/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ljflores_ceph_57bd6abdec7bb457ae7999d9c96682e9ac678e27/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ljflores_ceph_57bd6abdec7bb457ae7999d9c96682e9ac678e27/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ljflores_ceph_57bd6abdec7bb457ae7999d9c96682e9ac678e27/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi155 with status 5: 'sudo systemctl stop ceph-6081b54a-c7f2-11ee-95b8-87774f69a715@mon.smithi155' 2024-02-10T09:13:58.149 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-02-10T09:13:58.160 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-02-10T09:13:58.175 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-02-10T09:13:58.175 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-10T09:13:58.189 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-10T09:13:58.258 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-02-10T09:13:58.267 INFO:teuthology.task.clock:Checking final clock skew... 2024-02-10T09:13:58.267 DEBUG:teuthology.orchestra.run.smithi077:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-10T09:13:58.270 DEBUG:teuthology.orchestra.run.smithi155:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-10T09:13:58.285 INFO:teuthology.orchestra.run.smithi077.stderr:bash: ntpq: command not found 2024-02-10T09:13:58.286 INFO:teuthology.orchestra.run.smithi155.stderr:bash: ntpq: command not found 2024-02-10T09:13:58.290 INFO:teuthology.orchestra.run.smithi077.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-10T09:13:58.290 INFO:teuthology.orchestra.run.smithi077.stdout:=============================================================================== 2024-02-10T09:13:58.291 INFO:teuthology.orchestra.run.smithi077.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 19 +35us[ +57us] +/- 82ms 2024-02-10T09:13:58.291 INFO:teuthology.orchestra.run.smithi077.stdout:^* hv02.front.sepia.ceph.com 3 6 377 16 +192us[ +214us] +/- 59ms 2024-02-10T09:13:58.291 INFO:teuthology.orchestra.run.smithi077.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 17 -199us[ -177us] +/- 64ms 2024-02-10T09:13:58.291 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T09:13:58.292 INFO:teuthology.orchestra.run.smithi155.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-10T09:13:58.293 INFO:teuthology.orchestra.run.smithi155.stdout:=============================================================================== 2024-02-10T09:13:58.293 INFO:teuthology.orchestra.run.smithi155.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 20 +43us[ +49us] +/- 82ms 2024-02-10T09:13:58.293 INFO:teuthology.orchestra.run.smithi155.stdout:^* hv02.front.sepia.ceph.com 3 6 377 18 +211us[ +217us] +/- 59ms 2024-02-10T09:13:58.293 INFO:teuthology.orchestra.run.smithi155.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 18 -197us[ -191us] +/- 64ms 2024-02-10T09:13:58.293 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T09:13:58.293 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-02-10T09:13:58.303 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-02-10T09:13:58.304 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-02-10T09:13:58.315 DEBUG:teuthology.orchestra.run.smithi077:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-10T09:13:58.396 DEBUG:teuthology.orchestra.run.smithi155:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-10T09:13:58.461 DEBUG:teuthology.orchestra.run.smithi077:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-10T09:13:58.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T09:13:58.492 DEBUG:teuthology.orchestra.run.smithi155:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-10T09:13:58.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T09:13:58.522 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-02-10T09:13:58.531 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-02-10T09:13:58.540 INFO:teuthology.task.internal:Duration was 2074.099504 seconds 2024-02-10T09:13:58.541 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-02-10T09:13:58.553 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-02-10T09:13:58.553 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-10T09:13:58.556 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-10T09:13:58.600 INFO:teuthology.orchestra.run.smithi077.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-10T09:13:58.607 INFO:teuthology.orchestra.run.smithi155.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-10T09:13:58.910 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-02-10T09:13:58.911 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi077.front.sepia.ceph.com 2024-02-10T09:13:58.911 DEBUG:teuthology.orchestra.run.smithi077:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-10T09:13:58.934 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi155.front.sepia.ceph.com 2024-02-10T09:13:58.934 DEBUG:teuthology.orchestra.run.smithi155:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-10T09:13:58.956 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-02-10T09:13:58.956 DEBUG:teuthology.orchestra.run.smithi077:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-10T09:13:58.976 DEBUG:teuthology.orchestra.run.smithi155:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-10T09:13:59.046 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-02-10T09:13:59.046 DEBUG:teuthology.orchestra.run.smithi077:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-10T09:13:59.049 DEBUG:teuthology.orchestra.run.smithi155:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-10T09:13:59.150 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-02-10T09:13:59.160 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-02-10T09:13:59.160 DEBUG:teuthology.orchestra.run.smithi077:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-10T09:13:59.187 DEBUG:teuthology.orchestra.run.smithi155:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-10T09:13:59.214 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-02-10T09:13:59.224 DEBUG:teuthology.orchestra.run.smithi077:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-10T09:13:59.229 DEBUG:teuthology.orchestra.run.smithi155:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-10T09:13:59.253 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern = core 2024-02-10T09:13:59.280 INFO:teuthology.orchestra.run.smithi155.stdout:kernel.core_pattern = core 2024-02-10T09:13:59.307 DEBUG:teuthology.orchestra.run.smithi077:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-10T09:13:59.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T09:13:59.330 DEBUG:teuthology.orchestra.run.smithi155:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-10T09:13:59.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T09:13:59.365 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-02-10T09:13:59.376 INFO:teuthology.task.internal:Transferring archived files... 2024-02-10T09:13:59.377 DEBUG:teuthology.misc:Transferring archived files from smithi077:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-09_21:34:44-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7555330/remote/smithi077 2024-02-10T09:13:59.378 DEBUG:teuthology.orchestra.run.smithi077:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-10T09:13:59.418 DEBUG:teuthology.misc:Transferring archived files from smithi155:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-09_21:34:44-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7555330/remote/smithi155 2024-02-10T09:13:59.419 DEBUG:teuthology.orchestra.run.smithi155:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-10T09:13:59.460 INFO:teuthology.task.internal:Removing archive directory... 2024-02-10T09:13:59.460 DEBUG:teuthology.orchestra.run.smithi077:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-10T09:13:59.462 DEBUG:teuthology.orchestra.run.smithi155:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-10T09:13:59.511 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-02-10T09:13:59.523 INFO:teuthology.task.internal:Not uploading archives. 2024-02-10T09:13:59.523 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-02-10T09:13:59.535 INFO:teuthology.task.internal:Tidying up after the test... 2024-02-10T09:13:59.535 DEBUG:teuthology.orchestra.run.smithi077:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-10T09:13:59.538 DEBUG:teuthology.orchestra.run.smithi155:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-10T09:13:59.552 INFO:teuthology.orchestra.run.smithi077.stdout: 525357 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 10 09:13 /home/ubuntu/cephtest 2024-02-10T09:13:59.568 INFO:teuthology.orchestra.run.smithi155.stdout: 525357 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 10 09:13 /home/ubuntu/cephtest 2024-02-10T09:13:59.570 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-02-10T09:13:59.583 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-02-10T09:13:59.611 INFO:teuthology.nuke:Checking targets against current locks 2024-02-10T09:13:59.639 DEBUG:teuthology.nuke:shortname: smithi077 2024-02-10T09:13:59.639 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-10T09:13:59.663 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi077.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_21:34:44-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7555330', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-10 08:25:48.722617', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOkHI/0YKMx3NR9cd+Rv5sFx5wEZthIIzyZxxo+3z4ApnP9wcOXcnzolJWz1MPWdK32Q4vEuZIOvPtoC69iERC0='} 2024-02-10T09:13:59.667 DEBUG:teuthology.nuke:shortname: smithi155 2024-02-10T09:13:59.667 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-10T09:13:59.690 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi155.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_21:34:44-rados-wip-yuri10-testing-2024-02-08-0854-pacific-distro-default-smithi/7555330', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-10 08:25:48.721253', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLG3o25k+pBMjkJgMe4uujFxi29ZlQcrYW+WhAldqCGVL5blpah59RRoilRfwNzrFs/5war+okFvzmcnzMkaSnc='} 2024-02-10T09:13:59.719 INFO:teuthology.orchestra.console.smithi077:Power off 2024-02-10T09:13:59.720 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-10T09:13:59.743 DEBUG:teuthology.orchestra.console.smithi077:power off output: Chassis Power Control: Down/Off 2024-02-10T09:13:59.743 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T09:13:59.761 INFO:teuthology.orchestra.console.smithi155:Power off 2024-02-10T09:13:59.762 DEBUG:teuthology.orchestra.console.smithi155:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-10T09:13:59.771 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is on 2024-02-10T09:13:59.785 DEBUG:teuthology.orchestra.console.smithi155:power off output: Chassis Power Control: Down/Off 2024-02-10T09:13:59.785 DEBUG:teuthology.orchestra.console.smithi155:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T09:13:59.811 DEBUG:teuthology.orchestra.console.smithi155:check power output: Chassis Power is on 2024-02-10T09:14:03.772 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T09:14:03.812 DEBUG:teuthology.orchestra.console.smithi155:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T09:14:03.882 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is on 2024-02-10T09:14:03.922 DEBUG:teuthology.orchestra.console.smithi155:check power output: Chassis Power is on 2024-02-10T09:14:07.886 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T09:14:07.924 DEBUG:teuthology.orchestra.console.smithi155:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T09:14:08.001 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is off 2024-02-10T09:14:08.034 DEBUG:teuthology.orchestra.console.smithi155:check power output: Chassis Power is off 2024-02-10T09:14:08.103 INFO:teuthology.orchestra.console.smithi077:Power off completed 2024-02-10T09:14:08.135 INFO:teuthology.orchestra.console.smithi155:Power off completed 2024-02-10T09:14:08.240 INFO:teuthology.run:Summary data: description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} duration: 2074.099503517151 failure_reason: 'Command failed on smithi155 with status 5: ''sudo systemctl stop ceph-6081b54a-c7f2-11ee-95b8-87774f69a715@mon.smithi155''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=fde892d9921c45989b0a03a4d75dd586 status: fail success: false 2024-02-10T09:14:08.241 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-10T09:14:08.324 INFO:teuthology.run:FAIL